### What is this PR for?
Currently, zeppelin only support 3 kinds of dynamic form controls: TextBox, Select, CheckBox. All the things are in `Input.java`, this is hard to add new controls, this PR is for refactoring Input to make dynamic forms extensible. Main Changes:
* Make `Input` as the base class of dynamic forms also use it as the factory class
* All the concret dynamic forms extend `Input`
* Add method `toJson` and `fromJson` for `GUI` for `GUI`'s serialization/deserialization. I plan to do it for other classes as well, so that we can remove duplicated serde code and also make it easy to test serialization/deserialization
* Change `z.input` to `z.textbox` as I think z.input is a little misleading. But I still keep `z.input` and make `z.input` as deprecated.
* Ideally the new input forms' json should be the same as the old input form json. But there's one bug in the old input form, `type` is missing if the input forms are created in frontend for textbox and select. So I keep the old input forms for compatibility. I will load the old input forms json and convert it into new input forms, and after saving, `note.json` would have the new input forms json.
After this PR, user needs to do 3 things to add new ui controls
* Implement its UI control classes, (refer TextBox/CheckBox/Select), and specify it in `TypeAdapterFactory` of `Input` for serde.
* Add parsing logic in `Input.getInputForm` if you want to support this control in frontend.
* Add display logic in `paragraph-parameterizedQueryForm.html`
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2395
### How should this be tested?
Test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2245 from zjffdu/ZEPPELIN-2395 and squashes the following commits:
16d42a8 [Jeff Zhang] ZEPPELIN-2395. Refactor Input.java to make dynamic forms extensible
### What is this PR for?
`clear output` (`cmd` + `opt` + `L`) doesn't work in the personalized mode.
### What type of PR is it?
[Bug Fix]
### Todos
NONE
### What is the Jira issue?
[ZEPPELIN-2179](https://issues.apache.org/jira/browse/ZEPPELIN-2179)
### How should this be tested?
1. Configure shiro
2. Start Zeppelin and login in 2 browsers (e.g chrome and safari) with 2 different users (**DO NOT USE `dev` mode: localhost:9000**)
3. Create a note
4. Enable personalized mode
5. Run paragraph and clear output in each browser
### Screenshots (if appropriate)
NONE
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#2253 from 1ambda/ZEPPELIN-2179/clear-output-doesnt-work-in-person-mode-for-master and squashes the following commits:
eeff440ec [1ambda] fix: Clear personalized output before running
b7387849f [1ambda] fix: Clear output in personalized paragraph
### What is this PR for?
I can reproduce it via the test code in ZeppelinIT.testAngularDisplay. First run the 4 paragraphs to initiate the state. Then when I click paragraph 0 the first time, paragraph 2 will run correctly. But if I click paragraph 0 again, interpreter would raise the following exception. The cause is that ZeppelinContext's interpreterContext now point to paragraph 2, because paragraph 2 is the last paragraph that runs in this interpreter. So I propose to add parameter for check running current paragraph in ZeppelinContext.
* Add 2 methods in ZeppelinContext. `run(idx, checkCurrentParagraph) `, 'run(noteId, idx, context, checkCurrentParagraph)'
* Also make some changes for debug logging.
### What type of PR is it?
[Bug Fix | Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2386
### How should this be tested?
Tested manually and also add integration test.
### Screenshots (if appropriate)
Before
After
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2240 from zjffdu/ZEPPELIN-2386 and squashes the following commits:
03995a3 [Jeff Zhang] ZEPPELIN-2386. Add parameter to check running current paragraph in ZeppelinContext
### What is this PR for?
Currently it is not convenient to access the individual item of the return value of z.checkbox, I would propose to return `Seq` for `SparkInterpreter` and `list` for `PySparkInterpreter`. This might cause some incompatibility, but I think it is acceptable considering the benefits. Besides that, before this PR, all the items of checkbox would be checked by default in `PySparkInterpreter` which is inconsistent with `SparkInterpreter`, so I change it to nothing is selected by default in this PR.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2390
### How should this be tested?
Unit test is added
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2241 from zjffdu/ZEPPELIN-2390 and squashes the following commits:
75e3afc [Jeff Zhang] ZEPPELIN-2390. Improve returnType for z.checkbox
### What is this PR for?
In zeppelin SystemUser fails to authenticate with AD, using the password set in hadoop credential store.
### What type of PR is it?
Bug Fix
### Todos
* [ ] - Task
### What is the Jira issue?
[ZEPPELIN-2366](https://issues.apache.org/jira/browse/ZEPPELIN-2366)
### How should this be tested?
Configure AD system user password using hadoop credential in shiro.ini. _(sample config in JIRA ticket)_
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? n/a
* Is there breaking changes for older versions? n/a
* Does this needs documentation? n/a
Author: Renjith Kamath <renjith.kamath@gmail.com>
Closes#2230 from r-kamath/ZEPPELIN-2366 and squashes the following commits:
11a8ab43d [Renjith Kamath] ZEPPELIN-2366 In zeppelin SystemUser fails to authenticate with AD, using the password set in hadoop credential store.
### What is this PR for?
Bundle helium packages one by one because
#### Summary
Let's say you have one helium package called `ultimate-line-chart`.
**1. Package Path**
- **(before)** `$ZEPPELIN_HOME/local-repo/helium-bundles/node_modules/ultimate-line-chart`
- **(after)** `$ZEPPELIN_HOME/local-repo/helium-bundles/bundles/ultimate-line-chart`
**2. Zeppelin Local Module Path**
- **(before)** `$ZEPPELIN_HOME/local-repo/helium-bundles/node_modules/zeppelin-tabledata`
- **(after)** `$ZEPPELIN_HOME/local-repo/helium-bundles/local_modules/zeppelin-tabledata`
**3. Bundle Cache Path**
- **(before)** `$ZEPPELIN_HOME/local-repo/helium-bundles/helium.bundle.cache.js` (mixed one)
- **(after)** `$ZEPPELIN_HOME/local-repo/helium-bundles/bundles/ultimate-line-chart/helium.bundle.cache.js` (for each helium package)
#### Details
- Bundling them in one file is not good idea because the whole bundling is broken when only 1 bundle fails
- Some node packages might not installed correctly. For example, `amchart/amcharts3` will be installed in `helium-bundles/node_modules/my-helium-vis/node_modules` but `amchart-export` will be placed in `helium-bundles/node_modules` so `my-helium-vis` can't import `amchart-export`. (This is real case)
- Additionally, I used yarn to install required node_modules. separated bundling requires install same dependencies multiple times. This takes more time than before and npm is too slow to install them all.
- Install Zeppelin framework modules using [local path](https://docs.npmjs.com/files/package.json#local-paths). It's more safe and correct way to install local modules.
### What type of PR is it?
[Improvement]
### Todos
NONE
### What is the Jira issue?
[ZEPPELIN-2088](https://issues.apache.org/jira/browse/ZEPPELIN-2088)
### How should this be tested?
1. Install any local helium package
2. Install any online helium package
3. Use them
4. Disable
5. Test `npm run dev:helium`
### Screenshots (if appropriate)
NONE
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#2210 from 1ambda/ZEPPELIN-2088/evaluate-helium-bundle-one-by-one and squashes the following commits:
bea6c09 [1ambda] fix: Bundling errors
53145fa [1ambda] fix: Add error handling for failed bundle
212c737 [1ambda] feat: Apply #2178736acee [1ambda] fix: Add nodeInstallationDir
11ef0ae [1ambda] fix: HeliumBundleFactoryTest
5a2a938 [1ambda] fix: Log single bundle error to browser console
a99f981 [1ambda] test: Fix APIs
743aba4 [1ambda] feat: Loading bundles
697c5e6 [1ambda] feat: enable, disable
002e66f [1ambda] feat: Build online packages
512508d [1ambda] feat: Build each bundle using yarn
### What is this PR for?
fixing String comparison and improving map browsing
### What type of PR is it?
Bug Fix
### Todos
* [ ] - Task
### What is the Jira issue?
[ZEPPELIN-2320](https://issues.apache.org/jira/browse/ZEPPELIN-2320)
### How should this be tested?
build with tests.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Remilito <remy.gayet@gmail.com>
Closes#2192 from Remilito/ZEPPELIN-2320 and squashes the following commits:
9ad16bc [Remilito] [ZEPPELIN-2320] fixing test + modifying map browsing
### What is this PR for?
The order of dynamic forms should be the order that you create them. So I made the following 2 changes for this:
* change the type of forms in GUI from TreeMap to LinkedHashMap
* remove orderBy in paragraph-parameterizedQueryForm.html
Besides, I also did some code refactoring.
### What type of PR is it?
[Bug Fix | Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2189
### How should this be tested?
Tested manually.
### Screenshots (if appropriate)
Before
After
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2204 from zjffdu/ZEPPELIN-2189 and squashes the following commits:
a69ab5a [Jeff Zhang] add test
167d162 [Jeff Zhang] fix code style
369900e [Jeff Zhang] ZEPPELIN-2189. The order of dynamic forms should be the order that you create them
### What is this PR for?
Adding Selenium tests to ensure proper behaviour of dynamic forms.
### What type of PR is it?
Test
### Todos
N/A
### What is the Jira issue?
[ZEPPELIN-1720](https://issues.apache.org/jira/browse/ZEPPELIN-1720)
### How should this be tested?
1. Once should first get Firefox v. 41 as it is the latest version that works with the current version of Selenium in Apache Zeppelin.
2. You can then run the tests with following command:
`TEST_SELENIUM="true" mvn package -DfailIfNoTests=false -pl 'zeppelin-interpreter,zeppelin-zengine,zeppelin-server' -Dtest=ParagraphActionsIT
`
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Guillermo Cabrera <guicaro@gmail.com>
Closes#2141 from guicaro/ZEPPELIN-1720-AddingTestsForDynamicForms and squashes the following commits:
a5bc7db [Guillermo Cabrera] Updated tests to work with new paragraph behaviour
3fa2033 [Guillermo Cabrera] Updated testMultipleDynamicFormsSameType based on new behaviour of paragraphs
92102bd [Guillermo Cabrera] Updated testSingleDynamicFormSelectForm for "Run on selection change" option
f194979 [Guillermo Cabrera] Fixing other issues on test case messages for readability
f7e99ca [Guillermo Cabrera] Corrected grammar in a test case output message
bc1e7f9 [Guillermo Cabrera] Removed unused import, alignment and removed unnecesary condition in test case
274b2c1 [Guillermo Cabrera] Added tests that cover behaviour of dynamic forms
eed42f0 [Guillermo Cabrera] Completed and verified corrct behaviour of testSingleDynamicFormTextInput
7a4f121 [Guillermo Cabrera] Added method stubbs for new UI tests checking correct behaviour of dynamic forms
### What is this PR for?
Use uppercase for keyboard shortcut description so that user can recognize shortcuts easily.
it's hard to understand for some shortcuts since we used lowercases. For example, `Ctrl+Option+l` (actually this `L`, not `I`)
I attached screenshots.
### What type of PR is it?
[Improvement]
### Todos
NONE
### What is the Jira issue?
[ZEPPELIN-2243](https://issues.apache.org/jira/browse/ZEPPELIN-2243)
### How should this be tested?
1. Open the paragraph shortcut panel.
2. Open the note shortcut dialog.
### Screenshots (if appropriate)
#### Before
<img width="261" alt="before_2243" src="https://cloud.githubusercontent.com/assets/4968473/23786191/fb3dafe8-05ae-11e7-8e1e-36b5f151070f.png">
#### After
<img width="263" alt="after_2243" src="https://cloud.githubusercontent.com/assets/4968473/23786194/fe59b1e0-05ae-11e7-9741-371f8e3f77de.png">
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#2119 from 1ambda/ZEPPELIN-2243/use-uppercase-for-keyboard-shortcut-description and squashes the following commits:
9bd868a [1ambda] fix: Remove white spaces
6a163d6 [1ambda] fix: ParagraphActionsIT
5366743 [1ambda] fix: Use uppercase for shortcut desc
### What is this PR for?
When deploying Zeppelin in private Cloud or within closed networks, it happens that Npm Registry isn't directly reachable, so we want to configure our own proxy/registry otherwise we couldn't fetch dependencies. It is anyhow worth to let user configure npm registry instead of hardcoding the URL.
### What type of PR is it?
Improvement
### What is the Jira issue?
[ZEPPELIN-2278]
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Andrea Peruffo <andrea.peruffo1982@gmail.com>
Closes#2150 from andreaTP/npmConfig and squashes the following commits:
56fab25 [Andrea Peruffo] [ZEPPELIN-2278] Env variable to configure Npm registry
Change-Id: Iad87ebe0b5dd6bd67a12e47fe83fbd0e1e71bda9
### What is this PR for?
ZEPPELIN-173 implemented an optional check for the Origin header during the websocket connection creation. This check is no longer active since jetty is upgraded to 9 as the checkOrigin method is no longer called automatically.
This fix, just calls the existing check from the WebsocketCreator manually.
### What type of PR is it?
Bug Fix
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2288
### How should this be tested?
1. Set `zeppelin.server.allowed.origins` in zeppelin-site.xml other than the default `*`. Eg. `xxxx`
2. Start zeppelin
3. Try to open the zeppelin ws conection with different origin:
```
curl 'http://localhost:8080/ws' -H 'Host: localhost:8080' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Sec-WebSocket-Version: 13' -H 'origin: http://other:8080' -H 'Sec-WebSocket-Extensions: permessage-deflate' -H 'Sec-WebSocket-Key: BpiqAMwZaQUJQ//NtEaQPw==' -H 'Connection: keep-alive, Upgrade' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'Upgrade: websocket'
```
4. The websocket creation should be failed if the origin is not localhost or not the string configurated at 2.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Elek, Márton <elek@users.noreply.github.com>
Closes#2166 from elek/ZEPPELIN-2288 and squashes the following commits:
74b37a4 [Elek, Márton] [ZEPPELIN-2288] Fix Cross-Site WebSocket check
### What is this PR for?
* Allow to provide full paragraph config directly in the Create Paragraph and Create Note endpoint.
* This saves some calls to [noteId]/paragraph/[paragraphId]/config
* Updated doc.
### What type of PR is it?
Improvement
### Todos
### What is the Jira issue?
[ZEPPELIN-2106](https://issues.apache.org/jira/browse/ZEPPELIN-2106)
### How should this be tested?
Outline the steps to test the PR here.
1. Clone the first paragraph of 'Zeppelin Tutorial/Basic Features (Spark)' to get the bank data loaded in a new note.
2. curl -X POST -d testAPI.json http://localhost:8080/api/notebook/$YOURNOTEID/paragraph
3. When running the paragraphes, the graphs will show up with the appropriate settings.
testAPI.json:
`{
"title":"Example providing config",
"text":"%sql\nselect age, marital, count(1) cvalue from bank group by age, marital order by age",
"config": {
"title":true,
"colWidth":6.0,
"results": [
{
"graph": {
"mode": "scatterChart",
"optionOpen": true
}
}
]
},
"colWidth":9.0
}
`
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Remilito <remy.gayet@gmail.com>
Closes#2099 from Remilito/ZEPPELIN-2106b and squashes the following commits:
0994ac0 [Remilito] [ZEPPELIN-2106]: keeping only the API providing the whole config
76af44a [Remilito] [ZEPPELIN-2106] providing paragraph config in create note/paragraph call * Allow to describe graph, colWidth, showTitle or full paragraph config directly in the Create Paragraph and Create Note endpoint. * Updated doc.
### What is this PR for?
This is to fix problem with ws connection that was introduced in 0.7.0 with `ZeppelinHubRepo` storage layer.
### What type of PR is it?
Bug Fix
### Todos
* [x] - fix multi-user ws connection issue
* [x] - handle back connection
* [x] - handle switch tokens
* [x] - handle single token in conf
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1697
### How should this be tested?
scenario 1.
1. setup storage as in [here](https://zeppelin.apache.org/docs/0.7.0/storage/storage.html#storage-in-zeppelinhub), note that can use single storage as well (useful when switching instances)
2. setup authentication through ZeppelinHub realm as in [here](https://zeppelin.apache.org/docs/0.7.0/security/shiroauthentication.html#zeppelinhub).
3. login and run paragraph
scenario 2.
this scenario has basic Zeppelin authentication
1. setup storage as in [here](https://zeppelin.apache.org/docs/0.7.0/storage/storage.html#storage-in-zeppelinhub), note that can use single storage as well (useful when switching instances)
2. use non-authenticated mode
3. start zeppelin and run paragrpaph
other cases would be nice to try and report
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Khalid Huseynov <khalidhnv@gmail.com>
Closes#2161 from khalidhuseynov/fix/ZEPPELIN-1697 and squashes the following commits:
0b76a9b [Khalid Huseynov] minor: add two paragraph update ops
37e4a9f [Khalid Huseynov] add watcher principal
4616a33 [Khalid Huseynov] update ops
4e36ce6 [Khalid Huseynov] token container instance -> getInstance
b465b6e [Khalid Huseynov] clean logs
7960400 [Khalid Huseynov] fix live msg
c24b8a1 [Khalid Huseynov] set conf token for anonymous
941794f [Khalid Huseynov] realm token precedence over conf token
91cb537 [Khalid Huseynov] handle empty tokens in rest handler
d14f04f [Khalid Huseynov] switch tokens routine
5755a5c [Khalid Huseynov] fix ws back connection
7e1ac5d [Khalid Huseynov] fix empty token case
39a9a18 [Khalid Huseynov] multi ws one way connection
### What is this PR for?
Removing some interpreters while removing note. In case of "Per note", "Isolated" interpreter process should be removed if a note be removed. In case of "Scoped" of "Per note", that session should be removed and if there's no more session in that process, the process should be removed, too.
### What type of PR is it?
[Hot Fix]
### Todos
* [x] - Handle removing interpreters when removing note
### What is the Jira issue?
N/A
### How should this be tested?
1. set python interpreter as "per note/scoped"
1. run ```%python print(1)``` in a note
1. remove that note
1. check if python process exists with grep
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#2159 from jongyoul/hotfix/remove-note-remove-interpreter-instances and squashes the following commits:
cb3de97 [Jongyoul Lee] Added method to handle removing interpreters while removing note
### What is this PR for?
Minor change to only call getProgress one time.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2256
### How should this be tested?
Before
```
DEBUG [2017-03-14 10:03:44,376] ({pool-2-thread-6} BaseLivyInterprereter.java[callRestAPI]:441) - Get response, StatusCode: 200, responseBody: {"id":1,"state":"running","output":null,"progress":0.6}
INFO [2017-03-14 10:03:44,777] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:60
INFO [2017-03-14 10:03:44,778] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:60
INFO [2017-03-14 10:03:45,284] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:60
INFO [2017-03-14 10:03:45,284] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:60
DEBUG [2017-03-14 10:03:45,378] ({pool-2-thread-6} BaseLivyInterprereter.java[callRestAPI]:407) - Call rest api in http://localhost:8999/sessions/11/statements/1, method: GET, jsonData:
DEBUG [2017-03-14 10:03:45,387] ({pool-2-thread-6} BaseLivyInterprereter.java[callRestAPI]:441) - Get response, StatusCode: 200, responseBody: {"id":1,"state":"running","output":null,"progress":0.8}
INFO [2017-03-14 10:03:45,786] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:80
INFO [2017-03-14 10:03:45,786] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:80
INFO [2017-03-14 10:03:46,291] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:80
INFO [2017-03-14 10:03:46,292] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:80
DEBUG [2017-03-14 10:03:46,392] ({pool-2-thread-6} BaseLivyInterprereter.java[callRestAPI]:407) - Call rest api in http://localhost:8999/sessions/11/statements/1, method: GET, jsonData:
DEBUG [2017-03-14 10:03:46,422] ({pool-2-thread-6} BaseLivyInterprereter.java[callRestAPI]:441) - Get response, StatusCode: 200, responseBody: {"id":1,"state":"running","output":null,"progress":0.9}
INFO [2017-03-14 10:03:46,795] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:90
INFO [2017-03-14 10:03:46,796] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:90
INFO [2017-03-14 10:03:47,297] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:90
INFO [2017-03-14 10:03:47,297] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:181) - progress:90
```
After
```
DEBUG [2017-03-14 11:15:57,176] ({pool-2-thread-2} BaseLivyInterprereter.java[callRestAPI]:432) - Get response, StatusCode: 200, responseBody: {"id":0,"state":"running","output":null,"progress":0.6}
INFO [2017-03-14 11:15:57,439] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:176) - getProcess:**********
INFO [2017-03-14 11:15:57,942] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:176) - getProcess:**********
DEBUG [2017-03-14 11:15:58,177] ({pool-2-thread-2} BaseLivyInterprereter.java[callRestAPI]:398) - Call rest api in http://localhost:8999/sessions/4/statements/0, method: GET, jsonData:
DEBUG [2017-03-14 11:15:58,193] ({pool-2-thread-2} BaseLivyInterprereter.java[callRestAPI]:432) - Get response, StatusCode: 200, responseBody: {"id":0,"state":"running","output":null,"progress":0.8}
INFO [2017-03-14 11:15:58,443] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:176) - getProcess:**********
INFO [2017-03-14 11:15:58,947] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:176) - getProcess:**********
DEBUG [2017-03-14 11:15:59,195] ({pool-2-thread-2} BaseLivyInterprereter.java[callRestAPI]:398) - Call rest api in http://localhost:8999/sessions/4/statements/0, method: GET, jsonData:
DEBUG [2017-03-14 11:15:59,213] ({pool-2-thread-2} BaseLivyInterprereter.java[callRestAPI]:432) - Get response, StatusCode: 200, responseBody: {"id":0,"state":"running","output":null,"progress":0.9}
INFO [2017-03-14 11:15:59,451] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:176) - getProcess:**********
INFO [2017-03-14 11:15:59,957] ({pool-1-thread-3} BaseLivyInterprereter.java[getProgress]:176) - getProcess:**********
```
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2131 from zjffdu/ZEPPELIN-2256 and squashes the following commits:
d2c27b7 [Jeff Zhang] [ZEPPELIN-2256]. poll job progress is called twice
### What is this PR for?
When client (zeppelin-web) send message with null ticket, NotebookServer does not handle and throw NPE.
### What type of PR is it?
Bug Fix
### Todos
* [x] - Null check
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2282
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#2153 from Leemoonsoo/ZEPPELIN-2282 and squashes the following commits:
1a05bed [Lee moon soo] prevent NPE
### What is this PR for?
This is a second part of ZEPPELIN-2047. This issue relates to #2140
### What type of PR is it?
[Hot Fix]
### Todos
* [x] - Per user with Isolated
* [x] - Per note with Scoped
* [x] - Per note with Isolated
* [x] - Restart all interpreter when user click the restart button "Interpreter tab"
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2037
* https://issues.apache.org/jira/browse/ZEPPELIN-1832
### How should this be tested?
N/A
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#2149 from jongyoul/ZEPPELIN-2037-2-per-note and squashes the following commits:
8341348 [Jongyoul Lee] Changed "restart" in interpreter tab to restart all of interpreterGroups in that interpreterSetting
bcccbb9 [Jongyoul Lee] Added test cases for "per note" as "isolated"
0d53d1d [Jongyoul Lee] Fixed to run "per note" as "scoped"
9d5b4b4 [Jongyoul Lee] Fixed to run "per user" as "isolated"
### What is this PR for?
I added "Auto Run" checkbox for select dynamic form to make user turn on / off automatic running after the form value changed.
### What type of PR is it?
Improvement
### Todos
* [x] - update docs after getting feedback
### What is the Jira issue?
[ZEPPELIN-2060](https://issues.apache.org/jira/browse/ZEPPELIN-2060)
### How should this be tested?
1. Apply this patch and run Zeppelin web as dev mode
```bash
# under zeppelin-web
$ yarn run dev
```
2. Go to Spark tutorial note and try to change value in select form as below screenshot imgs
### Screenshots (if appropriate)
- "Auto Run" checkbox button will be shown only in select dynamic form's dropdown menu
- _**turn on**_ "Auto Run"-> auto run right after the value changed / _**turn off**_ -> need to press `Enter`
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes, maybe [this part](https://github.com/apache/zeppelin/blob/master/docs/manual/dynamicform.md#select-form)
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Closes#2100 from AhyoungRyu/feature/turnOnOrOffAutoRun and squashes the following commits:
4043579 [AhyoungRyu] Add 'setting: {forms: {}}' info to paragraphMock
47dcb5f [AhyoungRyu] Persist 'runOnSelectionChange' under note.json's config field
4904f38 [AhyoungRyu] Update dynamicForm docs
66c6c21 [AhyoungRyu] Add test case for behaviour of 'Run on selection change'
8a0ca30 [AhyoungRyu] Change label to 'Run on selection change'
5075711 [AhyoungRyu] Add a space next to checkbox
f2e3259 [AhyoungRyu] Make dynamic select form turn on or off using checkbox
### What is this PR for?
A common use case in LDAP/AD setup is the hierarchical structuring of
groups - a.k.a. adding groups to other groups. Such nesting groups can
help reduce the number of roles that need to be managed.
Current zeppelin realm implementations doesn't have support for looking
up memberships throughout nested group structures.
E.g. consider the following nested group scenario:
```
acme_employees
\__department_a
\__sub_department_x
```
User 'bob' is in Group 'sub_department_x'.
Notebook 'note1' has a Reader Role assignment for 'department_a' or
'acme_employees'.
Then access must be granted for 'bob' on 'note1'.
In AD enviroments this scenarios can be efficiently implemented using
the so called LDAP_MATCHING_RULE_IN_CHAIN operator
'1.2.840.113556.1.4.1941'.
This PR introduces a property 'groupSearchEnableMatchingRuleInChain' to
org.apache.zeppelin.realm.LdapRealm which defaults to false. If enabled,
all roles of potential nested group hierarchies will be resolved using
the LDAP_MATCHING_RULE_IN_CHAIN operator.
### What type of PR is it?
Improvement
### Todos
-
### What is the Jira issue?
[ZEPPELIN-2161]
### How should this be tested?
Set groupSearchEnableMatchingRuleInChain = true for the ldap realm.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? n
* Is there breaking changes for older versions? n
* Does this needs documentation? y
Author: Andreas Weise <a.weise@avm.de>
Closes#2062 from weand/ZEPPELIN-2161 and squashes the following commits:
c08d015 [Andreas Weise] ZEPPELIN-2161 Nested group support in LdapRealm for AD
### What is this PR for?
Fixing empty output issue while using personalized mode.
### What type of PR is it?
[Hot Fix]
### Todos
* [x] - Making default user's paragraph while accessing at the first time.
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2178
### How should this be tested?
Use personalized mode
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#2084 from jongyoul/ZEPPELIN-2178 and squashes the following commits:
5e3ac05 [Jongyoul Lee] Fixed the style
1ffbe00 [Jongyoul Lee] Prevent from broadcasting to all users
2d0d7ca [Jongyoul Lee] Fixed that output and text became reverted into previous status Updated status after running paragraph
47c978f [Jongyoul Lee] Clean outputs when users turn off personalized mode
a190f2f [Jongyoul Lee] Fix the initial status to READY
faacb5f [Jongyoul Lee] Prevent from cleaning output in Personalized mode
### What is this PR for?
Supporting helium package configurations. I attached screenshots.
#### Implementation details.
In case of spell, spell developer can create config spec in their `package.json` and it will be part of `helium.json` which is consumed by Zeppelin.
```
"config": {
"repeat": {
"type": "number",
"description": "How many times to repeat",
"defaultValue": 1
}
},
```
1. Persists conf per `package namepackage version` since each version can require different configs even if they are the same package.
2. Saves key-value config only. Since config spec (e.g `type`, `desc`, `defaultValue`) can be provided. So it's not efficient save both of them.
3. Extracts config related functions to `helium.service.js` since it can be used not only in `helium.controller.js` for view but also should be used in `paragraph.controller.js`, `result.controller.js` for executing spell.
### What type of PR is it?
[Feature]
### Todos
* [x] - create config view in `/helium`
* [x] - persist config per `packageversion`
* [x] - pass config to spell
### What is the Jira issue?
[ZEPPELIN-2069](https://issues.apache.org/jira/browse/ZEPPELIN-2069)
### How should this be tested?
- Build with examples `mvn clean package -Phelium-dev -Pexamples -DskipTests;`
- Open `/helium` page
- Update the `echo-spell` config
- Execute the spell like the screenshot below. (you don't need to refresh the page, since executing spell will fetch config from server)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#1982 from 1ambda/ZEPPELIN-2069/helium-package-configuration and squashes the following commits:
dbc4f10 [1ambda] fix: Add getAllPackageInfoWithoutRefresh
ce5f8c0 [1ambda] fix: Remove version 'local'
696f7f8 [1ambda] fix: DON'T serialize version field in HeliumPackage
e599ab9 [1ambda] feat: Close spell config panel after saving
c9b0145 [1ambda] feat: Make spell execution transactional
d9e87a8 [1ambda] refactor: Create API call for config
453016b [1ambda] fix: configExists
e6d5181 [1ambda] fix: Lint error
33a2bd8 [1ambda] refactor: HeliumService
f31bf3c [1ambda] feat: Add disabled class to cfg button while fetching
76d50ca [1ambda] fix: Use artifact as key of config
729c5ba [1ambda] fix: Remove digest from para ctrl
4d3c2c7 [1ambda] feat: Add config to framework, examples
70ebe29 [1ambda] feat: Pass confs to spell interpret()
115191e [1ambda] refactor: Extact spell related code to helium
3aa6c54 [1ambda] feat: Support helium conf in frontend
dea2929 [1ambda] chore: Add conf to example spells
6910e97 [1ambda] feat: Support config for helium pkg in backend
0a0c565 [1ambda] feat: Support config, version field for helium pkg
### What is this PR for?
This PR fixes test failure described in ZEPPELIN-2155.
All the PR test fails with this problem at the moment.
Problem came from https://github.com/apache/zeppelin/pull/2049, which fixes wrong authentication configuration for the test. And previously test were written for wrong configuration.
This PR fixes wrong test and remove some assert statement that may occur flaky test but don't really verify anything.
### What type of PR is it?
Hot Fix
### Todos
* [x] - fix problem
### What is the Jira issue?
### How should this be tested?
CI green
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#2081 from Leemoonsoo/ZEPPELIN-2155 and squashes the following commits:
0360108 [Lee moon soo] Correct test implementation with Authentication Enable
21d91ef [Lee moon soo] Handle multiple Set-Cookie headers
0ff3ff7 [Lee moon soo] Turn off Authentication on DirAccessTest because it is not necessary
52266a2 [Lee moon soo] Remove unnecessary assert
### What is this PR for?
Adding endpoint in NotebookRepoRestApi to trigger reload and broadcast of the note list.
Sending a GET request to /api/notebook-repositories/reload will trigger a reload of the note list, just like the reload button does on the homepage.
Q: If you think this endpoint belongs to another API (NotebookRestApi?), let me know.
### What type of PR is it?
Improvement
### Todos
### What is the Jira issue?
<https://issues.apache.org/jira/browse/ZEPPELIN-2114>
### How should this be tested?
0. add (or remove) a note in the notebook repository
1. curl -X GET http://HOST:PORT/api/notebook-repository/reload
2. verify that you the note list in Zeppelin's home page now contains (or does not contain anymore) the note.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? NotebookRepoRestApi does not have documentation yet. Could add it in another PR if needed.
Author: Remilito <remy.gayet@gmail.com>
Closes#2043 from Remilito/ZEPPELIN-2114 and squashes the following commits:
b1d741a [Remilito] adding log message for reload endpoint
cf66c2b [Remilito] [ZEPPELIN-2114] adding reload endpoint * ability to reload note list from repo by API
### What is this PR for?
When trying to delete a note of another user Zeppelin correctly throws a popup for insufficient privileges but has a small typo "removenote" (no space between remove note). Similar issue exist for renaming notebook "renamenote".
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
* [ZEPPELIN-2164](https://issues.apache.org/jira/browse/ZEPPELIN-2164)
### How should this be tested?
Refer before/after screen shot
### Screenshots (if appropriate)
Before:
<img width="628" alt="screen shot 2017-02-24 at 11 49 59 am 1" src="https://cloud.githubusercontent.com/assets/674497/23292690/66b808fa-fa87-11e6-9bde-4f9fbbb1cb67.png">
After:
<img width="645" alt="screen shot 2017-02-24 at 11 48 35 am" src="https://cloud.githubusercontent.com/assets/674497/23292691/66c0e9c0-fa87-11e6-9abc-16dc7b52ce3c.png">
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2064 from prabhjyotsingh/ZEPPELIN-2164 and squashes the following commits:
c4c8222 [Prabhjyot Singh] ZEPPELIN-2164: Typo in Insufficient Privileges popup of Zeppelin
### What is this PR for?
Zeppelin should redirect to login page after browser session expires
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
* [https://issues.apache.org/jira/browse/ZEPPELIN-2147](https://issues.apache.org/jira/browse/ZEPPELIN-2147)
### How should this be tested?
- Open any Zeppelin's notebook on a browser
- Restart zeppelin server
- now as soon as next web-socket message or API request is sent to server that returns either "ticket is invalid" or 405. This should get redirected to login page.
- On successful login it should take user back to the page from where he was coming from.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? N/A
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2058 from prabhjyotsingh/ZEPPELIN-2147 and squashes the following commits:
27bc361 [Prabhjyot Singh] user $timeout with $location
9189089 [Prabhjyot Singh] in case of session lost or logout, show login window
### What is this PR for?
This is a minor bug fix in ZEPPELIN-1586; add new line char before "[urls]"
### What type of PR is it?
[Minor Bug Fix]
### What is the Jira issue?
* [ZEPPELIN-1586](https://issues.apache.org/jira/browse/ZEPPELIN-1586)
### How should this be tested?
CI should be green.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? N/A
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2049 from prabhjyotsingh/ZEPPELIN-1586 and squashes the following commits:
f7e9c89 [Prabhjyot Singh] add new line char before "[urls]"
### What is this PR for?
Reducing size of InterpreterFactory and divide some functions from InterpreterFactory. Currently, InterpreterFactory has a lot of functions including management on InterpreterSetting and interpreter processes. This PR extracts InterpreterSetting-related parts into `InterpreterSettingManager`. It also has two unrelated functions: InterpreterSetting and InterpreterGroup. I'll treat it with another PR. This PR has some mechanical changes because it can help understand which parts has been changed. I'll do refactoring them later.
### What type of PR is it?
[Refactoring]
### Todos
* [x] - Extract all methods and variables related InterpreterSetting from InterpreterFactory
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2057
### How should this be tested?
Works exactly as same before
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#2001 from jongyoul/ZEPPELIN-2057 and squashes the following commits:
a3a844e [Jongyoul Lee] Set transient instance into InterpreterSetting
1377ef6 [Jongyoul Lee] Set transient object into value objects
0affba7 [Jongyoul Lee] Mapped the location of some functions related to InterpreterSetting again
c378c3f [Jongyoul Lee] Fixed the style
1e995f5 [Jongyoul Lee] Extracted InterpreterSetting from InterpreterFactory
6b304a9 [Jongyoul Lee] ing...
### What is this PR for?
This PR reduces travis build time
| Test profile | Before (rough avg min) | After (rough avg min) |
| ------------- |:-------------:| -----:|
| RAT check | 1.5 | 1.5 |
| All modules with spark 2.1 and scala 2.11 | 35 | 35 |
| All modules with spark 2.0 and scala 2.10 | 35 | 35 |
| Spark 1.6 with scala 2.10 | 35 | **15** |
| Spark 1.6 with scala 2.11 | 35 | **15** |
| Spark 1.5 | 25 | **removed** |
| Spark 1.4 | 25 | **removed** |
| Selenium | 25 | 25 |
| Python2 | 8 | 8 |
| Python3 | 8 | 8 |
| Livy | 15 | **8** |
| Total | 247.5 | 149 |
### What type of PR is it?
Improvement
### Todos
* [x] - Remove spark 1.5 profile and 1.4 profile
* [x] - optimize profiles
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2058
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#2003 from Leemoonsoo/ZEPPELIN-2058 and squashes the following commits:
7e50ea5 [Lee moon soo] create zeppelin-web/dist directory for test
b14b97e [Lee moon soo] prevent concurrent test execution in DirAccessTest
7860259 [Lee moon soo] reduce build time
bdcfd1f [Lee moon soo] Reduce log
### What is this PR for?
Spark interpreter prints unnecessary new line before the evaluation output is printed.
See https://github.com/apache/zeppelin/pull/1975#issuecomment-277581660.
This PR make SparkInterpreter ignores unnecessary preceding newline from ReplReporter.error()
### What type of PR is it?
Bug Fix
### Todos
* [x] - Ignore unnecessary preceding newline
* [x] - unittest
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2067
### How should this be tested?
run
```
%spark
import java.util.Date
import java.net.URL
```
and see if result looks like (no new line in front of each lines)
```
import java.util.Date
import java.net.URL
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#1981 from Leemoonsoo/ZEPPELIN-2067 and squashes the following commits:
1a31a66 [Lee moon soo] add issue id in the comment
23d5d55 [Lee moon soo] Add unittest
6908bdf [Lee moon soo] Ignore preceding newline from scala RepleReporter.error
### What is this PR for?
When IDE runs ZeppelinServer in debugging mode, it doesn't load httpclient because its scope is test but it needs zeppelin-zengine that zeppelin-server depends on. ZeppelinServer cannot run itself successfully while loading libraries.
### What type of PR is it?
[Bug Fix | Hot Fix]
### Todos
* [x] - Change the scope
### What is the Jira issue?
N/A
### How should this be tested?
Run ZeppelinServer in IntelliJ
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#1944 from jongyoul/hotfix/change-scope-of-zeppelin-server-pom and squashes the following commits:
b6dd68f [Jongyoul Lee] Remove httpcomponent dependency from zeppelin-server
c86393d [Jongyoul Lee] Change the scope of httpclient to runtime from test
### What is this PR for?
A paragraph execution may result in spark job(s).
Adding ability to access the spark job UI(corresponding to the job generated by the paragraph run), directly from the paragraph.
### What type of PR is it?
Improvement
### Todos
* [x] Write tests
### What is the Jira issue?
ZEPPELIN-1692
### How should this be tested?
Run paragraphs with spark code(scala, pyspark, sql, R).
The paragraph will display a button on the top right corner, which on click will open up the corresponding job UI
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NA
* Is there breaking changes for older versions? NA
* Does this needs documentation? NA
Author: Karup <karuppayya@outlook.com>
Author: karuppayya <karuppayya1990@gmail.com>
Author: karuppayya <karuppayyar@qubole.com>
Closes#1663 from karuppayya/ZEPPELIN-1692 and squashes the following commits:
4253d0b [Karup] Fix bad rebase
d7eb3b6 [Karup] Fix paragraph.js
8e2cd85 [Karup] tryout: fix selenium tests based on moons suggstion
732b0a4 [karuppayya] Fix test
890107d [Karup] Fix test - tryout
ed4685c [Karup] Fix tooltip
d27221d [Karup] Adding license header
87214a7 [Karup] Fix incorrect rebase
19513a6 [Karup] Send para runtimeinfos via websocker, but dont persist in json
09fc0e2 [Karup] Fix compilation
fc44d9b [Karup] Address review comments
b837c6c [karuppayya] Fix incorrect variable used
42d92ac [karuppayya] Fix test
d4e54e8 [karuppayya] Address review feedbacks
1a45284 [Karup] Fix test
717eedf [Karup] Add tests , refactor
25379aa [Karup] Clear job urls when we clear output
7383c0a [Karup] Address review comments
e2cd4db [karuppayya] Fix NPE in tests
3d9a573 [karuppayya] Fix NPE and some refactoring
9b3a3e2 [karuppayya] Fix checkstyle
f16422f [karuppayya] Ability to view spark job urls in each paragraph
### What is this PR for?
Added property for enable/disable public access to directories on server from Web
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2014
### How should this be tested?
Run application and try get list of files in app directory from web.
You will see a response with the code 403. Previously, we saw all files in the directory.
Change property "zeppelin.server.default.dir.allowed" to true and restart server.
Try again, all files should be visible.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Viktor Boginskii <Viktor_Boginskii@epam.com>
Closes#1962 from vboginskii/ZEPPELIN-2014 and squashes the following commits:
c06ec30 [Viktor Boginskii] [ZEPPELIN-2014] Added property for control public access to directories on server.
### What is this PR for?
Implemented **Spell** as one of Helium categories. *Technically, it's the frontend interpreter* runs on browser not backend.
Spell can provide many benefits.
1. Anyone can install, remove easily using helium package registry by #1936
2. Implementing spell is extremely easier rather than adding backend interpreter
3. Can use existing javsacript libraries. (e.g [flowchart.js](http://flowchart.js.org/), [sequence diagram js](https://github.com/bramp/js-sequence-diagrams), ...). This enable us to add many visualization tools. Imagine that you can implement some custom interpreters with few lines of code like [flowchart-spell-example](https://github.com/apache/zeppelin/compare/master...1ambda:ZEPPELIN-2008/introduce-spell?expand=1#diff-364845b20d68e4d94688e44fef03da98)
4. The most important thing is, spell is not only interpreter but also display system. Because it runs on browser. So we can use spell display system with another spell **Display System with Spell** (see the screenshot section below)
**In future**, we will be able to combine existing backend interpreters with spell like (**not supported in this PR cause we need to modify backend code a lot**)
```
// if we have markdown spell, we can use `%markdown` display in the spark interpreter
%spark
val calculated = doSomething()
println(s"%markdown _${calculated})
```
I added some examples. Checkout `echo`, `markdown`, `translator`, `flowchart` spells.
### What type of PR is it?
[Feature]
### Todos
* [x] - Add `SPELL` as one of Helium categories.
* [x] - Implement framework code (`zeppelin-spell`)
* [x] - Make some examples (flowchart, google translator, markdown, echo)
* [x] - Support custom display system
* [x] - Fix some bugs in `HeliumBundleFactory`
* [x] - Save spell rendering result into `note.json` while broadcasting to other websocket clients
* [x] - Fix `renderText` for stream output
### What is the Jira issue?
[ZEPPELIN-2008](https://issues.apache.org/jira/browse/ZEPPELIN-2008)
### How should this be tested?
- Build `mvn clean package -Phelium-dev -Pexamples -DskipTests;`
- Go to helium page `http://localhost:8080/#/helium`
- Enable all spells
- Go to a notebook and refresh
- Follow actions in the screenshots below.
### Screenshots (if appropriate)
#### Flowchart Spell (Sample)
#### Google Translator Spell (Sample)
#### Display System with Spell
### Questions:
* Does the licenses files need update - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - YES, but framework can be enhanced so i would like to defer to write document right now.
Author: 1ambda <1amb4a@gmail.com>
Closes#1940 from 1ambda/ZEPPELIN-2008/introduce-spell and squashes the following commits:
c1b5356 [1ambda] fix: RAT issues
e07ecd3 [1ambda] fix: Set width for spell usage
6c91892 [1ambda] feat: Display magic, usage for spell
5be2890 [1ambda] feat: Support spell info
822a1d8 [1ambda] style: Remove useless func wrap for helium
35d0fcc [1ambda] fix: Update desc for spell examples
49e03fc [1ambda] fix: List visualziation bundles only in order
4029c02 [1ambda] fix: ParagraphIT, parameterizedQueryForm
08eba10 [1ambda] refactor: renderGraph in result.controller.js
69ce880 [1ambda] fix: Resolve append (stream) output
0f2d8b6 [1ambda] fix: Resolve output issue
fc4389e [1ambda] fix: Resolve RAT issues
c8c8f0e [1ambda] fix: Add setErrorMessage method to Job
4fec44c [1ambda] refactor: NotebookServer.java
1227d7d [1ambda] refactor: result controller retry
9fb7438 [1ambda] feat: Save spell result and propagate
3cdf2da [1ambda] fix: NPM installation error
72aadbf [1ambda] feat: Enhance translator spell
bd2b3ef [1ambda] style: Rename generator -> data
cac0667 [1ambda] style: Rename to Spell
e81cb03 [1ambda] example: Add echo, markdown
0fa7eda [1ambda] feat: Support custom display
c906da6 [1ambda] feat: Update examples to use single FrontIntpRes
5c49e6e [1ambda] feat: Automated display type checking in result
5810bf1 [1ambda] feat: Apply frontend interpreter to paragraph
a163044 [1ambda] feat: Add flowchart, translator examples
247d00f [1ambda] feat: Add frontend interpreter framework
e925967 [1ambda] feat: Support FRONTEND_INTERPRETER type in frontend
c02d00a [1ambda] feat: Support FRONTEND_INTERPRETER type in backend
### What is this PR for?
Selenium test sometimes fails with
```
Caused by: org.openqa.selenium.NoSuchElementException: Unable to locate element: {"method":"xpath","selector":"//i[tooltip='WebSocket Connected']"}
```
This PR gives enough timeout (30s) for waiting 'WebSocket Connected' element.
Tested in my travis account 5 times and they're all green.
https://travis-ci.org/Leemoonsoo/zeppelin/builds/196428795https://travis-ci.org/Leemoonsoo/zeppelin/builds/196429337https://travis-ci.org/Leemoonsoo/zeppelin/builds/196429559https://travis-ci.org/Leemoonsoo/zeppelin/builds/196429593https://travis-ci.org/Leemoonsoo/zeppelin/builds/196430020
However, I'm not 100% sure if it really fixes the flaky test.
So, I'd like to merge this PR but keep ZEPPELIN-2026 open for next 1 week.
And then see if no selenium test fails with the error for next 1 week to close ZEPPELIN-2026.
### What type of PR is it?
Improvement
### Todos
* [x] - Increase timeout of detecting 'connected'
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2026
### How should this be tested?
CI green on Selenium test profile
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#1959 from Leemoonsoo/ZEPPELIN-2026 and squashes the following commits:
87432d5 [Lee moon soo] increase timeout
### What is this PR for?
ZEPPELIN-1973 will provides catalogue for all available helium (visualization) packages in npm registry. And https://github.com/apache/zeppelin/pull/1935 shows available packages in Zeppelin website.
This PR make Zeppelin reads package information and display in Zeppelin's helium gui menu.
To do that, this PR changes configuration environment variable (java property) from
```
ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT (zeppelin.helium.localregistry.default)
```
to
```
ZEPPELIN_HELIUM_REGISTRY (zeppelin.helium.registry)
```
and allow multiple comma separated items.
Registry is either filesystem directory (e.g. `/helium`) or http location.
default value is `helium,https://s3.amazonaws.com/helium-package/helium.json`
### What type of PR is it?
Feature
### Todos
* [x] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2004
### How should this be tested?
Go to helium menu and check if you can see packages available.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? yes
`ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT` changed to `ZEPPELIN_HELIUM_REGISTRY`
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#1936 from Leemoonsoo/ZEPPELIN-2004 and squashes the following commits:
80bbebd [Lee moon soo] update doc
53f0caa [Lee moon soo] update test
65c6092 [Lee moon soo] Cache online registry under local-repo for offline support
b2985e9 [Lee moon soo] Add user agent
f562b12 [Lee moon soo] Use only local helium registry to run test
9fabeae [Lee moon soo] implement online registry
### What is this PR for?
This is to solve the problem with cron job scheduling. basically after 6177c819b1 all empty paragraphs are skipped when executing, but if the paragraph had status `READY` that status will be stayed same and then the note won't be considered as terminated in [here](https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java#L866) resulting on keeping cron threads up running. So after all 10 threads in the threadpool are exhausted, new jobs are not scheduled. Here i change the status of paragraph to `FINISHED` when it's empty and skipping run.
### What type of PR is it?
Bug Fix
### Todos
* [x] - set status to `FINISHED`
### What is the Jira issue?
[ZEPPELIN-2009](https://issues.apache.org/jira/browse/ZEPPELIN-2009)
### How should this be tested?
try to schedule note using cron expression (e.g. `0/5 * * * * ?` every 5 secs) before and after this PR
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Khalid Huseynov <khalidhnv@gmail.com>
Closes#1941 from khalidhuseynov/fix/cron-scheduling and squashes the following commits:
b39cb2a [Khalid Huseynov] fix notebook rest api test
48fd94e [Khalid Huseynov] fix test
54e5ce9 [Khalid Huseynov] set status FINISHED on empty paragraph
### What is this PR for?
This PR implements interpreter output message limit.
`ZEPPELIN_INTERPRETER_OUTPUT_LIMIT` env variable or `zeppelin.interpreter.output.limit` jvm property can set limit of the interpreter output message in byte.
The limit applied to only TEXT and TABLE type output, not in HTML or other types.
### What type of PR is it?
Improvement
### Todos
* [x] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1976
### How should this be tested?
try to print more than the limit
```
%spark
(1 to 10000).foreach(i=>
println(s"Print line ${i} times")
)
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#1908 from Leemoonsoo/ZEPPELIN-1976 and squashes the following commits:
639868b [Lee moon soo] update description
a9b4139 [Lee moon soo] Truncate output
### What is this PR for?
Try fix problem described in https://issues.apache.org/jira/browse/ZEPPELIN-1986.
Almost all recent build has this failure on CI build.
### What type of PR is it?
Hot Fix
### Todos
* [x] - Increase timeout
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1986
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#1919 from Leemoonsoo/ZEPPELIN-1986-fix and squashes the following commits:
2a7c1ba [Lee moon soo] Increase timeout from AbstractFunctionalSuite, too
21e6840 [Lee moon soo] Increase WelcomePageSuite timeout
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
### What type of PR is it?
Documentation | change
### Todos
- [x] replace to doc
- [x] change response 201 -> 200
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1869
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: cloverhearts <cloverheartsdev@gmail.com>
Closes#1814 from cloverhearts/ZEPPELIN-STATUS-CHANGE-API and squashes the following commits:
0f2f5bd [cloverhearts] change test case in notebooksecurity
1be0968 [cloverhearts] fixed more isCreated to isAllowed
cfcb6fd [cloverhearts] change test case in zeppelin rest api test
7bbfcd1 [cloverhearts] change status value in test case http
f9fcbc8 [cloverhearts] Revert "refix"
ed31788 [cloverhearts] refix
754309e [cloverhearts] Apply to api success status code 200
### What is this PR for?
webpack.config.js for creating visualization bundle has unnecessary extension configuration, which makes unable to import some libraries.
This PR removes the unnecessary configuration and propagate 'npm install' error message to front-end.
### What type of PR is it?
Bug Fix
### Todos
* [x] - exclude 'extensions' from webpack.config.js for visualization bundle
* [x] - propagate 'npm install' error to front-end
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1974
### Screenshots (if appropriate)
Before propagate error message to front-end
After propagate error message to front-end
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#1905 from Leemoonsoo/npm_install_error and squashes the following commits:
b665588 [Lee moon soo] Propagate npm install error to front-end
7635c55 [Lee moon soo] Remove extension from webpack config for visualization bundle
### What is this PR for?
The issue is that spark 2.1 use `commons-lang3` 3.5 while `zeppelin-interpreter` use 3.4. We can not just upgrade `commons-lang3` to 3.5, as it just make spark 2.1 work, but would fail other versions of spark. This PR remove `commons-lang3` from zeppelin-interpreter. We should keep zeppelin-interpreter's dependencies as minimum as possible. We can remove other dependencies (like `commons-lang`) from `zeppelin-interpreter` in a followup PR.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1977
### How should this be tested?
Tested manually
on 3 versions of spark (2.1, 2.0.2, 1.6.2)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#1910 from zjffdu/ZEPPELIN-1977 and squashes the following commits:
8bac97f [Jeff Zhang] ZEPPELIN-1977. spark 2.1 uses a more recent commons-lang3
### What is this PR for?
Bump up version to 0.8.0-SNAPSHOT
Author: Mina Lee <minalee@apache.org>
Closes#1883 from minahlee/0.8.0-SNAPSHOT and squashes the following commits:
b28ebe8 [Mina Lee] Bump up table/visualization plugin package version
fcaaa46 [Mina Lee] Bump up helium-dev version
180d2c6 [Mina Lee] Bump up version to 0.8.0-SNAPSHOT
### What is this PR for?
This PR would only restart the trigger user's interpreter rather than all the interpreter. So that restarting won't affect other users.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1770
### How should this be tested?
Tested manually.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#1846 from zjffdu/ZEPPELIN-1770 and squashes the following commits:
5ee076d [Jeff Zhang] fix scoped mode and add unit test
8cb28a3 [Jeff Zhang] ZEPPELIN-1770. Restart only the client user's interpreter when restarting interpreter setting
### What is this PR for?
Current helium plugin application api (experimental) requires create library in java class, and need to create both backend / frontend code in the package. Which is good if your plugin requires both frontend and backend code running.
However, when user just want to make new visualization which completely runs on front-end side in javascript, creating helium application in java project and taking care of backend code can be bit of overhead and barrier for javascript developers.
This PR adds capability to load pure javascript package as a visualization.
### how it works
1. create (copy, download) 'helium package json' file into `ZEPPELIN_HOME/helium` directory.
The json file point visualization js package in npm repository or local file system in `artifact` field.
`type` field in the json file need to be `VISUALIZATION`
Here's an example (zeppelin-examples/zeppelin-example-horizontalbar/zeppelin-example-horizontalbar.json)
```
{
"type" : "VISUALIZATION",
"name" : "zeppelin_horizontalbar",
"description" : "Horizontal Bar chart (example)",
"artifact" : "./zeppelin-examples/zeppelin-example-horizontalbar",
"icon" : "<i class='fa fa-bar-chart rotate90flipX'></i>"
}
```
2. Go to helium GUI menu. (e.g. http://localhost:8080/#/helium).
The menu will list all available packages.
<img width="796" alt="writing_visualization_helium_menu" src="https://cloud.githubusercontent.com/assets/1540981/21749660/0f401c10-d558-11e6-9961-b6d0a9c023d8.png">
3. click 'enable' in any package want to use.
Once a visualization package is enabled, `HeliumVisualizationFactory` will collect all enabled visualizations and create js bundle on the fly.
4. js bundle will be loaded on notebook and additional visualization becomes available
### Programming API to create new plugin visualization.
Simply extends [visualization.js](https://github.com/apache/zeppelin/blob/master/zeppelin-web/src/app/visualization/visualization.js) and overrides some methods, such as
```
/**
* get transformation
*/
getTransformation() {
// override this
};
/**
* Method will be invoked when data or configuration changed
*/
render(tableData) {
// override this
};
/**
* Refresh visualization.
*/
refresh() {
// override this
};
/**
* method will be invoked when visualization need to be destroyed.
* Don't need to destroy this.targetEl.
*/
destroy() {
// override this
};
/**
* return {
* template : angular template string or url (url should end with .html),
* scope : an object to bind to template scope
* }
*/
getSetting() {
// override this
};
```
This is exactly the same api that built-in visualization uses.
an example implementation included `zeppelin-examples/zeppelin-example-horizontalbar/horizontalbar.js`.
Actually [all built-in visualizations](https://github.com/apache/zeppelin/tree/master/zeppelin-web/src/app/visualization/builtins) are example
### Packaging and publishing visualization
Each visualization will need `package.json` file (e.g. `zeppelin-examples/zeppelin-example-horizontalbar/package.json`) to be packaged.
Package can be published in npm repository or package can be deployed to the local filesystem.
`zeppelin-examples/zeppelin-example-horizontalbar/` is an example package that is deployed in the local filesystem
### Development mode
First, locally install and enable your development package by setting `artifact` field to the development directory.
Then run zeppelin-web in visualization development mode with following command
```
cd zeppelin-web
npm run visdev
```
When you have change in your local development package, just reload your notebook. Then Zeppelin will automatically rebuild / reload the package.
Any feedback would be appreciated!
### What type of PR is it?
Feature
### Todos
* [x] - Load plugin visualization js package on runtime
* [x] - Make the feature works in zeppelin Binary package
* [x] - Show loading indicator while 'enable' / 'disable' package
* [x] - Add document
* [x] - Add license of new dependency
* [x] - Development mode
* [x] - Propagate error to front-end
* [x] - Display multiple versions of a package.
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1619
### How should this be tested?
Build Zeppelin with `-Pexamples` flag. That'll install example visualization package `horizontalbar`.
You'll able to select `horizontalbar` along with other built-in visualizations
To test npm online package install capability, Place [zeppelin-bubble.json](https://github.com/Leemoonsoo/zeppelin-bubble/blob/master/zeppelin-bubble.json) in hour local registry (`ZEPPELIN_HOME/helium`) and enable it in Helium gui menu.
And then zeppelin will download package from npm repository and load.
### Questions:
* Does the licenses files need update? yes
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes#1842 from Leemoonsoo/ZEPPELIN-1619-rebased and squashes the following commits:
7c49bbb [Lee moon soo] Let Zeppelin continue to bootstrap on offline
816bdec [Lee moon soo] Display license of package when enabling
28fb37d [Lee moon soo] beautifulize helium menu
295768e [Lee moon soo] fix drag and drop visualization reorder
bb304db [Lee moon soo] Sort version in decreasing order
e7f18f1 [Lee moon soo] fix english in docs and labels
c7b187f [Lee moon soo] Merge branch 'master' into ZEPPELIN-1619-rebased
4c87983 [Lee moon soo] Merge remote-tracking branch 'apache-github/master' into ZEPPELIN-1619-rebased
ecd925b [Lee moon soo] Merge remote-tracking branch 'apache-github/master' into ZEPPELIN-1619-rebased
a92cadd [Lee moon soo] Use minifiable syntax
cec534c [Lee moon soo] Reduce log message
f373f1d [Lee moon soo] Ignore removed package
e18d9a4 [Lee moon soo] Ability to customize order of visualization package display
cd74396 [Lee moon soo] Add rest api doc
9de5d6d [Lee moon soo] exclude .npmignore and package.json from zeppelin-web rat check
08abded [Lee moon soo] exclude package.json from rat check
661c26b [Lee moon soo] update screenshot and keep experimental tag only in docs
4515805 [Lee moon soo] Display multiple versions of a package
408c512 [Lee moon soo] Make unittest test bundling with proper vis package on npm registry
fb7a147 [Lee moon soo] display svg icon
47de6d9 [Lee moon soo] Propagate bundle error to the front-end
0fe5e00 [Lee moon soo] visualization development mode
022e8f6 [Lee moon soo] exclude zeppelin-examples/zeppelin-example-horizontalbar/package.json file from rat check
2ef3b69 [Lee moon soo] Add new dependency license
f943d33 [Lee moon soo] Add doc
f494dbd [Lee moon soo] package npm dependency module in binary package
b655fa6 [Lee moon soo] use any version of dependency in example. so zeppelin version bumpup doesn't need to take care of them
2aec52d [Lee moon soo] show loading indicator while enable/disable package
6c380f6 [Lee moon soo] refactor code to fix HeliumTest
e142336 [Lee moon soo] update unittest
7d5e0ae [Lee moon soo] Resolve dependency conflict
c50a524 [Lee moon soo] Add conf/helium.json in .gitignore
1c7b73a [Lee moon soo] add result.css
d2823ad [Lee moon soo] load visualization and tabledata module from source instead npm if accessible
4e1b061 [Lee moon soo] Convert horizontalbar to VISUALIZATION example
a5a935b [Lee moon soo] connect visualization factory with restapi
4b21252 [Lee moon soo] initial implementation of helium menu
0c4da2e [Lee moon soo] pass bundled visualization to result.controller.js
f5ce99e [Lee moon soo] import helium service js
1663582 [Lee moon soo] initial implementation of helium menu
74d52d4 [Lee moon soo] bundle visualization package from npm repository on the fly
### What is this PR for?
Remove unused imports;
Replace "Thread.sleep" (unstable solution);
Add "deleteTestNotebook()" method -> testWidth.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
Travis-CI
### Screenshots (if appropriate)
https://travis-ci.org/apache/zeppelin/builds/190644901 -> https://travis-ci.org/apache/zeppelin/jobs/190644918
### Questions:
* Does the licenses files need update? (no)
* Is there breaking changes for older versions? (no)
* Does this needs documentation? (no)
Author: Unknown <Sokurito@gmail.com>
Closes#1882 from bitchelov/automationTestFix and squashes the following commits:
f898c59 [Unknown] [MINOR] Small Selenium test fixes
### What is this PR for?
DON't show the multiple same dialog when user doesn't have permission for **Run all paragraphs** inside a note
#### Implementation details
- Introduce new websocket message `RUN_ALL_PARAGRAPHS` since we need to broadcast *bringing dialog* message only once
(We did same thing about `CLEAR_ALL_PARAGRAPHS`)
- Refactor `NotebookServer.runParagraph` to avoid duplication
- Add necessary functions to backend and frontend
### What type of PR is it?
[Bug Fix]
### Todos
Fixed at once
### What is the Jira issue?
[ZEPPELIN-1910](https://issues.apache.org/jira/browse/ZEPPELIN-1910)
### How should this be tested?
1. Set permission to a note
2. Enable shiro and login as an user **who doens't have write permission**
3. Click *Run all paragraphs* button
### Screenshots (if appropriate)
#### 1. Before
#### 2. After
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#1852 from 1ambda/ZEPPELIN-1910/run-all-para-bring-multi-dialog and squashes the following commits:
837b7be [1ambda] fix: func name in ParagraphActionsIT
c200585 [1ambda] fix: zeppeiln-web test for runNote
284e8e6 [1ambda] fix: Multiple dialog when don't have permission for run all para
