### What is this PR for?
minor fix to show correct package name `ipykernel` in logs when prerequisite for ipython interpreter isn't met.
### What type of PR is it?
[ Improvement ]
### Todos
* [x] - fix naming
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
if `ipykernel` isn't installed then it would show correct message in logs when running ipython interpreter
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Khalid Huseynov <khalidhnv@gmail.com>
Closes#3046 from khalidhuseynov/fix/ipykernel-msg and squashes the following commits:
7a3b124f9 [Khalid Huseynov] fix minor
b6a3d5c5f [Khalid Huseynov] fix package name shown to user when not installed
### What is this PR for?
This is to add password for dynamic forms. It is almost the same as input box but but with invisible input. And there's no default value for password form, as it doesn't make sense to provide default value for password.
### What type of PR is it?
[ Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2528
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3041 from zjffdu/ZEPPELIN-2528 and squashes the following commits:
084abd39c [Jeff Zhang] ZEPPELIN-2528. Add a password text input to the ZeppelinContext
### What is this PR for?
After ZEPPELIN-3375, PySparkInterpreter extends PythonInterpreter. PySparkInterpreter may fail to bootstrap PySpark as we miss to add hook to the pyspark bootstrap code.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3538
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3016 from zjffdu/ZEPPELIN-3538 and squashes the following commits:
3a1d8a737 [Jeff Zhang] ZEPPELIN-3538. Fail to bootstrap PySpark in yarn cluster mode
### What is this PR for?
Fix the code style issue of python module
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3163
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3023 from zjffdu/ZEPPELIN-3163 and squashes the following commits:
e89c41108 [Jeff Zhang] [ZEPPELIN-3163]. Fix checkstyle of python
### What is this PR for?
This PR introduce the bi directional RPC framework between ZeppelinServer and InterpreterProcess on top of thrift, see the design doc here. https://docs.google.com/document/d/1B_4b7UuvE7RwbfpTjxV2RSxdDeF-p9vXVYC8ace76-Q/edit?usp=sharing
This PR is pretty big, most of the code changes is due to regeneration of thrift code.
Another improvement is introduce the builder pattern of InterpreterContext as the arguments of InterpreterContext constructor is too many.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2035
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2986 from zjffdu/ZEPPELIN-2035 and squashes the following commits:
c02d5587a [Jeff Zhang] ZEPPELIN-2035. BI directional RPC framework between ZeppelinServer and InterpreterProcess on top of thrift
[ZEPPELIN-3423] Fix deprecated dynamic forms document URL on Python README.md
### What is this PR for?
Python dynamic forms document URL is based on "0.6.0-SNAPSHOT" and already deprecated.
Replace the deprecated URL with the latest URL.
### What type of PR is it?
Documentation
### Todos
N/A
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3423
### How should this be tested?
N/A
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Changhyup Jwa <jwagun@gmail.com>
Closes#2939 from jwagun/ZEPPELIN-3423 and squashes the following commits:
360261c2f [Changhyup Jwa] [ZEPPELIN-3423] Fix deprecated dynamic forms document URL on Python README.md
Change-Id: I540a222678a82d6f88d91bf8790acf4fca19d992
### What is this PR for?
This PR is trying to remove the code duplication between PySparkInterpreter and PythonInterpreter. So here's the main things this PR did:
* PySparkInterpreter extends PythonInterpreter
* PySparkInterpreterTest extends PythonInterpreterTest so that we can verify PySparkInterpreter can do whatever PythonInterpreter can do
* Move interpreter/lib/python/backend_zinline.py and interpreter/lib/python/mpl_config.py into python module, so that python module can ship these resources together.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3375
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2919 from zjffdu/ZEPPELIN-3375 and squashes the following commits:
738c6c5 [Jeff Zhang] ZEPPELIN-3375. Make PySparkInterpreter extends PythonInterpreter
This is what I see in zeppelin and jupyter. The difference is due to zeppelin also add code completion of ace editor. I think code completion of ace editor should only be enabled when the backend interpreter doesn't support completion.
[Bug Fix]
* [ZEPPELIN-3001](https://issues.apache.org/jira/browse/ZEPPELIN-3001)
* Does the licenses files need update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? N/A
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2885 from prabhjyotsingh/ZEPPELIN-3001 and squashes the following commits:
fcd551fe1 [Prabhjyot Singh] meta can be either empty string or undefined
e6ad8d1c7 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-3001
508cd239c [Prabhjyot Singh] fix websocketMsgSrv.completion paragraph id issue
9380c32fa [Prabhjyot Singh] upgrade ace-builds to 1.3.2
701f93f85 [Prabhjyot Singh] make default suggestion if there is no suggestion from back-end.
97f594dc8 [Prabhjyot Singh] completionSupport false for SparkRInterpreter
d2f70ae7d [Prabhjyot Singh] get completionSupport from interpreter (backend)
adaabcbf9 [Prabhjyot Singh] ZEPPELIN-3001: Disable ace editor code completion when back-end interpreter support completion
Change-Id: I03e39422bafebf9c96b01b88f542a16dd6959f23
### What is this PR for?
This PR address the remaining issue of ZEPPELIN-3362, and also add more test for ZeppelinContext.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3364
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2894 from zjffdu/ZEPPELIN-3364 and squashes the following commits:
891f1e1 [Jeff Zhang] ZEPPELIN-3364. Followup of ZEPPELIN-3362, improve ZeppelinContext & add more test
### What is this PR for?
Unify the ZeppelinContext of PythonInterpreter & IPythonInterpreter into one file to avoid code duplication.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3362
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2890 from zjffdu/ZEPPELIN-3362 and squashes the following commits:
b5dcbc9 [Jeff Zhang] ZEPPELIN-3362. Unify ZeppelinContext of PythonInterpreter & IPythonInterpreter
### What is this PR for?
Trivial PR for removing the unnecessary message
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3290
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2882 from zjffdu/ZEPPELIN-3290 and squashes the following commits:
7bf8d6e [Jeff Zhang] ZEPPELIN-3290. Unnecessary message 'IPython is available, use IPython for PySparkInterpreter'
### What is this PR for?
Currently the interpreter hook mechanism is broken. This PR fix this issue and also add unit test for interpreter hook, (including global hook and note level hook)
### What type of PR is it?
[Bug Fix | Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3343
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2875 from zjffdu/ZEPPELIN-3343 and squashes the following commits:
3c55f07 [Jeff Zhang] ZEPPELIN-3343. Interpreter Hook is broken
### What is this PR for?
```
%python
import time
print("Hello")
time.sleep(0.5) # in case of Ipython kernel, print may not immediately flushed and cleared.
z.getInterpreterContext().out().clear()
print("world")
```
Expected to print `world` only.
This worked in zeppelin-0.7.x and it'll be nice keep this feature in the future version.
### What type of PR is it?
Bug fix | Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3284
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Lee moon soo <moon@apache.org>
Closes#2829 from Leemoonsoo/ZEPPELIN-3284 and squashes the following commits:
e0dcda61f [Lee moon soo] make z.getInterpreterContext().out().clear() work in IPythonInterprete
224a2df8e [Lee moon soo] make z.getInterpreterContext().out().clear() work in PythonInterpreter
### What is this PR for?
Remove Redundant method `InterpreterOutput.getInterpreterResultMessages()`
### What type of PR is it?
Refactoring
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3283
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#2828 from Leemoonsoo/ZEPPELIN-3283 and squashes the following commits:
12771a145 [Lee moon soo] remove unused import
82bda9c02 [Lee moon soo] Remove redundant method getInterpreterResultMessages()
### What is this PR for?
trivial change for property name renaming. Followup of #2802
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2813 from zjffdu/HOTFIX_GRPC and squashes the following commits:
d899a29 [Jeff Zhang] [HOTFIX] rename zeppelin.ipython.grpc.framesize to zeppelin.ipython.grpc.message_size
### What is this PR for?
Add one new property `zeppelin.ipython.grpc.framesize` which is an advanced configuration. By default it is 32M which should be sufficient for most of scenarios.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3236
### How should this be tested?
* Unit test is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2802 from zjffdu/ZEPPELIN-3236 and squashes the following commits:
ffce774 [Jeff Zhang] ZEPPELIN-3236. Make grpc framesize configurable
### What is this PR for?
Fix the unicode issues in python2.
### What type of PR is it?
[Bug Fix | Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3239
### How should this be tested?
* Unit test is added
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2810 from zjffdu/ZEPPELIN-3239 and squashes the following commits:
70c8b4c [Jeff Zhang] ZEPPELIN-3239. unicode characters in an iPython paragraph makes Spark interpreter irrsponsive
### What is this PR for?
Enhance the ZeppelinContext in IPySparkInterpreter
### What type of PR is it?
[Bug Fix | Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3234
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2807 from zjffdu/ZEPPELIN-3234 and squashes the following commits:
39637ee [Jeff Zhang] ZEPPELIN-3234. z.show() compatibility with previous release
### What is this PR for?
This is for the refactoring of SparkInterpreter. See design doc. https://docs.google.com/document/d/1AfGg3aGXonDyri1jrP4MMFT4Y4j3wpN1t8kL-GAKSUc/edit?usp=sharing
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3111
### How should this be tested?
* Unit test is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2709 from zjffdu/ZEPPELIN-3111 and squashes the following commits:
aae4b09 [Jeff Zhang] ZEPPELIN-3111. Refactor SparkInterpreter
### What is this PR for?
trivial pom file change
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3192
### How should this be tested?
* Travis pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2748 from zjffdu/ZEPPELIN-3192 and squashes the following commits:
635acde [Jeff Zhang] ZEPPELIN-3192. Bump up Zeppelin version to 0.9.0-SNAPSHOT
### What is this PR for?
This is for better code completion support for IPythonInterpreter
### What type of PR is it?
[Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3136
### How should this be tested?
* Unit test is updated and Manually verified.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2725 from zjffdu/ZEPPELIN-3136 and squashes the following commits:
1179fc8 [Jeff Zhang] ZPEPELIN-3136. IPython Code Completion Improvement
### What is this PR for?
Added dynamic forms for Note. All paragraphs has access to note forms.
### What type of PR is it?
[Feature]
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1363
### How should this be tested?
Create global interpreter JDBC and Spark. Create note with 2 paragraphs
```
%pyspark
print("Textbox paragraph " + z.textbox('input', 'default'))
print("Textbox note " + z.noteTextbox('note_input', 'default_note'))
print("Select paragraph " + z.select("sel", [("1","opt1"), ("2","opt2"), ("3","opt3")]))
print("Select note " + z.noteSelect("sel_note", [("1","noteOpt1"), ("2","noteOpt2"), ("3","noteOpt3")]))
options = [("key1","Name1"), ("key2","Name2")]
print("Checkbox paragraph "+ " and ".join(z.checkbox("chk", options, ["key1"])))
print("Checkbox note "+ " and ".join(z.noteCheckbox("chk_note", options, ["key1","key2"])))
```
```
%jdbc
select '$${checkbox:chk_note=key1|key2,key1|key2} $${note_input} ${note_input=sameName}'
```
### Screenshots (if appropriate)
1) native forms
2) remove
3) simple forms
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: tinkoff-dwh <tinkoff.dwh@gmail.com>
Closes#2641 from tinkoff-dwh/ZEPPELIN-1363 and squashes the following commits:
3ee4826 [tinkoff-dwh] [ZEPPELIN-1363] autosave textbox
f30033a [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1363
29eaca2 [tinkoff-dwh] [ZEPPELIN-1363] fix tests
bf8194e [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1363
9b2f3e9 [tinkoff-dwh] [ZEPPELIN-1363] button to remove form
c566462 [tinkoff-dwh] [ZEPPELIN-1363] note dynamic forms (simple; native: spark, python)
### What is this PR for?
This PR will add tab as auto complete invoker if paragraph is non-md and user has not pressed the tab as a first character in the line
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-277
* https://issues.apache.org/jira/browse/ZEPPELIN-2736
### How should this be tested?
- Build: mvn clean package -Denforcer.skip -DskipTests -Drat.skip
- Open a paragraph
- Press tab with following options: first character, after space
### Questions:
* Does the licenses files need an update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Malay Majithia <malay.majithia@gmail.com>
Author: Lee moon soo <moon@apache.org>
Closes#2542 from malayhm/ZEPPELIN-277 and squashes the following commits:
436f22d [Malay Majithia] Added Tab auto completion flag for python sql and r
b37e084 [Malay Majithia] Fixed lint error
18fc814 [Malay Majithia] Merge branch 'master' into ZEPPELIN-277
b09730e [Malay Majithia] Merge branch 'master' into ZEPPELIN-277
63d69e1 [Malay Majithia] Merge branch 'ZEPPELIN-277' of github.com:malayhm/zeppelin into ZEPPELIN-277
a75f0fe [Malay Majithia] Improved the first character check logic
2ec879d [Malay Majithia] Merge pull request #1 from Leemoonsoo/ZEPPELIN-277-completion-key
77afdba [Lee moon soo] fix style
77b47b6 [Malay Majithia] If all the previous line characters are tab, don't show autocomplete on tab
46f612a [Malay Majithia] ZEPPELIN-277 Add Tab as Autocomplete for Notebook non-md paragraphs
865c0a6 [Lee moon soo] Set python and spark interpreter completionKey
05d5860 [Lee moon soo] Update doc
973068b [Lee moon soo] apply tab completion based on editor.completionKey
5f4d81c [Malay Majithia] If all the previous line characters are tab, don't show autocomplete on tab
655ba88 [Malay Majithia] ZEPPELIN-277 Add Tab as Autocomplete for Notebook non-md paragraphs
### What is this PR for?
Trivial fix for matplotlib fail in `IPythonInterpreter`. Besides that, I also make some minor changes on zeppelin code to improve logging.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2990
### How should this be tested?
UT is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2622 from zjffdu/ZEPPELIN-2990 and squashes the following commits:
55e6f88 [Jeff Zhang] fix unit test
1e57afe [Jeff Zhang] ZEPPELIN-2990. Matplotlib sometimes fails in IPythonInterpreter
### What is this PR for?
Follow up of #2577. Main changes on Interpreter
* Add throw `InterpreterException` which is checked exception for the abstract methods of `Interpreter`, this would enforce the interpreter implementation to throw `InterpreterException`.
* field name refactoring.
* `property` -> `properties`
* `getProperty()` --> `getProperties()`
* Introduce launcher layer for interpreter launching. Currently we only use shell script to launch interpreter, but it could be any other service or component to launch interpreter, such as livy server , other 3rd party tools or even we may create a separate module for interpreter launcher
* abstract cass `InterpreterLauncher`
* For now, only 2 implementation: `ShellScriptLauncher` & `SparkInterpreterLauncher`. We could add method in class `Interpreter` to allow interpreter to specify its own launcher class, but it could be future work.
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2685
### How should this be tested?
Unit test is covered. `ShellScriptLauncherTest` & `SparkInterpreterLauncherTest`
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2592 from zjffdu/ZEPPELIN-2685 and squashes the following commits:
17dc2f1 [Jeff Zhang] address comments
e545cc3 [Jeff Zhang] ZEPPELIN-2685. Improvement on Interpreter class
### What is this PR for?
2 main changes.
* Introduce `interpreter-parent` module, so that all the interpreter can reuse the plugin defined in `interpreter-parent`
* Add new plugin for `copying interpreter-setting.json` to interpreter dir
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2982
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2613 from zjffdu/ZEPPELIN-2982 and squashes the following commits:
ccccd32 [Jeff Zhang] update doc
8f643a7 [Jeff Zhang] ZEPPELIN-2982. Copy interpreter-setting.json to interpreter dir
### What is this PR for?
Overview : The random number generator implemented by nextInt() cannot withstand a cryptographic attack.
In the file IdHashes.java similar issues were on line numbers 74
Comments : Use of SecureRandom is recommended
### What type of PR is it?
[Refactoring]
### What is the Jira issue?
* [https://issues.apache.org/jira/browse/ZEPPELIN-2970](https://issues.apache.org/jira/browse/ZEPPELIN-2970)
### How should this be tested?
CI should be green
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2606 from prabhjyotsingh/ZEPPELIN-2970 and squashes the following commits:
e1a9d3a5f [Prabhjyot Singh] Change all "java.util.Random" to "java.security.SecureRandom"
f3f22e803 [Prabhjyot Singh] use SecureRandom instead of Random
### What is this PR for?
Straightforward fix to set the timeout as 100 years so that spark streaming app won't be stopped unless 100 years later.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2943
### How should this be tested?
Tested manually, see screenshot.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2593 from zjffdu/ZEPPELIN-2943 and squashes the following commits:
9144409 [Jeff Zhang] Fix a test bug
0923b3c [Jeff Zhang] ZEPPELIN-2943 Streaming output will be closed in 30 seconds for IPythonInterpreter
### What is this PR for?
This is the first version for supporting yarn-cluster of `SparkInterpreter`. I just delegate all the function to `spark-submit` as yarn-cluster is natively supported by spark, we don't need to reinvent the wheel. But there's still improvement to be done in future, e.g. I put some spark specific logic in `InterpreterSetting` which is not a good practise. I plan to improve it when I refactor the `Interpreter` class (ZEPPELIN-2685).
Besides that, I also add `MiniHadoopCluster` & `MiniZeppelin` which help for the integration test of yarn-client & yarn-cluster mode, otherwise I have to manually verify yarn-client & yarn-cluster mode which would easily cause regression issue in future.
To be noticed:
* SPARK_HOME must be specified for yarn-cluster mode
* HADOOP_CONF_DIR must be specified for yarn-cluster mode
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
https://github.com/zjffdu/zeppelin/tree/ZEPPELIN-2898
### How should this be tested?
System test is added in `SparkInterpreterIT`.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2577 from zjffdu/ZEPPELIN-2898 and squashes the following commits:
9da7c4b [Jeff Zhang] ZEPPELIN-2898. Support Yarn-Cluster for Spark Interpreter
### What is this PR for?
Fixed a problem where conda's library was not installed and reflected properly due to changed python interpreter execution flow and structures.
ORIGINAL : https://github.com/apache/zeppelin/pull/2574
### What type of PR is it?
Bug Fix
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2921
### How should this be tested?
Please run the following command line for each paragraph.
If the module such as scipy is normally imported, it is a success.
`%python.conda create --name Hello6 python=2.7`
`%python.conda activate Hello6`
`%python.conda install seaborn pandas numpy scipy matplotlib`
```
%python
import scipy as sp
import seaborn as sns
```
`%python.conda deactivate`
### Screenshots (if appropriate)
#### Before
#### After
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: CloverHearts <cloverheartsdev@gmail.com>
Closes#2578 from cloverhearts/ZEPPELIN-2921 and squashes the following commits:
a09f6801 [CloverHearts] added test case for conda env
9b797269 [CloverHearts] add conda env option on list
8dfb33b1 [CloverHearts] replace conda install environment name
### What is this PR for?
This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well.
For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter.
Main features:
* IPython interpreter support
** All the ipython features are available, including visualization, ipython magics.
* ZeppelinContext support
* Streaming output support
* Support Ipython in PySpark
Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here.
In unit test, I only verify the following 3 popular visualization library. could add more later.
* matplotlib
* bokeh
* ggplot
### What type of PR is it?
[Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2753
### How should this be tested?
Unit test is added.
### Screenshots (if appropriate)
Verify bokeh in IPython Interpreter
Verify matplotlib
Verify ZeppelinContext
Verify Streaming
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits:
e869f31 [Jeff Zhang] address comments
b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
### What is this PR for?
The python interpreter has a bug when trying to render matplotlib images from the z.show() function. Line 139 of `python/src/main/resources/python/zeppelin_python.py` references un-imported package `base64`. `import base64` was added to the file to prevent this error in the future.
### What type of PR is it?
Bug Fix
### Todos
* [x] - Add missing `base64` module to `python/src/main/resources/python/zeppelin_python.py`
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2821
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
Display a matplotlib image with zeppelin:
```python
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
z.show(plt)
```
### Questions:
* Does the licenses files need update?
no
* Is there breaking changes for older versions?
no
* Does this needs documentation?
no
Author: Shiem Edelbrock <shiem.edelbrock@me.com>
Closes#2511 from Sdedelbrock/patch-1 and squashes the following commits:
2627e70 [Shiem Edelbrock] Fix Missing import
### What is this PR for?
At compile time Zeppelin is downloading several external resources.
I want to be able to provide alternative URLs to compile when internet is not available(i.e. behind corporate proxy).
### What type of PR is it?
[Improvement]
### What is the Jira issue?
[ZEPPELIN-2766]
### How should this be tested?
```
mvn -DskipTests -Dplugin.frontend.downloadRoot=<alernative-url> -Dplugin.frontend.nodeDownloadRoot=<alernative-url> -Dplugin.frontend.yarnDownloadRoot=<alernative-url> -Dplugin.frontend.npmDownloadRoot=<alernative-url> -Dpypi.repo.url=<alernative-url> clean package
```
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? NO
Author: andrea <andrea.peruffo1982@gmail.com>
Closes#2483 from andreaTP/softCodeRepos and squashes the following commits:
6894bcd5 [andrea] added docs for additional configurations
647417c3 [andrea] fix typo
1e5fd85f [andrea] remove deprecated downloadRoot
0aea864a [andrea] Keep online resources configurable
### What is this PR for?
I spoiled the previous PR #2251
Added widgets (string, text, url, password, url, checkbox) to properties of interpreters. Those are widgets for properties customization. Properties must have the ability to customize the display (for example password).
### What type of PR is it?
Feature
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2403
### How should this be tested?
- remove conf/interpreter.json
- Try new form (create, edit) of interpreter settings
### Screenshots (if appropriate)
edit
view
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Tinkoff DWH <tinkoff.dwh@gmail.com>
Author: isys.mreshetov <m.reshetov@i-sys.ru>
Closes#2268 from tinkoff-dwh/ZEPPELIN-2403 and squashes the following commits:
75a10464 [isys.mreshetov] ZEPPELIN-2403 imports fix
7be8ddff [isys.mreshetov] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
585fc364 [isys.mreshetov] ZEPPELIN-2403 documentation fix
4b633993 [isys.mreshetov] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
726c1f31 [isys.mreshetov] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
b17dfb59 [isys.mreshetov] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
098fbd14 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
a5f13272 [Tinkoff DWH] [ZEPPELIN-2403] checkstyle fix
fd25c467 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
e35ff58f [Tinkoff DWH] [ZEPPELIN-2403] fix checkstyle
7c25b6db [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
10ce996a [Tinkoff DWH] [ZEPPELIN-2403] merge widget and type
ca1e2bf7 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
99daca6d [Tinkoff DWH] [ZEPPELIN-2403] fix rest api test
f735c0a9 [Tinkoff DWH] [ZEPPELIN-2403] fix test
c6d24c4c [Tinkoff DWH] [ZEPPELIN-2403] converter for old settings to new (with widgets)
76a98083 [Tinkoff DWH] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2403
b41e7a3f [Tinkoff DWH] ZEPPELIN-2403 checkstyle
637cb0a1 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
e92713c7 [Tinkoff DWH] [ZEPPELIN-2403] generalized types, added new types
07160e00 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
a495137f [Tinkoff DWH] ZEPPELIN-2403 eslint fix
fd8d2781 [Tinkoff DWH] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2403_backup
4f271d9b [Tinkoff DWH] ZEPPELIN-2403 rename to widget added new widgets string, number, url
dd5d6c80 [Tinkoff DWH] ZEPPELIN-2403 did properties immutable, added new type 'checkbox'
14353b12 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
12499ae1 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
45f5f627 [Tinkoff DWH] ZEPPELIN-2403 added interpreter property types
### What is this PR for?
Currently, zeppelin only support 3 kinds of dynamic form controls: TextBox, Select, CheckBox. All the things are in `Input.java`, this is hard to add new controls, this PR is for refactoring Input to make dynamic forms extensible. Main Changes:
* Make `Input` as the base class of dynamic forms also use it as the factory class
* All the concret dynamic forms extend `Input`
* Add method `toJson` and `fromJson` for `GUI` for `GUI`'s serialization/deserialization. I plan to do it for other classes as well, so that we can remove duplicated serde code and also make it easy to test serialization/deserialization
* Change `z.input` to `z.textbox` as I think z.input is a little misleading. But I still keep `z.input` and make `z.input` as deprecated.
* Ideally the new input forms' json should be the same as the old input form json. But there's one bug in the old input form, `type` is missing if the input forms are created in frontend for textbox and select. So I keep the old input forms for compatibility. I will load the old input forms json and convert it into new input forms, and after saving, `note.json` would have the new input forms json.
After this PR, user needs to do 3 things to add new ui controls
* Implement its UI control classes, (refer TextBox/CheckBox/Select), and specify it in `TypeAdapterFactory` of `Input` for serde.
* Add parsing logic in `Input.getInputForm` if you want to support this control in frontend.
* Add display logic in `paragraph-parameterizedQueryForm.html`
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2395
### How should this be tested?
Test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2245 from zjffdu/ZEPPELIN-2395 and squashes the following commits:
16d42a8 [Jeff Zhang] ZEPPELIN-2395. Refactor Input.java to make dynamic forms extensible
### What is this PR for?
If you override the reserved word ZeppelinContext such as `z` in the python language, the whole paragraph output problem occurred.
I have taken care to avoid this issue.
`z` == `_zc` == `zeppelin context`
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-802
### How should this be tested?
The error should not occur in the following situations:
```
%python
z = 1
print("Hello Zeppelin")
```
```
%pyspark
z = 1
print("Hello Zeppelin")
```
### Screenshots (if appropriate)
#### before
#### after
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: CloverHearts <cloverheartsdev@gmail.com>
Author: cloverhearts <cloverheartsdev@gmail.com>
Closes#2207 from cloverhearts/ZEPPELIN-802-pyspark-zeppelin-context and squashes the following commits:
cc986010 [CloverHearts] added completion on namespace
14695cb8 [CloverHearts] Recovering a member name that is not associated with a namespace
31af92ab [CloverHearts] fix test case _zc to __zeppelin__
6697d677 [CloverHearts] apply to namespace and replace name _zc to __zeppelin__
ca795cff [cloverhearts] replace output
1d372df4 [cloverhearts] change name logger
4e8435ac [CloverHearts] added test case on python
b6b804ad [CloverHearts] replace name zeppelin context on python
9fbf70d6 [CloverHearts] fix pyspark test case
987e2118 [CloverHearts] added test code
5da3d6ed [CloverHearts] replace name zeppelin context on pyspark
### What is this PR for?
Currently users who build Zeppelin from source need to include `-Ppyspark` to use `%pyspark` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Ppyspark` and make pyspark related libraries automatically downloaded when we build Zeppelin.
### What type of PR is it?
Improvement
### Todos
* [x] - remove the rest of `-Ppyspark` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after getting feedback
### What is the Jira issue?
[ZEPPELIN-2298](https://issues.apache.org/jira/browse/ZEPPELIN-2298)
### How should this be tested?
1. Apply this patch
2. Build source with below command
```
mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies'
```
Aftr this step, there will be `pyspark` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Ppyspark` build profile.
4. Restart Zeppelin. To make sure, run any python code e.g.
```
%pyspark
print("Hello "+z.input("name"))
```
It should be run successfully without any error
### Screenshots (if appropriate)
tl;dr Without `-Ppyspark` profile
- Before
<img width="856" alt="screen shot 2017-04-02 at 2 50 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584778/0e8ec6b0-17b4-11e7-9f0d-f2599fd7bd63.png">
- After
<img width="893" alt="screen shot 2017-04-02 at 2 28 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584779/10b7ed68-17b4-11e7-90d4-aa95eb9bba2d.png">
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
I want to include `SparkR` by default(= remove `-PsparkR` build profile) like this as a next step. I want to ask how Zeppelin community think about this.
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Closes#2213 from AhyoungRyu/ZEPPELIN-2298/includePysparkByDefault and squashes the following commits:
f7bcf06 [AhyoungRyu] Remove -Ppyspark in virtual_machine.md
458ac02 [AhyoungRyu] Remove the rest of -Ppyspark in blind side of Zeppelin :)
cee1e87 [AhyoungRyu] Change py4j.version -> python.py4j.version
ce43158 [AhyoungRyu] Change py4j.version -> spark.py4j.version
fa4fb36 [AhyoungRyu] Remove the rest of -Ppyspark
30aac81 [AhyoungRyu] Remove -Ppyspark build flag
### What is this PR for?
https://github.com/apache/zeppelin/pull/2106 rewrote python interpreter. But dynamic form feature is not rewritten correctly.
### What type of PR is it?
Hot Fix
### Todos
* [x] - Bring dynamic form back
### What is the Jira issue?
https://github.com/apache/zeppelin/pull/2106
### How should this be tested?
run
```
%python
print("Hello "+z.input("name", "sun"))
```
```
%python
print("Hello "+z.select("day", [("1","mon"),
("2","tue"),
("3","wed"),
("4","thurs"),
("5","fri"),
("6","sat"),
("7","sun")]))
```
```
%python
options = [("apple","Apple"), ("banana","Banana"), ("orange","Orange")]
print("Hello "+ " and ".join(z.checkbox("fruit", options, ["apple"])))
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#2155 from Leemoonsoo/python_get_interpreter_context and squashes the following commits:
c5e584a [Lee moon soo] fix matplotlib display error on python 3.4
3e6603b [Lee moon soo] correctly handle zeppelin.python property.
5be8db4 [Lee moon soo] Expose a method to get InterpreterOutput, so user can call InterpreterOutput.clear()
a405a93 [Lee moon soo] implement dynamic form
### What is this PR for?
I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter.
1. not working streaming output.
- https://issues.apache.org/jira/browse/ZEPPELIN-2225
2. printed "..." when there is indent in the python code.
- https://issues.apache.org/jira/browse/ZEPPELIN-1929
3. very slow output of matplotlib
- https://issues.apache.org/jira/browse/ZEPPELIN-1894
- https://issues.apache.org/jira/browse/ZEPPELIN-1360
4. Unexpected output of matplotlib.
- https://issues.apache.org/jira/browse/ZEPPELIN-2107
so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues.
and I am going to recreate conda, docker for python interpreter ASAP.
### What type of PR is it?
Bug Fix | Hot Fix | Refactoring
### How should this be tested?
1. not working streaming output.
```
import time
for x in range(0, 5):
print x
time.sleep(1)
```
2. printed "..." when there is indent in the python code.
```
def fn():
print("hi")
fn()
```
3. very slow output of matplotlib.
```
import matplotlib
import sys
import matplotlib.pyplot as plt
plt.plot([1,2,3])
```
4. Unexpected output of matplotlib.
```
import matplotlib.pyplot as plt
import matplotlib as mpl
# Make a figure and axes with dimensions as desired.
fig = plt.figure(figsize=(8, 3))
ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15])
ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15])
ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15])
# Set the colormap and norm to correspond to the data for which
# the colorbar will be used.
cmap = mpl.cm.cool
norm = mpl.colors.Normalize(vmin=5, vmax=10)
# ColorbarBase derives from ScalarMappable and puts a colorbar
# in a specified axes, so it has everything needed for a
# standalone colorbar. There are many more kwargs, but the
# following gives a basic continuous colorbar with ticks
# and labels.
cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap,
norm=norm,
orientation='horizontal')
cb1.set_label('Some Units')
# The second example illustrates the use of a ListedColormap, a
# BoundaryNorm, and extended ends to show the "over" and "under"
# value colors.
cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c'])
cmap.set_over('0.25')
cmap.set_under('0.75')
# If a ListedColormap is used, the length of the bounds array must be
# one greater than the length of the color list. The bounds must be
# monotonically increasing.
bounds = [1, 2, 4, 7, 8]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap,
norm=norm,
# to use 'extend', you must
# specify two extra boundaries:
boundaries=[0] + bounds + [13],
extend='both',
ticks=bounds, # optional
spacing='proportional',
orientation='horizontal')
cb2.set_label('Discrete intervals, some other units')
# The third example illustrates the use of custom length colorbar
# extensions, used on a colorbar with discrete intervals.
cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.],
[1., .8, 0.], [1., .4, 0.]])
cmap.set_over((1., 0., 0.))
cmap.set_under((0., 0., 1.))
bounds = [-1., -.5, 0., .5, 1.]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap,
norm=norm,
boundaries=[-10] + bounds + [10],
extend='both',
# Make the length of each extension
# the same as the length of the
# interior colors:
extendfrac='auto',
ticks=bounds,
spacing='uniform',
orientation='horizontal')
cb3.set_label('Custom extension lengths, some other units')
plt.show()
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@zepl.com>
Author: Lee moon soo <moon@apache.org>
Author: HyungSung <hsshim@nflabs.com>
Closes#2106 from astroshim/py4jPythonInterpreter and squashes the following commits:
c9b195b [HyungSung] Merge pull request #16 from Leemoonsoo/py4jdocker
e511ebe [Lee moon soo] add PythonDockerInterpreter to interpreter-setting.json
a76b0d8 [Lee moon soo] fix test on python3
2eb5de7 [Lee moon soo] Fix PythonDockerInterpreterTest.java test
9fcf144 [Lee moon soo] Make python docker interpreter work using py4j
8a016c9 [astroshim] Merge branch 'master' into py4jPythonInterpreter
aad7ee8 [astroshim] fix testcase
ac92cdb [astroshim] fix python interpreter testcase
e8570d2 [astroshim] fix ci for pandassql
be5db4d [astroshim] fix pandas sql testcase
f8e19be [astroshim] fix matplotlib testcase
046db88 [astroshim] add testcase
e49ad24 [astroshim] add pandas
60e9820 [astroshim] bug fix about copying library
574bd21 [astroshim] fix interpreter-setting error
a48df58 [astroshim] Merge branch 'master' into py4jPythonInterpreter
3c9585f [astroshim] update interpreter-setting.json
a50179e [astroshim] add conda interpreter
cbbc15c [astroshim] fix py4j path
5ae5120 [astroshim] fix interpreter-setting
f17bff4 [astroshim] fix testcase failure.
af097ac [astroshim] add testcase
c3f5b78 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into py4jPythonInterpreter
1395875 [astroshim] removed unnecessary code.
276011e [astroshim] add py4j lib
7304919 [astroshim] initialize python interpreter using py4j
### What is this PR for?
Bump up version to 0.8.0-SNAPSHOT
Author: Mina Lee <minalee@apache.org>
Closes#1883 from minahlee/0.8.0-SNAPSHOT and squashes the following commits:
b28ebe8 [Mina Lee] Bump up table/visualization plugin package version
fcaaa46 [Mina Lee] Bump up helium-dev version
180d2c6 [Mina Lee] Bump up version to 0.8.0-SNAPSHOT
### What is this PR for?
Add missing commands to the `python.conda` interpreter
- `conda info`
- `conda list`
- `conda create`
- `conda install`
- `conda uninstall (alias of remove)`
- `conda env *`
#### Implementation Detail
The reason I modified `PythonProcess` is due to NPE
```java
// https://github.com/apache/zeppelin/blob/master/python/src/main/java/org/apache/zeppelin/python/PythonProcess.java#L107-L118
public String sendAndGetResult(String cmd) throws IOException {
writer.println(cmd);
writer.println();
writer.println("\"" + STATEMENT_END + "\"");
StringBuilder output = new StringBuilder();
String line = null;
// NPE when line is null
while (!(line = reader.readLine()).contains(STATEMENT_END)) {
logger.debug("Read line from python shell : " + line);
output.append(line + "\n");
}
return output.toString();
}
```
```
java.lang.NullPointerException
at org.apache.zeppelin.python.PythonProcess.sendAndGetResult(PythonProcess.java:113)
at org.apache.zeppelin.python.PythonInterpreter.sendCommandToPython(PythonInterpreter.java:250)
at org.apache.zeppelin.python.PythonInterpreter.bootStrapInterpreter(PythonInterpreter.java:272)
at org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:100)
at org.apache.zeppelin.python.PythonCondaInterpreter.restartPythonProcess(PythonCondaInterpreter.java:139)
at org.apache.zeppelin.python.PythonCondaInterpreter.interpret(PythonCondaInterpreter.java:88)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:494)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
```
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [x] - info
* [x] - list
* [x] - create
* [x] - install
* [x] - uninstall (= remove)
* [x] - env *
### What is the Jira issue?
[ZEPPELIN-1917](https://issues.apache.org/jira/browse/ZEPPELIN-1917)
### How should this be tested?
1. Install [miniconda](http://conda.pydata.org/miniconda.html)
2. Make sure that your python interpreter can use `conda` (check the Interpreter Binding page)
3. Remove `test` conda env since we will create in the following section
```sh
$ conda env remove --yes --name test
```
4. Run these commands with `%python.conda`
```
%python.conda info
%python.conda env list
%python.conda create --name test
# you should be able to see `test` in the list
%python.conda env list
%python.conda activate pymysql
%python.conda install pymysql
# you should be able to import
%python
import pymysql.cursors
%python.conda uninstall pymysql
%python.conda deactivate pymysql
# you should be able to see `No module named pymysql.cursor` since we deactivated
%python
import pymysql.cursors
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#1868 from 1ambda/ZEPPELIN-1917/improve-conda-interpreter and squashes the following commits:
3ba171a [1ambda] fix: Wrap output style
292ed6d [1ambda] refactor: Throw exception in runCommand
2d4aa7d [1ambda] test: Add some tests
49a4a11 [1ambda] feat: Supports other env commands
6eb7e92 [1ambda] fix: NPE in PythonProcess
9c5dd86 [1ambda] refactor: Activate, Deactivate
f955889 [1ambda] fix: minor
935cb89 [1ambda] refactor: Abstract commands
b1c4c9f [1ambda] feat: Add conda remove (uninstall)
e539c42 [1ambda] feat: Add conda install
4f58fa2 [1ambda] feat: Add conda create
7da132d [1ambda] docs: Add missing conda list description
929ca8a [1ambda] feat: Make conda output beautiful
0c6ebb4 [1ambda] feat: Add list conda command
017c76f [1ambda] refactor: Import InterpreterResult.{Code, Type} to short codes
b8a5154 [1ambda] refactor: Simplify exception flow so private funcs don't need care exceptions
64d4bef [1ambda] style: Rename some funcs
afc456d [1ambda] refactor: Add private to member vars
f36fc74 [1ambda] feat: Add info command
2eb9bf5 [1ambda] style: Remove useless newlines
bd2564e [1ambda] refactor: PythonCondaInterpreter.interpret
f0d69bc [1ambda] fix: Use specific command for env list in conda
### What is this PR for?
Use shared test library versions in maven config so that lib versions do mot be fragmented.
Previously we used multiple versions of
- Junit (4.11, 4.12)
- mockito (1.9.0, 1.10.8, ...)
- powermock (...)
### What type of PR is it?
[Improvement]
### What is the Jira issue?
[ZEPPELIN-1695](https://issues.apache.org/jira/browse/ZEPPELIN-1695)
### How should this be tested?
Use this command to see test libraries share versions or not
```
$ mvn org.apache.maven.plugins:maven-help-plugin:2.2:effective-pom | vim -
```
### Questions:
* Does the licenses files need update? - YES, I updated JUnit version to 4.12 from 4.11
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#1727 from 1ambda/feat/centralise-testing-libraries and squashes the following commits:
b6fd336 [1ambda] chore: Shared mockito, powermock version
941f1ba [1ambda] chore: Update junit to 4.12
### What is this PR for?
Because `conda run` command removed since version `4.0.9`, PythonCondaInterpreter not working after the `conda-4.0.9`.
This PR fixes this issue.
I tested conda-4.2.12 and conda-4.0.9 .
### What type of PR is it?
Bug Fix | Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1724
### How should this be tested?
Please refer to https://github.com/apache/zeppelin/pull/1645
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@nflabs.com>
Closes#1699 from astroshim/ZEPPELIN-1724 and squashes the following commits:
294b6f9 [astroshim] refactoring and fix testcase
8c3fbd3 [astroshim] fix conda version
### What is this PR for?
Currently a paragraph can display only a single type of result.
For example
```
print("""
%text textout
%html htmlout
""")
```
will display only last display system detected, "htmlout".
This pr implements multiple results supports, so not only the last, but also all the display systems in a paragraph outputs can be displayed.
To do that, Note.json format need to be changed. from
```
paragraph : [
{
result : {
code: "",
type: "",
msg: ""
},
config : {
graph : {},
...
},
...
},
...
],
...
```
to
```
paragraph : [
{
results : {
{
code: "",
msg: [
{
type: "",
data: ""
},
{
type: "",
data: ""
},
...
]
},
config : {
results : [
{
graph : {},
},
{
graph : {},
},
...
]
},
...
},
...
],
...
```
### What type of PR is it?
Improvement
### Todos
* [x] - Make InterpreterResult and InterpreterOutput support multiple display system
* [x] - Automatic migration old format to new format
* [x] - Render multiple results in front-end
* [x] - Take care Importing old notebook format
* [x] - Make helium framework support multiple results
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-212
### How should this be tested?
run following code in a single spark paragraph.
```
%spark
// default output is text
(1 to 3).foreach{i=>
println(new java.util.Date())
Thread.sleep(1000)
}
// print something in html
println(s"""%html <h3><font style="color:blue">Some HTML</font></h3>""")
// create table
println(s"""%table key\tvalue
sun\t100
moon\t200
""")
// display text again
println("""%text""")
(1 to 3).foreach{i=>
println(new java.util.Date())
Thread.sleep(1000)
}
// another table
Thread.sleep(1000)
println(s"""%table key\tvalue
apple\t100
banana\t200
""")
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? yes
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes#1658 from Leemoonsoo/ZEPPELIN-212 and squashes the following commits:
0c6a13c [Lee moon soo] Merge remote-tracking branch 'origin/master' into ZEPPELIN-212
519cf23 [Lee moon soo] Update PythonInterpreterPandasSqlTest
e6268ac [Lee moon soo] Update PythonInterpreterMatplotlibTest.java
f5034b8 [Lee moon soo] Merge remote-tracking branch 'origin/master' into ZEPPELIN-212
d5981d5 [Lee moon soo] Merge remote-tracking branch 'origin/master' into ZEPPELIN-212
a1fe729 [Lee moon soo] document note format change
282504e [Lee moon soo] Merge remote-tracking branch 'origin/master' into ZEPPELIN-212
7ab6679 [Lee moon soo] update selenium test
6a897a5 [Lee moon soo] Update rest-api doc
cbbd58a [Lee moon soo] update unittest
e89c9b8 [Lee moon soo] restore tutoral note
3ba37b7 [Lee moon soo] Let old version import note without error
d09e03f [Lee moon soo] enable helium only in the last result
6682908 [Lee moon soo] Remove unnecessary listener method
0f1d28f [Lee moon soo] Merge branch 'master' into ZEPPELIN-212
73b3a81 [Lee moon soo] update selenium test
e69f1a1 [Lee moon soo] update test
f12230f [Lee moon soo] Remove unnecessary newline in test
26e201a [Lee moon soo] Update testcase
b01d70f [Lee moon soo] Make helium app work
0a5b4d1 [Lee moon soo] Update HeliumApplicationFactoryTest
d8ca07f [Lee moon soo] update NotebookTest
aaf9778 [Lee moon soo] fix r test
3e43df4 [Lee moon soo] make zeppelin-web test pass
1d6fd3e [Lee moon soo] fix compile errors
0156ffc [Lee moon soo] take care angular object
804768d [Lee moon soo] Take care output streaming
57d6b2f [Lee moon soo] Render multiple results
95b6037 [Lee moon soo] Multiple results
