### What is this PR for?
This PR will break python code execution if the python process is existed. Besides that, I also improve the error message for ipython interpreter although it doesn't have such issue.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4081
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3338 from zjffdu/ZEPPELIN-4081 and squashes the following commits:
907faacf6 [Jeff Zhang] ZEPPELIN-4081. when the python process is killed,the task state is still running
(cherry picked from commit c9514e26a0)
Signed-off-by: Jeff Zhang <zjffdu@apache.org>
### What is this PR for?
The pr is the last fix of the `ZEPPELIN-4089` It will aims to notify user when ipython kernel crash. Indeed, right now, execution will be stuck forever when ipython kernel crash.
### What type of PR is it?
Bug Fix
### Todos
* [x] - logic to check kernel status
* [ ] - decide what to do when we spot the failure
* [x] - test
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4089
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3339 from AyWa/fix/kernel-crash and squashes the following commits:
6f4910c67 [marc hurabielle] fix lint
73424d17d [marc hurabielle] Revert "example for kernel die and request stuck"
f37d5f95d [marc hurabielle] add test for kernel die
c9ec0335c [marc hurabielle] example for kernel die and request stuck
5fe84dfef [marc hurabielle] handle kernel crash
(cherry picked from commit ef5e173d3a)
Signed-off-by: Jeff Zhang <zjffdu@apache.org>
### What is this PR for?
This PR will break python code execution if the python process is existed. Besides that, I also improve the error message for ipython interpreter although it doesn't have such issue.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4081
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3338 from zjffdu/ZEPPELIN-4081 and squashes the following commits:
907faacf6 [Jeff Zhang] ZEPPELIN-4081. when the python process is killed,the task state is still running
### What is this PR for?
The pr is the last fix of the `ZEPPELIN-4089` It will aims to notify user when ipython kernel crash. Indeed, right now, execution will be stuck forever when ipython kernel crash.
### What type of PR is it?
Bug Fix
### Todos
* [x] - logic to check kernel status
* [ ] - decide what to do when we spot the failure
* [x] - test
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4089
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3339 from AyWa/fix/kernel-crash and squashes the following commits:
6f4910c67 [marc hurabielle] fix lint
73424d17d [marc hurabielle] Revert "example for kernel die and request stuck"
f37d5f95d [marc hurabielle] add test for kernel die
c9ec0335c [marc hurabielle] example for kernel die and request stuck
5fe84dfef [marc hurabielle] handle kernel crash
### What is this PR for?
This PR will exist wait ipython process loop if the python process is failed. This can save time to exit the waiting loop earlier, otherwise it would wait until timeout (30 seconds by default)
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4062
### How should this be tested?
* Manually tested.
Before this PR
```
INFO [2019-03-13 10:06:30,070] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,175] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,281] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,386] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,489] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,595] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,700] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,805] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
WARN [2019-03-13 10:06:30,810] ({Exec Default Executor} IPythonInterpreter.java[onProcessFailed]:419) - Exception happens in Python Process
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:06:30,910] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:31,015] ({FIFOScheduler-Worker-1}
```
After this PR
```
INFO [2019-03-13 10:28:12,805] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:12,907] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:13,011] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:13,116] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:13,220] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:13,325] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
WARN [2019-03-13 10:28:13,347] ({Exec Default Executor} IPythonInterpreter.java[onProcessFailed]:423) - Exception happens in Python Process
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:28:13,430] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
WARN [2019-03-13 10:28:13,431] ({FIFOScheduler-Worker-1} PythonInterpreter.java[open]:98) - Fail to open IPythonInterpreter
java.lang.RuntimeException: Fail to open IPythonInterpreter
at org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:152)
at org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:93)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:598)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:507)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:121)
at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Fail to launch IPython Kernel as the python process is failed
at org.apache.zeppelin.python.IPythonInterpreter.launchIPythonKernel(IPythonInterpreter.java:323)
at org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:149)
... 10 more
INFO [2019-03-13 10:28:13,431] ({FIFOScheduler-Worker-1} PythonInterpreter.java[open]:104) - IPython is not available, use the native PythonInterpreter
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3332 from zjffdu/ZEPPELIN-4062 and squashes the following commits:
130c2a82a [Jeff Zhang] ZEPPELIN-4062. Don't wait ipython kernel if python process failed
### What is this PR for?
Trivial PR to fix the IPythonClient not shutdown issue. Otherwise user will get the following error
```
Channel ManagedChannelImpl{logId=6, target=127.0.0.1:43464} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:410)
...
```
### What type of PR is it?
[Bug Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4042
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3324 from zjffdu/ZEPPELIN-4042 and squashes the following commits:
72e1ceb58 [Jeff Zhang] [ZEPPELIN-4042]. IPythonClient is not shutdown properly
### What is this PR for?
This PR add local property `limit` for SparkInterpreter and JdbcInterpreter, so that users can not control the max number of returned rows in paragraph level instead of interpreter level.
### What type of PR is it?
[ Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-3975
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3301 from zjffdu/ZEPPELIN-3975 and squashes the following commits:
6350433cc [Jeff Zhang] ZEPPELIN-3975. Add limit local property for SparkInterpreter and JdbcInterpreter
### What is this PR for?
delete temporary file and close inputstream in IPythonInterpreter.checkIPythonPrerequisite()
### What type of PR is it?
[Bug Fix]
### Todos
*no
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-3910, [ZEPPELIN-3910]
### How should this be tested?
* manual.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
no
* Is there breaking changes for older versions?
no
* Does this needs documentation?
no
Author: yx91490 <yx91490@126.com>
Closes#3287 from yx91490/ipy_tmp_file and squashes the following commits:
c811e3736 [yx91490] delete temp file and close stream
### What is this PR for?
This ticket create AbstractInterpreter which do some common things:
* Support ZeppelinContext
* Support z variable replacement
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-3976
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3293 from zjffdu/ZEPPELIN-3976 and squashes the following commits:
7b42a42f3 [Jeff Zhang] ZEPPELIN-3976. Create AbstractInterprter for common usage
### What is this PR for?
This is refactoring PR which move common dependencies into zeppelin-interpreter-parent
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3790
### How should this be tested?
* Ci pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3194 from zjffdu/ZEPPELIN-3790 and squashes the following commits:
0b34645a0 [Jeff Zhang] move junit to zeppelin-interpreter-parent
e4baa595e [Jeff Zhang] ZEPPELIN-3790. Move common interpreter dependencies into zeppelin-interpreter-parent
### What is this PR for?
By using the Raft protocol, multiple Zeppelin-Server groups are built into a Zeppelin cluster, the cluster State Machine is maintained through the Raft protocol, and the services in the cluster are agreed upon. The Zeppelin-Server and Zeppelin-Interperter services and processes are stored in the Cluster MetaData. Metadata information;
### What type of PR is it?
[Feature]
### Todos
* [x] add raft algorithm atomix jar
* [x] add cluster state machine
* [x] add state machine query command
* [x] add state machine delete command
* [x] add state machine put command
* [x] Isolate the netty JAR package introduced by atomix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3610
### How should this be tested?
[CI pass](https://travis-ci.org/liuxunorg/zeppelin/builds/418742522)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: liuxunorg <33611720@qq.com>
Author: Jeff Zhang <zjffdu@gmail.com>
Closes#3183 from liuxunorg/ZEPPELIN-3610 and squashes the following commits:
c899b151f [liuxunorg] uncomment maven-dependency-plugin.
af728bd2c [liuxunorg] add more doc to explain this class's responsibility.
d268ee72e [Jeff Zhang] fix test (#2)
97f17acac [liuxunorg] Cluster Raft module design
### What is this PR for?
Trivial PR for module name refactoring.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3774
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3175 from zjffdu/ZEPPELIN-3774 and squashes the following commits:
ea72df9e5 [Jeff Zhang] ZEPPELIN-3774. Rename module interpreter-parent to zeppelin-interpreter-parent
### What is this PR for?
This PR introduce new module zeppelin-interpreter-api which will shade all dependencies of zeppelin-interpreter, this is just to avoid the conflict with the custom interpreter implementation which may use jars that conflicts with dependency of zeppelin-interpreter.
### What type of PR is it?
[ Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3689
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3170 from zjffdu/ZEPPELIN-3689 and squashes the following commits:
f3f210aee [Jeff Zhang] ZEPPELIN-3689. Shade all dependencies of zeppelin-interpreter
### What is this PR for?
Avoiding outdated checkstyle error.
To check format
`mvn validate`
To fix non-complying files
`mvn fmt:format`
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [x] - Adopt `fmt-maven-plugin`
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3740
### How should this be tested?
CI passs
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes for contributors
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#3158 from jongyoul/ZEPPELIN-3740 and squashes the following commits:
f06c50e90 [Jongyoul Lee] Adopt `fmt-maven-plugin` for google-java-format Adopt all files
### What is this PR for?
This is to protobuf prerequisite check when starting ipython kernel. This is required since from grpcio 1.12.0 it's independent from protobuf
### What type of PR is it?
[ Improvement ]
### Todos
* [x] - add protobuf prerequisite check
* [x] - add to ipython docs
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3639
### How should this be tested?
1. remove grpcio and protobuf
2. install latest grpcio and try to run ipython interpreter - should fail
3. then install protobuf and try again - should work
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Khalid Huseynov <khalidhnv@gmail.com>
Closes#3084 from khalidhuseynov/fix/ZEPPELIN-3639 and squashes the following commits:
066e061d2 [Khalid Huseynov] fix ipython documentation
58c06898b [Khalid Huseynov] add protobuf prerequsite check
### What is this PR for?
Increase the maven-shade-plugin version to 3.1.1
### What type of PR is it?
Bug Fix
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-3631
### How should this be tested?
Check if build succeeds
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
No
Author: Armin Wasicek <arminw@sumologic.com>
Closes#3077 from arminwasicek/bump-shade-plugin-version and squashes the following commits:
0697f0e1c [Armin Wasicek] Merge branch 'bump-shade-plugin-version' of github.com:arminwasicek/zeppelin into bump-shade-plugin-version
6c4cbe722 [Armin Wasicek] removed individual versions from scio and spark-dependencies
f0691f6eb [Armin Wasicek] increase the maven-shade-plugin version to 3.1.1
9e026a0ae [Armin Wasicek] removed individual versions from scio and spark-dependencies
2f282f080 [Armin Wasicek] increase the maven-shade-plugin version to 3.1.1
### What is this PR for?
This PR is trying to refactor method getInterpreterInTheSameSession in Interpreter, so that each interpreter don't need to reimplement its own and can reduce many code duplication.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3621
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3069 from zjffdu/ZEPPELIN-3621 and squashes the following commits:
06a6cc624 [Jeff Zhang] ZEPPELIN-3621. refactor getInterpreterInTheSameSession of Interpreter to reduce code duplication
### What is this PR for?
fix pandasql query over dataframe for `master`. the reason is that it can't load `bootstrap_sql.py` from resources because path of path
### What type of PR is it?
[Bug Fix]
### Todos
* [x] - fix path to use relative
* [x] - enable tests
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3625
### How should this be tested?
follow the steps in issue
### Screenshots (if appropriate)
before:
<img width="1268" alt="screen shot 2018-07-15 at 7 08 24 pm" src="https://user-images.githubusercontent.com/1642088/42732816-822682f0-8862-11e8-9d55-dd7d00b09b48.png">
after:
<img width="1239" alt="screen shot 2018-07-15 at 7 03 59 pm" src="https://user-images.githubusercontent.com/1642088/42732771-e712f848-8861-11e8-9ab9-eeb479082d1d.png">
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Khalid Huseynov <khalidhnv@gmail.com>
Closes#3071 from khalidhuseynov/fix-master/ZEPPELIN-3625 and squashes the following commits:
33aad7589 [Khalid Huseynov] fix matplotlib test
f94139b33 [Khalid Huseynov] enable all python tests
fd5685552 [Khalid Huseynov] remove unnecessary test
bde2316c7 [Khalid Huseynov] add test
8f9541411 [Khalid Huseynov] fix test setup not to interpret empty line
3a561d247 [Khalid Huseynov] fix from absolute to relative path in resources
### What is this PR for?
minor fix to show correct package name `ipykernel` in logs when prerequisite for ipython interpreter isn't met.
### What type of PR is it?
[ Improvement ]
### Todos
* [x] - fix naming
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
if `ipykernel` isn't installed then it would show correct message in logs when running ipython interpreter
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Khalid Huseynov <khalidhnv@gmail.com>
Closes#3046 from khalidhuseynov/fix/ipykernel-msg and squashes the following commits:
7a3b124f9 [Khalid Huseynov] fix minor
b6a3d5c5f [Khalid Huseynov] fix package name shown to user when not installed
### What is this PR for?
This is to add password for dynamic forms. It is almost the same as input box but but with invisible input. And there's no default value for password form, as it doesn't make sense to provide default value for password.
### What type of PR is it?
[ Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2528
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3041 from zjffdu/ZEPPELIN-2528 and squashes the following commits:
084abd39c [Jeff Zhang] ZEPPELIN-2528. Add a password text input to the ZeppelinContext
### What is this PR for?
After ZEPPELIN-3375, PySparkInterpreter extends PythonInterpreter. PySparkInterpreter may fail to bootstrap PySpark as we miss to add hook to the pyspark bootstrap code.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3538
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3016 from zjffdu/ZEPPELIN-3538 and squashes the following commits:
3a1d8a737 [Jeff Zhang] ZEPPELIN-3538. Fail to bootstrap PySpark in yarn cluster mode
### What is this PR for?
Fix the code style issue of python module
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3163
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3023 from zjffdu/ZEPPELIN-3163 and squashes the following commits:
e89c41108 [Jeff Zhang] [ZEPPELIN-3163]. Fix checkstyle of python
### What is this PR for?
This PR introduce the bi directional RPC framework between ZeppelinServer and InterpreterProcess on top of thrift, see the design doc here. https://docs.google.com/document/d/1B_4b7UuvE7RwbfpTjxV2RSxdDeF-p9vXVYC8ace76-Q/edit?usp=sharing
This PR is pretty big, most of the code changes is due to regeneration of thrift code.
Another improvement is introduce the builder pattern of InterpreterContext as the arguments of InterpreterContext constructor is too many.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2035
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2986 from zjffdu/ZEPPELIN-2035 and squashes the following commits:
c02d5587a [Jeff Zhang] ZEPPELIN-2035. BI directional RPC framework between ZeppelinServer and InterpreterProcess on top of thrift
[ZEPPELIN-3423] Fix deprecated dynamic forms document URL on Python README.md
### What is this PR for?
Python dynamic forms document URL is based on "0.6.0-SNAPSHOT" and already deprecated.
Replace the deprecated URL with the latest URL.
### What type of PR is it?
Documentation
### Todos
N/A
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3423
### How should this be tested?
N/A
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Changhyup Jwa <jwagun@gmail.com>
Closes#2939 from jwagun/ZEPPELIN-3423 and squashes the following commits:
360261c2f [Changhyup Jwa] [ZEPPELIN-3423] Fix deprecated dynamic forms document URL on Python README.md
Change-Id: I540a222678a82d6f88d91bf8790acf4fca19d992
### What is this PR for?
This PR is trying to remove the code duplication between PySparkInterpreter and PythonInterpreter. So here's the main things this PR did:
* PySparkInterpreter extends PythonInterpreter
* PySparkInterpreterTest extends PythonInterpreterTest so that we can verify PySparkInterpreter can do whatever PythonInterpreter can do
* Move interpreter/lib/python/backend_zinline.py and interpreter/lib/python/mpl_config.py into python module, so that python module can ship these resources together.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3375
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2919 from zjffdu/ZEPPELIN-3375 and squashes the following commits:
738c6c5 [Jeff Zhang] ZEPPELIN-3375. Make PySparkInterpreter extends PythonInterpreter
This is what I see in zeppelin and jupyter. The difference is due to zeppelin also add code completion of ace editor. I think code completion of ace editor should only be enabled when the backend interpreter doesn't support completion.
[Bug Fix]
* [ZEPPELIN-3001](https://issues.apache.org/jira/browse/ZEPPELIN-3001)
* Does the licenses files need update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? N/A
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2885 from prabhjyotsingh/ZEPPELIN-3001 and squashes the following commits:
fcd551fe1 [Prabhjyot Singh] meta can be either empty string or undefined
e6ad8d1c7 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-3001
508cd239c [Prabhjyot Singh] fix websocketMsgSrv.completion paragraph id issue
9380c32fa [Prabhjyot Singh] upgrade ace-builds to 1.3.2
701f93f85 [Prabhjyot Singh] make default suggestion if there is no suggestion from back-end.
97f594dc8 [Prabhjyot Singh] completionSupport false for SparkRInterpreter
d2f70ae7d [Prabhjyot Singh] get completionSupport from interpreter (backend)
adaabcbf9 [Prabhjyot Singh] ZEPPELIN-3001: Disable ace editor code completion when back-end interpreter support completion
Change-Id: I03e39422bafebf9c96b01b88f542a16dd6959f23
### What is this PR for?
This PR address the remaining issue of ZEPPELIN-3362, and also add more test for ZeppelinContext.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3364
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2894 from zjffdu/ZEPPELIN-3364 and squashes the following commits:
891f1e1 [Jeff Zhang] ZEPPELIN-3364. Followup of ZEPPELIN-3362, improve ZeppelinContext & add more test
### What is this PR for?
Unify the ZeppelinContext of PythonInterpreter & IPythonInterpreter into one file to avoid code duplication.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3362
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2890 from zjffdu/ZEPPELIN-3362 and squashes the following commits:
b5dcbc9 [Jeff Zhang] ZEPPELIN-3362. Unify ZeppelinContext of PythonInterpreter & IPythonInterpreter
### What is this PR for?
Trivial PR for removing the unnecessary message
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3290
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2882 from zjffdu/ZEPPELIN-3290 and squashes the following commits:
7bf8d6e [Jeff Zhang] ZEPPELIN-3290. Unnecessary message 'IPython is available, use IPython for PySparkInterpreter'
### What is this PR for?
Currently the interpreter hook mechanism is broken. This PR fix this issue and also add unit test for interpreter hook, (including global hook and note level hook)
### What type of PR is it?
[Bug Fix | Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3343
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2875 from zjffdu/ZEPPELIN-3343 and squashes the following commits:
3c55f07 [Jeff Zhang] ZEPPELIN-3343. Interpreter Hook is broken
### What is this PR for?
```
%python
import time
print("Hello")
time.sleep(0.5) # in case of Ipython kernel, print may not immediately flushed and cleared.
z.getInterpreterContext().out().clear()
print("world")
```
Expected to print `world` only.
This worked in zeppelin-0.7.x and it'll be nice keep this feature in the future version.
### What type of PR is it?
Bug fix | Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3284
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Lee moon soo <moon@apache.org>
Closes#2829 from Leemoonsoo/ZEPPELIN-3284 and squashes the following commits:
e0dcda61f [Lee moon soo] make z.getInterpreterContext().out().clear() work in IPythonInterprete
224a2df8e [Lee moon soo] make z.getInterpreterContext().out().clear() work in PythonInterpreter
### What is this PR for?
Remove Redundant method `InterpreterOutput.getInterpreterResultMessages()`
### What type of PR is it?
Refactoring
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3283
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#2828 from Leemoonsoo/ZEPPELIN-3283 and squashes the following commits:
12771a145 [Lee moon soo] remove unused import
82bda9c02 [Lee moon soo] Remove redundant method getInterpreterResultMessages()
### What is this PR for?
trivial change for property name renaming. Followup of #2802
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2813 from zjffdu/HOTFIX_GRPC and squashes the following commits:
d899a29 [Jeff Zhang] [HOTFIX] rename zeppelin.ipython.grpc.framesize to zeppelin.ipython.grpc.message_size
### What is this PR for?
Add one new property `zeppelin.ipython.grpc.framesize` which is an advanced configuration. By default it is 32M which should be sufficient for most of scenarios.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3236
### How should this be tested?
* Unit test is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2802 from zjffdu/ZEPPELIN-3236 and squashes the following commits:
ffce774 [Jeff Zhang] ZEPPELIN-3236. Make grpc framesize configurable
### What is this PR for?
Fix the unicode issues in python2.
### What type of PR is it?
[Bug Fix | Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3239
### How should this be tested?
* Unit test is added
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2810 from zjffdu/ZEPPELIN-3239 and squashes the following commits:
70c8b4c [Jeff Zhang] ZEPPELIN-3239. unicode characters in an iPython paragraph makes Spark interpreter irrsponsive
### What is this PR for?
Enhance the ZeppelinContext in IPySparkInterpreter
### What type of PR is it?
[Bug Fix | Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3234
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2807 from zjffdu/ZEPPELIN-3234 and squashes the following commits:
39637ee [Jeff Zhang] ZEPPELIN-3234. z.show() compatibility with previous release
### What is this PR for?
This is for the refactoring of SparkInterpreter. See design doc. https://docs.google.com/document/d/1AfGg3aGXonDyri1jrP4MMFT4Y4j3wpN1t8kL-GAKSUc/edit?usp=sharing
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3111
### How should this be tested?
* Unit test is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2709 from zjffdu/ZEPPELIN-3111 and squashes the following commits:
aae4b09 [Jeff Zhang] ZEPPELIN-3111. Refactor SparkInterpreter
### What is this PR for?
trivial pom file change
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3192
### How should this be tested?
* Travis pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2748 from zjffdu/ZEPPELIN-3192 and squashes the following commits:
635acde [Jeff Zhang] ZEPPELIN-3192. Bump up Zeppelin version to 0.9.0-SNAPSHOT
### What is this PR for?
This is for better code completion support for IPythonInterpreter
### What type of PR is it?
[Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3136
### How should this be tested?
* Unit test is updated and Manually verified.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2725 from zjffdu/ZEPPELIN-3136 and squashes the following commits:
1179fc8 [Jeff Zhang] ZPEPELIN-3136. IPython Code Completion Improvement
### What is this PR for?
Added dynamic forms for Note. All paragraphs has access to note forms.
### What type of PR is it?
[Feature]
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1363
### How should this be tested?
Create global interpreter JDBC and Spark. Create note with 2 paragraphs
```
%pyspark
print("Textbox paragraph " + z.textbox('input', 'default'))
print("Textbox note " + z.noteTextbox('note_input', 'default_note'))
print("Select paragraph " + z.select("sel", [("1","opt1"), ("2","opt2"), ("3","opt3")]))
print("Select note " + z.noteSelect("sel_note", [("1","noteOpt1"), ("2","noteOpt2"), ("3","noteOpt3")]))
options = [("key1","Name1"), ("key2","Name2")]
print("Checkbox paragraph "+ " and ".join(z.checkbox("chk", options, ["key1"])))
print("Checkbox note "+ " and ".join(z.noteCheckbox("chk_note", options, ["key1","key2"])))
```
```
%jdbc
select '$${checkbox:chk_note=key1|key2,key1|key2} $${note_input} ${note_input=sameName}'
```
### Screenshots (if appropriate)
1) native forms
2) remove
3) simple forms
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: tinkoff-dwh <tinkoff.dwh@gmail.com>
Closes#2641 from tinkoff-dwh/ZEPPELIN-1363 and squashes the following commits:
3ee4826 [tinkoff-dwh] [ZEPPELIN-1363] autosave textbox
f30033a [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1363
29eaca2 [tinkoff-dwh] [ZEPPELIN-1363] fix tests
bf8194e [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1363
9b2f3e9 [tinkoff-dwh] [ZEPPELIN-1363] button to remove form
c566462 [tinkoff-dwh] [ZEPPELIN-1363] note dynamic forms (simple; native: spark, python)
### What is this PR for?
This PR will add tab as auto complete invoker if paragraph is non-md and user has not pressed the tab as a first character in the line
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-277
* https://issues.apache.org/jira/browse/ZEPPELIN-2736
### How should this be tested?
- Build: mvn clean package -Denforcer.skip -DskipTests -Drat.skip
- Open a paragraph
- Press tab with following options: first character, after space
### Questions:
* Does the licenses files need an update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Malay Majithia <malay.majithia@gmail.com>
Author: Lee moon soo <moon@apache.org>
Closes#2542 from malayhm/ZEPPELIN-277 and squashes the following commits:
436f22d [Malay Majithia] Added Tab auto completion flag for python sql and r
b37e084 [Malay Majithia] Fixed lint error
18fc814 [Malay Majithia] Merge branch 'master' into ZEPPELIN-277
b09730e [Malay Majithia] Merge branch 'master' into ZEPPELIN-277
63d69e1 [Malay Majithia] Merge branch 'ZEPPELIN-277' of github.com:malayhm/zeppelin into ZEPPELIN-277
a75f0fe [Malay Majithia] Improved the first character check logic
2ec879d [Malay Majithia] Merge pull request #1 from Leemoonsoo/ZEPPELIN-277-completion-key
77afdba [Lee moon soo] fix style
77b47b6 [Malay Majithia] If all the previous line characters are tab, don't show autocomplete on tab
46f612a [Malay Majithia] ZEPPELIN-277 Add Tab as Autocomplete for Notebook non-md paragraphs
865c0a6 [Lee moon soo] Set python and spark interpreter completionKey
05d5860 [Lee moon soo] Update doc
973068b [Lee moon soo] apply tab completion based on editor.completionKey
5f4d81c [Malay Majithia] If all the previous line characters are tab, don't show autocomplete on tab
655ba88 [Malay Majithia] ZEPPELIN-277 Add Tab as Autocomplete for Notebook non-md paragraphs
### What is this PR for?
Trivial fix for matplotlib fail in `IPythonInterpreter`. Besides that, I also make some minor changes on zeppelin code to improve logging.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2990
### How should this be tested?
UT is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2622 from zjffdu/ZEPPELIN-2990 and squashes the following commits:
55e6f88 [Jeff Zhang] fix unit test
1e57afe [Jeff Zhang] ZEPPELIN-2990. Matplotlib sometimes fails in IPythonInterpreter
### What is this PR for?
Follow up of #2577. Main changes on Interpreter
* Add throw `InterpreterException` which is checked exception for the abstract methods of `Interpreter`, this would enforce the interpreter implementation to throw `InterpreterException`.
* field name refactoring.
* `property` -> `properties`
* `getProperty()` --> `getProperties()`
* Introduce launcher layer for interpreter launching. Currently we only use shell script to launch interpreter, but it could be any other service or component to launch interpreter, such as livy server , other 3rd party tools or even we may create a separate module for interpreter launcher
* abstract cass `InterpreterLauncher`
* For now, only 2 implementation: `ShellScriptLauncher` & `SparkInterpreterLauncher`. We could add method in class `Interpreter` to allow interpreter to specify its own launcher class, but it could be future work.
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2685
### How should this be tested?
Unit test is covered. `ShellScriptLauncherTest` & `SparkInterpreterLauncherTest`
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2592 from zjffdu/ZEPPELIN-2685 and squashes the following commits:
17dc2f1 [Jeff Zhang] address comments
e545cc3 [Jeff Zhang] ZEPPELIN-2685. Improvement on Interpreter class
