### What is this PR for?
The root cause of this issue is that we didn't redirect java output to interpreter output. This PR fix it via redirect java output before interpreting python code in both PySparkInterpreter & IPySparkInterpreter. Unit test is also added to verify this feature.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4692
### How should this be tested?
* Unit test is added, also manually tested it
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3696 from zjffdu/ZEPPELIN-4692 and squashes the following commits:
e1a5ead7e [Jeff Zhang] [ZEPPELIN-4692]. zeppelin pyspark doesn't print java output
### What is this PR for?
This PR would replace all the special characters(\t, \n, \r\n) in table content with white space. so that it won't break the table format display.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4611
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3643 from zjffdu/ZEPPELIN-4611 and squashes the following commits:
d12eea943 [Jeff Zhang] [ZEPPELIN-4611]. Fetching rows with newline character (\n) breaks entire table
### What is this PR for?
Use the SqlSplitter in `zeppelin-interpreter` to split sql and execute in SparkSqlInterpreter. Nothing changes for the previous single sql statement paragraph. But just multiple result will be displayed for multiple sql statements.
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4522
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3579 from zjffdu/ZEPPELIN-4522 and squashes the following commits:
eda573649 [Jeff Zhang] fix failed test
68d5a30c8 [Jeff Zhang] Add test for no sql but just 2 comments
4ff15e4fb [Jeff Zhang] address comment
bc3c1feff [Jeff Zhang] [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
### What is this PR for?
This PR move the ipython code into module zeppelin-jupyter-adapter. And ipython will just depends on this module. zeppelin-jupyter-adapter could be used for connecting any jupyter kernel.
### What type of PR is it?
[ Feature | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4480
### How should this be tested?
* Ci pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3551 from zjffdu/ZEPPELIN-4480 and squashes the following commits:
78453473e [Jeff Zhang] [ZEPPELIN-4480]. Move the ipython code into a general jupyter kernel bridge
### What is this PR for?
HvPlot is broken due to bug in `ipython_server.py`. IPython may produce multiple types of output for one single python object. And the orders of these output matters, we should output html type over plain text type.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4396
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3515 from zjffdu/ZEPPELIN-4396 and squashes the following commits:
f71dbf231 [Jeff Zhang] [ZEPPELIN-4396]. HvPlot is broken
### What is this PR for?
`ExecutorService` is not properly shutdown due to we didn't use the correct api. We should use `shutdownNow` instead of `shutdown`. See https://stackoverflow.com/questions/11520189/difference-between-shutdown-and-shutdownnow-of-executor-service.
The effect of this issue will cause thread resource leakage.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4390
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3491 from zjffdu/ZEPPELIN-4390 and squashes the following commits:
439221492 [Jeff Zhang] [ZEPPELIN-4390]. ExecutorService is not properly shutdown
### What is this PR for?
There's redundant line separator when there's multiple text output. The root cause is that each `%text ` will append new line separator in frontend. This PR fix this issue by removing `%text ` is the previous output type is also `%text `.
### What type of PR is it?
[Bug Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4369
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
Before
After
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3478 from zjffdu/ZEPPELIN-4369 and squashes the following commits:
8c3d380db [Jeff Zhang] [ZEPPELIN-4369] Redundant line separator for multiple text output
### What is this PR for?
The pr is to fix a bug that will make the **ipython** `execute_interactive` hang forever if a auto `complete` call is make at the same time. (see unit test for example that is failing on master).
For now the fix is to synchronize those method : `execute` / `complete`. It will not bring regression because anyway, the kernel does not support concurrent execute and auto complete (see https://github.com/jupyter/notebook/issues/3763)
### What type of PR is it?
Bug Fix
### Todos
* [x] - unit test failing in master / succeed on this branch
* [x] - fix with lock
### What is the Jira issue?
It is one part of the jira issue. Other fix will come soon
https://issues.apache.org/jira/browse/ZEPPELIN-4091
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3336 from AyWa/fix/concurrent-auto-complete and squashes the following commits:
86dab7345 [marc hurabielle] fix rebase
5bed19496 [marc hurabielle] fix lint
6e48c1380 [marc hurabielle] try single threaded
f14d8b242 [marc hurabielle] Revert "just test ci behavior"
be6663f89 [marc hurabielle] just test ci behavior
bc2b4f6e6 [marc hurabielle] bring back test
d43f03da9 [marc hurabielle] use initIntpProperties instead of empty one
c37414cc2 [marc hurabielle] increase timeout
f7cae9538 [marc hurabielle] move synchronize near the thread check
616f0122f [marc hurabielle] add test to ensure that autocomplete and interpret can be call concurrently
409b75f0f [marc hurabielle] add lock to ensure ipython execute will not be stuck forever when complete is call
### What is this PR for?
For now, python test is missing in travis, this PR enable it. But we have to disable some test, because some test will fail, I will fix them in a followup ticket.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4234
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3408 from zjffdu/ZEPPELIN-4234 and squashes the following commits:
42d6877aa [Jeff Zhang] ZEPPELIN-4234. Python test is missing in travis
### What is this PR for?
There're several places in zeppelin to launch processes, such as interpreter process launch, python process launch. Sometimes it is very hard to diagnose if the process is failed to launch or some error happens during its lifecycle. This PR introduce ProcessLauncher to encapsulate process launch. ProcessLauncher will do most of the process launching stuff but also allow for customization. E.g. how to check the process is started properly. And when error happens, ProcessLauncher will capture the necessary error output for further diagnose.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4066
### How should this be tested?
* Ci pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3346 from zjffdu/ZEPPELIN-4066 and squashes the following commits:
b847d6d86 [Jeff Zhang] Add state COMPLETED & also add transition method for more detail logging
97049bd55 [Jeff Zhang] [ZEPPELIN-4066]. Introduce ProcessLauncher to encapsulate process launch
### What is this PR for?
This PR will break python code execution if the python process is existed. Besides that, I also improve the error message for ipython interpreter although it doesn't have such issue.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4081
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3338 from zjffdu/ZEPPELIN-4081 and squashes the following commits:
907faacf6 [Jeff Zhang] ZEPPELIN-4081. when the python process is killed,the task state is still running
(cherry picked from commit c9514e26a0)
Signed-off-by: Jeff Zhang <zjffdu@apache.org>
### What is this PR for?
The pr is the last fix of the `ZEPPELIN-4089` It will aims to notify user when ipython kernel crash. Indeed, right now, execution will be stuck forever when ipython kernel crash.
### What type of PR is it?
Bug Fix
### Todos
* [x] - logic to check kernel status
* [ ] - decide what to do when we spot the failure
* [x] - test
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4089
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3339 from AyWa/fix/kernel-crash and squashes the following commits:
6f4910c67 [marc hurabielle] fix lint
73424d17d [marc hurabielle] Revert "example for kernel die and request stuck"
f37d5f95d [marc hurabielle] add test for kernel die
c9ec0335c [marc hurabielle] example for kernel die and request stuck
5fe84dfef [marc hurabielle] handle kernel crash
(cherry picked from commit ef5e173d3a)
Signed-off-by: Jeff Zhang <zjffdu@apache.org>
### What is this PR for?
This PR will break python code execution if the python process is existed. Besides that, I also improve the error message for ipython interpreter although it doesn't have such issue.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4081
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3338 from zjffdu/ZEPPELIN-4081 and squashes the following commits:
907faacf6 [Jeff Zhang] ZEPPELIN-4081. when the python process is killed,the task state is still running
### What is this PR for?
The pr is the last fix of the `ZEPPELIN-4089` It will aims to notify user when ipython kernel crash. Indeed, right now, execution will be stuck forever when ipython kernel crash.
### What type of PR is it?
Bug Fix
### Todos
* [x] - logic to check kernel status
* [ ] - decide what to do when we spot the failure
* [x] - test
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4089
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3339 from AyWa/fix/kernel-crash and squashes the following commits:
6f4910c67 [marc hurabielle] fix lint
73424d17d [marc hurabielle] Revert "example for kernel die and request stuck"
f37d5f95d [marc hurabielle] add test for kernel die
c9ec0335c [marc hurabielle] example for kernel die and request stuck
5fe84dfef [marc hurabielle] handle kernel crash
### What is this PR for?
By using the Raft protocol, multiple Zeppelin-Server groups are built into a Zeppelin cluster, the cluster State Machine is maintained through the Raft protocol, and the services in the cluster are agreed upon. The Zeppelin-Server and Zeppelin-Interperter services and processes are stored in the Cluster MetaData. Metadata information;
### What type of PR is it?
[Feature]
### Todos
* [x] add raft algorithm atomix jar
* [x] add cluster state machine
* [x] add state machine query command
* [x] add state machine delete command
* [x] add state machine put command
* [x] Isolate the netty JAR package introduced by atomix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3610
### How should this be tested?
[CI pass](https://travis-ci.org/liuxunorg/zeppelin/builds/418742522)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: liuxunorg <33611720@qq.com>
Author: Jeff Zhang <zjffdu@gmail.com>
Closes#3183 from liuxunorg/ZEPPELIN-3610 and squashes the following commits:
c899b151f [liuxunorg] uncomment maven-dependency-plugin.
af728bd2c [liuxunorg] add more doc to explain this class's responsibility.
d268ee72e [Jeff Zhang] fix test (#2)
97f17acac [liuxunorg] Cluster Raft module design
### What is this PR for?
Avoiding outdated checkstyle error.
To check format
`mvn validate`
To fix non-complying files
`mvn fmt:format`
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [x] - Adopt `fmt-maven-plugin`
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3740
### How should this be tested?
CI passs
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes for contributors
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#3158 from jongyoul/ZEPPELIN-3740 and squashes the following commits:
f06c50e90 [Jongyoul Lee] Adopt `fmt-maven-plugin` for google-java-format Adopt all files
### What is this PR for?
This PR is trying to refactor method getInterpreterInTheSameSession in Interpreter, so that each interpreter don't need to reimplement its own and can reduce many code duplication.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3621
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3069 from zjffdu/ZEPPELIN-3621 and squashes the following commits:
06a6cc624 [Jeff Zhang] ZEPPELIN-3621. refactor getInterpreterInTheSameSession of Interpreter to reduce code duplication
### What is this PR for?
fix pandasql query over dataframe for `master`. the reason is that it can't load `bootstrap_sql.py` from resources because path of path
### What type of PR is it?
[Bug Fix]
### Todos
* [x] - fix path to use relative
* [x] - enable tests
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3625
### How should this be tested?
follow the steps in issue
### Screenshots (if appropriate)
before:
<img width="1268" alt="screen shot 2018-07-15 at 7 08 24 pm" src="https://user-images.githubusercontent.com/1642088/42732816-822682f0-8862-11e8-9d55-dd7d00b09b48.png">
after:
<img width="1239" alt="screen shot 2018-07-15 at 7 03 59 pm" src="https://user-images.githubusercontent.com/1642088/42732771-e712f848-8861-11e8-9ab9-eeb479082d1d.png">
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Khalid Huseynov <khalidhnv@gmail.com>
Closes#3071 from khalidhuseynov/fix-master/ZEPPELIN-3625 and squashes the following commits:
33aad7589 [Khalid Huseynov] fix matplotlib test
f94139b33 [Khalid Huseynov] enable all python tests
fd5685552 [Khalid Huseynov] remove unnecessary test
bde2316c7 [Khalid Huseynov] add test
8f9541411 [Khalid Huseynov] fix test setup not to interpret empty line
3a561d247 [Khalid Huseynov] fix from absolute to relative path in resources
### What is this PR for?
This is to add password for dynamic forms. It is almost the same as input box but but with invisible input. And there's no default value for password form, as it doesn't make sense to provide default value for password.
### What type of PR is it?
[ Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2528
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3041 from zjffdu/ZEPPELIN-2528 and squashes the following commits:
084abd39c [Jeff Zhang] ZEPPELIN-2528. Add a password text input to the ZeppelinContext
### What is this PR for?
Fix the code style issue of python module
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3163
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3023 from zjffdu/ZEPPELIN-3163 and squashes the following commits:
e89c41108 [Jeff Zhang] [ZEPPELIN-3163]. Fix checkstyle of python
### What is this PR for?
This PR introduce the bi directional RPC framework between ZeppelinServer and InterpreterProcess on top of thrift, see the design doc here. https://docs.google.com/document/d/1B_4b7UuvE7RwbfpTjxV2RSxdDeF-p9vXVYC8ace76-Q/edit?usp=sharing
This PR is pretty big, most of the code changes is due to regeneration of thrift code.
Another improvement is introduce the builder pattern of InterpreterContext as the arguments of InterpreterContext constructor is too many.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2035
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2986 from zjffdu/ZEPPELIN-2035 and squashes the following commits:
c02d5587a [Jeff Zhang] ZEPPELIN-2035. BI directional RPC framework between ZeppelinServer and InterpreterProcess on top of thrift
### What is this PR for?
This PR is trying to remove the code duplication between PySparkInterpreter and PythonInterpreter. So here's the main things this PR did:
* PySparkInterpreter extends PythonInterpreter
* PySparkInterpreterTest extends PythonInterpreterTest so that we can verify PySparkInterpreter can do whatever PythonInterpreter can do
* Move interpreter/lib/python/backend_zinline.py and interpreter/lib/python/mpl_config.py into python module, so that python module can ship these resources together.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3375
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2919 from zjffdu/ZEPPELIN-3375 and squashes the following commits:
738c6c5 [Jeff Zhang] ZEPPELIN-3375. Make PySparkInterpreter extends PythonInterpreter
### What is this PR for?
This PR address the remaining issue of ZEPPELIN-3362, and also add more test for ZeppelinContext.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3364
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2894 from zjffdu/ZEPPELIN-3364 and squashes the following commits:
891f1e1 [Jeff Zhang] ZEPPELIN-3364. Followup of ZEPPELIN-3362, improve ZeppelinContext & add more test
### What is this PR for?
Unify the ZeppelinContext of PythonInterpreter & IPythonInterpreter into one file to avoid code duplication.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3362
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2890 from zjffdu/ZEPPELIN-3362 and squashes the following commits:
b5dcbc9 [Jeff Zhang] ZEPPELIN-3362. Unify ZeppelinContext of PythonInterpreter & IPythonInterpreter
### What is this PR for?
```
%python
import time
print("Hello")
time.sleep(0.5) # in case of Ipython kernel, print may not immediately flushed and cleared.
z.getInterpreterContext().out().clear()
print("world")
```
Expected to print `world` only.
This worked in zeppelin-0.7.x and it'll be nice keep this feature in the future version.
### What type of PR is it?
Bug fix | Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3284
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Lee moon soo <moon@apache.org>
Closes#2829 from Leemoonsoo/ZEPPELIN-3284 and squashes the following commits:
e0dcda61f [Lee moon soo] make z.getInterpreterContext().out().clear() work in IPythonInterprete
224a2df8e [Lee moon soo] make z.getInterpreterContext().out().clear() work in PythonInterpreter
### What is this PR for?
Remove Redundant method `InterpreterOutput.getInterpreterResultMessages()`
### What type of PR is it?
Refactoring
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3283
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#2828 from Leemoonsoo/ZEPPELIN-3283 and squashes the following commits:
12771a145 [Lee moon soo] remove unused import
82bda9c02 [Lee moon soo] Remove redundant method getInterpreterResultMessages()
### What is this PR for?
trivial change for property name renaming. Followup of #2802
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2813 from zjffdu/HOTFIX_GRPC and squashes the following commits:
d899a29 [Jeff Zhang] [HOTFIX] rename zeppelin.ipython.grpc.framesize to zeppelin.ipython.grpc.message_size
### What is this PR for?
Add one new property `zeppelin.ipython.grpc.framesize` which is an advanced configuration. By default it is 32M which should be sufficient for most of scenarios.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3236
### How should this be tested?
* Unit test is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2802 from zjffdu/ZEPPELIN-3236 and squashes the following commits:
ffce774 [Jeff Zhang] ZEPPELIN-3236. Make grpc framesize configurable
### What is this PR for?
Fix the unicode issues in python2.
### What type of PR is it?
[Bug Fix | Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3239
### How should this be tested?
* Unit test is added
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2810 from zjffdu/ZEPPELIN-3239 and squashes the following commits:
70c8b4c [Jeff Zhang] ZEPPELIN-3239. unicode characters in an iPython paragraph makes Spark interpreter irrsponsive
### What is this PR for?
This is for the refactoring of SparkInterpreter. See design doc. https://docs.google.com/document/d/1AfGg3aGXonDyri1jrP4MMFT4Y4j3wpN1t8kL-GAKSUc/edit?usp=sharing
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3111
### How should this be tested?
* Unit test is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2709 from zjffdu/ZEPPELIN-3111 and squashes the following commits:
aae4b09 [Jeff Zhang] ZEPPELIN-3111. Refactor SparkInterpreter
### What is this PR for?
This is for better code completion support for IPythonInterpreter
### What type of PR is it?
[Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3136
### How should this be tested?
* Unit test is updated and Manually verified.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2725 from zjffdu/ZEPPELIN-3136 and squashes the following commits:
1179fc8 [Jeff Zhang] ZPEPELIN-3136. IPython Code Completion Improvement
### What is this PR for?
Added dynamic forms for Note. All paragraphs has access to note forms.
### What type of PR is it?
[Feature]
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1363
### How should this be tested?
Create global interpreter JDBC and Spark. Create note with 2 paragraphs
```
%pyspark
print("Textbox paragraph " + z.textbox('input', 'default'))
print("Textbox note " + z.noteTextbox('note_input', 'default_note'))
print("Select paragraph " + z.select("sel", [("1","opt1"), ("2","opt2"), ("3","opt3")]))
print("Select note " + z.noteSelect("sel_note", [("1","noteOpt1"), ("2","noteOpt2"), ("3","noteOpt3")]))
options = [("key1","Name1"), ("key2","Name2")]
print("Checkbox paragraph "+ " and ".join(z.checkbox("chk", options, ["key1"])))
print("Checkbox note "+ " and ".join(z.noteCheckbox("chk_note", options, ["key1","key2"])))
```
```
%jdbc
select '$${checkbox:chk_note=key1|key2,key1|key2} $${note_input} ${note_input=sameName}'
```
### Screenshots (if appropriate)
1) native forms
2) remove
3) simple forms
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: tinkoff-dwh <tinkoff.dwh@gmail.com>
Closes#2641 from tinkoff-dwh/ZEPPELIN-1363 and squashes the following commits:
3ee4826 [tinkoff-dwh] [ZEPPELIN-1363] autosave textbox
f30033a [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1363
29eaca2 [tinkoff-dwh] [ZEPPELIN-1363] fix tests
bf8194e [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1363
9b2f3e9 [tinkoff-dwh] [ZEPPELIN-1363] button to remove form
c566462 [tinkoff-dwh] [ZEPPELIN-1363] note dynamic forms (simple; native: spark, python)
### What is this PR for?
Trivial fix for matplotlib fail in `IPythonInterpreter`. Besides that, I also make some minor changes on zeppelin code to improve logging.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2990
### How should this be tested?
UT is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2622 from zjffdu/ZEPPELIN-2990 and squashes the following commits:
55e6f88 [Jeff Zhang] fix unit test
1e57afe [Jeff Zhang] ZEPPELIN-2990. Matplotlib sometimes fails in IPythonInterpreter
### What is this PR for?
Follow up of #2577. Main changes on Interpreter
* Add throw `InterpreterException` which is checked exception for the abstract methods of `Interpreter`, this would enforce the interpreter implementation to throw `InterpreterException`.
* field name refactoring.
* `property` -> `properties`
* `getProperty()` --> `getProperties()`
* Introduce launcher layer for interpreter launching. Currently we only use shell script to launch interpreter, but it could be any other service or component to launch interpreter, such as livy server , other 3rd party tools or even we may create a separate module for interpreter launcher
* abstract cass `InterpreterLauncher`
* For now, only 2 implementation: `ShellScriptLauncher` & `SparkInterpreterLauncher`. We could add method in class `Interpreter` to allow interpreter to specify its own launcher class, but it could be future work.
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2685
### How should this be tested?
Unit test is covered. `ShellScriptLauncherTest` & `SparkInterpreterLauncherTest`
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2592 from zjffdu/ZEPPELIN-2685 and squashes the following commits:
17dc2f1 [Jeff Zhang] address comments
e545cc3 [Jeff Zhang] ZEPPELIN-2685. Improvement on Interpreter class
### What is this PR for?
Fixed a problem where conda's library was not installed and reflected properly due to changed python interpreter execution flow and structures.
ORIGINAL : https://github.com/apache/zeppelin/pull/2574
### What type of PR is it?
Bug Fix
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2921
### How should this be tested?
Please run the following command line for each paragraph.
If the module such as scipy is normally imported, it is a success.
`%python.conda create --name Hello6 python=2.7`
`%python.conda activate Hello6`
`%python.conda install seaborn pandas numpy scipy matplotlib`
```
%python
import scipy as sp
import seaborn as sns
```
`%python.conda deactivate`
### Screenshots (if appropriate)
#### Before
#### After
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: CloverHearts <cloverheartsdev@gmail.com>
Closes#2578 from cloverhearts/ZEPPELIN-2921 and squashes the following commits:
a09f6801 [CloverHearts] added test case for conda env
9b797269 [CloverHearts] add conda env option on list
8dfb33b1 [CloverHearts] replace conda install environment name
### What is this PR for?
This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well.
For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter.
Main features:
* IPython interpreter support
** All the ipython features are available, including visualization, ipython magics.
* ZeppelinContext support
* Streaming output support
* Support Ipython in PySpark
Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here.
In unit test, I only verify the following 3 popular visualization library. could add more later.
* matplotlib
* bokeh
* ggplot
### What type of PR is it?
[Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2753
### How should this be tested?
Unit test is added.
### Screenshots (if appropriate)
Verify bokeh in IPython Interpreter
Verify matplotlib
Verify ZeppelinContext
Verify Streaming
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits:
e869f31 [Jeff Zhang] address comments
b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
### What is this PR for?
If you override the reserved word ZeppelinContext such as `z` in the python language, the whole paragraph output problem occurred.
I have taken care to avoid this issue.
`z` == `_zc` == `zeppelin context`
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-802
### How should this be tested?
The error should not occur in the following situations:
```
%python
z = 1
print("Hello Zeppelin")
```
```
%pyspark
z = 1
print("Hello Zeppelin")
```
### Screenshots (if appropriate)
#### before
#### after
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: CloverHearts <cloverheartsdev@gmail.com>
Author: cloverhearts <cloverheartsdev@gmail.com>
Closes#2207 from cloverhearts/ZEPPELIN-802-pyspark-zeppelin-context and squashes the following commits:
cc986010 [CloverHearts] added completion on namespace
14695cb8 [CloverHearts] Recovering a member name that is not associated with a namespace
31af92ab [CloverHearts] fix test case _zc to __zeppelin__
6697d677 [CloverHearts] apply to namespace and replace name _zc to __zeppelin__
ca795cff [cloverhearts] replace output
1d372df4 [cloverhearts] change name logger
4e8435ac [CloverHearts] added test case on python
b6b804ad [CloverHearts] replace name zeppelin context on python
9fbf70d6 [CloverHearts] fix pyspark test case
987e2118 [CloverHearts] added test code
5da3d6ed [CloverHearts] replace name zeppelin context on pyspark
### What is this PR for?
I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter.
1. not working streaming output.
- https://issues.apache.org/jira/browse/ZEPPELIN-2225
2. printed "..." when there is indent in the python code.
- https://issues.apache.org/jira/browse/ZEPPELIN-1929
3. very slow output of matplotlib
- https://issues.apache.org/jira/browse/ZEPPELIN-1894
- https://issues.apache.org/jira/browse/ZEPPELIN-1360
4. Unexpected output of matplotlib.
- https://issues.apache.org/jira/browse/ZEPPELIN-2107
so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues.
and I am going to recreate conda, docker for python interpreter ASAP.
### What type of PR is it?
Bug Fix | Hot Fix | Refactoring
### How should this be tested?
1. not working streaming output.
```
import time
for x in range(0, 5):
print x
time.sleep(1)
```
2. printed "..." when there is indent in the python code.
```
def fn():
print("hi")
fn()
```
3. very slow output of matplotlib.
```
import matplotlib
import sys
import matplotlib.pyplot as plt
plt.plot([1,2,3])
```
4. Unexpected output of matplotlib.
```
import matplotlib.pyplot as plt
import matplotlib as mpl
# Make a figure and axes with dimensions as desired.
fig = plt.figure(figsize=(8, 3))
ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15])
ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15])
ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15])
# Set the colormap and norm to correspond to the data for which
# the colorbar will be used.
cmap = mpl.cm.cool
norm = mpl.colors.Normalize(vmin=5, vmax=10)
# ColorbarBase derives from ScalarMappable and puts a colorbar
# in a specified axes, so it has everything needed for a
# standalone colorbar. There are many more kwargs, but the
# following gives a basic continuous colorbar with ticks
# and labels.
cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap,
norm=norm,
orientation='horizontal')
cb1.set_label('Some Units')
# The second example illustrates the use of a ListedColormap, a
# BoundaryNorm, and extended ends to show the "over" and "under"
# value colors.
cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c'])
cmap.set_over('0.25')
cmap.set_under('0.75')
# If a ListedColormap is used, the length of the bounds array must be
# one greater than the length of the color list. The bounds must be
# monotonically increasing.
bounds = [1, 2, 4, 7, 8]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap,
norm=norm,
# to use 'extend', you must
# specify two extra boundaries:
boundaries=[0] + bounds + [13],
extend='both',
ticks=bounds, # optional
spacing='proportional',
orientation='horizontal')
cb2.set_label('Discrete intervals, some other units')
# The third example illustrates the use of custom length colorbar
# extensions, used on a colorbar with discrete intervals.
cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.],
[1., .8, 0.], [1., .4, 0.]])
cmap.set_over((1., 0., 0.))
cmap.set_under((0., 0., 1.))
bounds = [-1., -.5, 0., .5, 1.]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap,
norm=norm,
boundaries=[-10] + bounds + [10],
extend='both',
# Make the length of each extension
# the same as the length of the
# interior colors:
extendfrac='auto',
ticks=bounds,
spacing='uniform',
orientation='horizontal')
cb3.set_label('Custom extension lengths, some other units')
plt.show()
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@zepl.com>
Author: Lee moon soo <moon@apache.org>
Author: HyungSung <hsshim@nflabs.com>
Closes#2106 from astroshim/py4jPythonInterpreter and squashes the following commits:
c9b195b [HyungSung] Merge pull request #16 from Leemoonsoo/py4jdocker
e511ebe [Lee moon soo] add PythonDockerInterpreter to interpreter-setting.json
a76b0d8 [Lee moon soo] fix test on python3
2eb5de7 [Lee moon soo] Fix PythonDockerInterpreterTest.java test
9fcf144 [Lee moon soo] Make python docker interpreter work using py4j
8a016c9 [astroshim] Merge branch 'master' into py4jPythonInterpreter
aad7ee8 [astroshim] fix testcase
ac92cdb [astroshim] fix python interpreter testcase
e8570d2 [astroshim] fix ci for pandassql
be5db4d [astroshim] fix pandas sql testcase
f8e19be [astroshim] fix matplotlib testcase
046db88 [astroshim] add testcase
e49ad24 [astroshim] add pandas
60e9820 [astroshim] bug fix about copying library
574bd21 [astroshim] fix interpreter-setting error
a48df58 [astroshim] Merge branch 'master' into py4jPythonInterpreter
3c9585f [astroshim] update interpreter-setting.json
a50179e [astroshim] add conda interpreter
cbbc15c [astroshim] fix py4j path
5ae5120 [astroshim] fix interpreter-setting
f17bff4 [astroshim] fix testcase failure.
af097ac [astroshim] add testcase
c3f5b78 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into py4jPythonInterpreter
1395875 [astroshim] removed unnecessary code.
276011e [astroshim] add py4j lib
7304919 [astroshim] initialize python interpreter using py4j
### What is this PR for?
Add missing commands to the `python.conda` interpreter
- `conda info`
- `conda list`
- `conda create`
- `conda install`
- `conda uninstall (alias of remove)`
- `conda env *`
#### Implementation Detail
The reason I modified `PythonProcess` is due to NPE
```java
// https://github.com/apache/zeppelin/blob/master/python/src/main/java/org/apache/zeppelin/python/PythonProcess.java#L107-L118
public String sendAndGetResult(String cmd) throws IOException {
writer.println(cmd);
writer.println();
writer.println("\"" + STATEMENT_END + "\"");
StringBuilder output = new StringBuilder();
String line = null;
// NPE when line is null
while (!(line = reader.readLine()).contains(STATEMENT_END)) {
logger.debug("Read line from python shell : " + line);
output.append(line + "\n");
}
return output.toString();
}
```
```
java.lang.NullPointerException
at org.apache.zeppelin.python.PythonProcess.sendAndGetResult(PythonProcess.java:113)
at org.apache.zeppelin.python.PythonInterpreter.sendCommandToPython(PythonInterpreter.java:250)
at org.apache.zeppelin.python.PythonInterpreter.bootStrapInterpreter(PythonInterpreter.java:272)
at org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:100)
at org.apache.zeppelin.python.PythonCondaInterpreter.restartPythonProcess(PythonCondaInterpreter.java:139)
at org.apache.zeppelin.python.PythonCondaInterpreter.interpret(PythonCondaInterpreter.java:88)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:494)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
```
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [x] - info
* [x] - list
* [x] - create
* [x] - install
* [x] - uninstall (= remove)
* [x] - env *
### What is the Jira issue?
[ZEPPELIN-1917](https://issues.apache.org/jira/browse/ZEPPELIN-1917)
### How should this be tested?
1. Install [miniconda](http://conda.pydata.org/miniconda.html)
2. Make sure that your python interpreter can use `conda` (check the Interpreter Binding page)
3. Remove `test` conda env since we will create in the following section
```sh
$ conda env remove --yes --name test
```
4. Run these commands with `%python.conda`
```
%python.conda info
%python.conda env list
%python.conda create --name test
# you should be able to see `test` in the list
%python.conda env list
%python.conda activate pymysql
%python.conda install pymysql
# you should be able to import
%python
import pymysql.cursors
%python.conda uninstall pymysql
%python.conda deactivate pymysql
# you should be able to see `No module named pymysql.cursor` since we deactivated
%python
import pymysql.cursors
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#1868 from 1ambda/ZEPPELIN-1917/improve-conda-interpreter and squashes the following commits:
3ba171a [1ambda] fix: Wrap output style
292ed6d [1ambda] refactor: Throw exception in runCommand
2d4aa7d [1ambda] test: Add some tests
49a4a11 [1ambda] feat: Supports other env commands
6eb7e92 [1ambda] fix: NPE in PythonProcess
9c5dd86 [1ambda] refactor: Activate, Deactivate
f955889 [1ambda] fix: minor
935cb89 [1ambda] refactor: Abstract commands
b1c4c9f [1ambda] feat: Add conda remove (uninstall)
e539c42 [1ambda] feat: Add conda install
4f58fa2 [1ambda] feat: Add conda create
7da132d [1ambda] docs: Add missing conda list description
929ca8a [1ambda] feat: Make conda output beautiful
0c6ebb4 [1ambda] feat: Add list conda command
017c76f [1ambda] refactor: Import InterpreterResult.{Code, Type} to short codes
b8a5154 [1ambda] refactor: Simplify exception flow so private funcs don't need care exceptions
64d4bef [1ambda] style: Rename some funcs
afc456d [1ambda] refactor: Add private to member vars
f36fc74 [1ambda] feat: Add info command
2eb9bf5 [1ambda] style: Remove useless newlines
bd2564e [1ambda] refactor: PythonCondaInterpreter.interpret
f0d69bc [1ambda] fix: Use specific command for env list in conda
### What is this PR for?
Because `conda run` command removed since version `4.0.9`, PythonCondaInterpreter not working after the `conda-4.0.9`.
This PR fixes this issue.
I tested conda-4.2.12 and conda-4.0.9 .
### What type of PR is it?
Bug Fix | Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1724
### How should this be tested?
Please refer to https://github.com/apache/zeppelin/pull/1645
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@nflabs.com>
Closes#1699 from astroshim/ZEPPELIN-1724 and squashes the following commits:
294b6f9 [astroshim] refactoring and fix testcase
8c3fbd3 [astroshim] fix conda version
### What is this PR for?
Currently a paragraph can display only a single type of result.
For example
```
print("""
%text textout
%html htmlout
""")
```
will display only last display system detected, "htmlout".
This pr implements multiple results supports, so not only the last, but also all the display systems in a paragraph outputs can be displayed.
To do that, Note.json format need to be changed. from
```
paragraph : [
{
result : {
code: "",
type: "",
msg: ""
},
config : {
graph : {},
...
},
...
},
...
],
...
```
to
```
paragraph : [
{
results : {
{
code: "",
msg: [
{
type: "",
data: ""
},
{
type: "",
data: ""
},
...
]
},
config : {
results : [
{
graph : {},
},
{
graph : {},
},
...
]
},
...
},
...
],
...
```
### What type of PR is it?
Improvement
### Todos
* [x] - Make InterpreterResult and InterpreterOutput support multiple display system
* [x] - Automatic migration old format to new format
* [x] - Render multiple results in front-end
* [x] - Take care Importing old notebook format
* [x] - Make helium framework support multiple results
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-212
### How should this be tested?
run following code in a single spark paragraph.
```
%spark
// default output is text
(1 to 3).foreach{i=>
println(new java.util.Date())
Thread.sleep(1000)
}
// print something in html
println(s"""%html <h3><font style="color:blue">Some HTML</font></h3>""")
// create table
println(s"""%table key\tvalue
sun\t100
moon\t200
""")
// display text again
println("""%text""")
(1 to 3).foreach{i=>
println(new java.util.Date())
Thread.sleep(1000)
}
// another table
Thread.sleep(1000)
println(s"""%table key\tvalue
apple\t100
banana\t200
""")
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? yes
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes#1658 from Leemoonsoo/ZEPPELIN-212 and squashes the following commits:
0c6a13c [Lee moon soo] Merge remote-tracking branch 'origin/master' into ZEPPELIN-212
519cf23 [Lee moon soo] Update PythonInterpreterPandasSqlTest
e6268ac [Lee moon soo] Update PythonInterpreterMatplotlibTest.java
f5034b8 [Lee moon soo] Merge remote-tracking branch 'origin/master' into ZEPPELIN-212
d5981d5 [Lee moon soo] Merge remote-tracking branch 'origin/master' into ZEPPELIN-212
a1fe729 [Lee moon soo] document note format change
282504e [Lee moon soo] Merge remote-tracking branch 'origin/master' into ZEPPELIN-212
7ab6679 [Lee moon soo] update selenium test
6a897a5 [Lee moon soo] Update rest-api doc
cbbd58a [Lee moon soo] update unittest
e89c9b8 [Lee moon soo] restore tutoral note
3ba37b7 [Lee moon soo] Let old version import note without error
d09e03f [Lee moon soo] enable helium only in the last result
6682908 [Lee moon soo] Remove unnecessary listener method
0f1d28f [Lee moon soo] Merge branch 'master' into ZEPPELIN-212
73b3a81 [Lee moon soo] update selenium test
e69f1a1 [Lee moon soo] update test
f12230f [Lee moon soo] Remove unnecessary newline in test
26e201a [Lee moon soo] Update testcase
b01d70f [Lee moon soo] Make helium app work
0a5b4d1 [Lee moon soo] Update HeliumApplicationFactoryTest
d8ca07f [Lee moon soo] update NotebookTest
aaf9778 [Lee moon soo] fix r test
3e43df4 [Lee moon soo] make zeppelin-web test pass
1d6fd3e [Lee moon soo] fix compile errors
0156ffc [Lee moon soo] take care angular object
804768d [Lee moon soo] Take care output streaming
57d6b2f [Lee moon soo] Render multiple results
95b6037 [Lee moon soo] Multiple results
### What is this PR for?
Take 2 of #1618 because I had some earlier problems with rebasing. Since, then I have added some new features, namely:
- Matplotlib integration tests for pyspark
- `install_external_dependencies.sh` which conditionally installs the R or python dependencies based on the specified build profile in `.travis.yml`. This saves several minutes of time for a few of the build profiles since the R dependencies are compiled from source and therefore take quite a bit of time to install.
- The extra python unit tests which require external dependencies (`matplotlib` and `pandas`) are now relegated to two separate build profiles. This is done primarily to efficiently test both Python 2 and 3.
- Some minor bugs in the python and pyspark interpreters (mostly with respect to python 3 compatibility) were caught as a result of these tests, and are also fixed in this PR.
### What type of PR is it?
Improvement and Bugfix
### What is the Jira issue?
[ZEPPELIN-1639](https://issues.apache.org/jira/browse/ZEPPELIN-1639)
### How should this be tested?
CI tests should be green!
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Alex Goodman <agoodm@users.noreply.github.com>
Closes#1632 from agoodm/ZEPPELIN-1639 and squashes the following commits:
01380c2 [Alex Goodman] Make sure python 3 profile uses scala 2.11
363019e [Alex Goodman] Use spark 2.0 with python 3
a4f43af [Alex Goodman] Update comments in .travis.yml
5a60181 [Alex Goodman] Isolate python tests
73663f6 [Alex Goodman] Update tests for new InterpreterContext constructor
5709c5d [Alex Goodman] Re-add pyspark to build profile
ee95d67 [Alex Goodman] Move python 3 tests to all modules with spark 2.0
3a76958 [Alex Goodman] Travis
42da31c [Alex Goodman] Shorten python version
b6b88be [Alex Goodman] Add python dependencies to .travis.yml
