### What is this PR for?
This is to run python interpreter in yarn mode and you can also customize the python runtime via conda. Check the `python.md` for more details
### What type of PR is it?
[Feature | Documentation]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5330
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#4097 from zjffdu/ZEPPELIN-5330 and squashes the following commits:
b6a957fd3e [Jeff Zhang] address code review comments
bdb4dcbbfa [Jeff Zhang] support hdfs file for
ee8d11b8fd [Jeff Zhang] minor code refacotring
a7dca0fd35 [Jeff Zhang] address comments
5fc3e86543 [Jeff Zhang] Fix Ci
e4a6c7146a [Jeff Zhang] address comments
f7e6b13d4d [Jeff Zhang] save
d5828d203b [Jeff Zhang] update python
3c963b6b61 [Jeff Zhang] update
fa36c5b997 [Jeff Zhang] update jupyter
e25c6b3f3b [Jeff Zhang] address comment
d27f580216 [Jeff Zhang] [ZEPPELIN-5330]. Support conda env for python interpreter in yarn mode
### What is this PR for?
This is a hotfix to fix the ci failure due to dns reverse lookup issue. Some of the code is copied from pulsar project.
b4300db307
### What type of PR is it?
[ Hot Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
*
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang <zjffdu@apache.org>
Closes#4096 from zjffdu/ci_test and squashes the following commits:
ed91ba328 [Jeff Zhang] fix comment
6a0d9fd1c [Jeff Zhang] save
### What is this PR for?
This PR fixes NPE exceptions in our tests.
### What type of PR is it?
- Hot Fix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5187
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Philipp Dallig <philipp.dallig@gmail.com>
Closes#4012 from Reamer/npe_interpreter_output and squashes the following commits:
b45c0f570 [Philipp Dallig] Use new constructor introduced with ZEPPELIN-5170
### What is this PR for?
This PR improve the github action CI
* Disable unstable test BasePythonInterpreterTest#testRedefinitionZeppelinContext
* Fix the failure of flink 1.10, this is due to python package issue.
The github action still fails sometimes, because of network issue or other weird issue, we have to rerun it again.
### What type of PR is it?
[ Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5160
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3994 from zjffdu/ZEPPELIN-5160 and squashes the following commits:
491e1409f [Jeff Zhang] address comment
6edcfe692 [Jeff Zhang] [ZEPPELIN-5160]. Improve github action CI
### What is this PR for?
This PR will change the CI-System from [Travis](https://travis-ci.com/) to [GitHub Actions](https://github.com/features/actions)
**Advantages:**
- [high usage limits](https://docs.github.com/en/free-pro-teamlatest/actions/reference/usage-limits-billing-and-administration#usage-limits)
- No CI setup is required from the contributors
- good integration with GitHub Pull request
**Disadvantage:**
- At the moment we have some flappy tests and external dependencies. Both can cause a failed test, but we cannot restart a single job, only the entire workflow.
**Miscellaneous:**
- CI is triggered with every push event
- CI is triggered when PR event is open and synchronized
- Selenium test is updated to python3
- The Livy-Python2 test is dropped because Python2 has reached [the end of its lifetime](https://www.python.org/doc/sunset-python-2/)
### What type of PR is it?
- Improvement
### Todos
* [ ] - Feedback for job splitting
* [x] - What is the purpose of the Jenkins job?
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4385
### How should this be tested?
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Philipp Dallig <philipp.dallig@gmail.com>
Closes#3986 from Reamer/github_actions and squashes the following commits:
eb13433a8 [Philipp Dallig] Goodbye travis
3de27c699 [Philipp Dallig] Adjust README
8302f5349 [Philipp Dallig] Adjust template for pull request
76d9bb22e [Philipp Dallig] Adjustments documentation
08155783e [Philipp Dallig] Remove travis_check.py
e865f2fc1 [Philipp Dallig] Remove "travis" in source code
6790b6e42 [Philipp Dallig] Add core.yml
27330791f [Philipp Dallig] Rewrite selenium tests to python3
1772024fc [Philipp Dallig] Add frontend tests
c81f48492 [Philipp Dallig] Add github actions workflow (rat)
18d5dbecb [Philipp Dallig] some cleanup
### What is this PR for?
In this PR, I would only print exception message when it is SQLException, otherwise would print the full stacktrace, because in this case it is most likely due to jdbc interpreter's internal error.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5103
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3952 from zjffdu/ZEPPELIN-5103 and squashes the following commits:
4001e1b84 [Jeff Zhang] [ZEPPELIN-5103]. Simply jdbc interpreter error message
### What is this PR for?
This PR is to fix the issue of FLINK-16936 by a workaround, already creating tableenv before execution scala or python code. Building tablenv is pretty light which won't cost much time. So it is acceptable for this workaround. Another this PR try to fix is the ClassLoader issue for PyFlinkInterpreter. This PR will always set classloader before executing python code so that pyflink api can call udf defined in scala.
### What type of PR is it?
[Bug Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4714
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3715 from zjffdu/ZEPPELIN-4714 and squashes the following commits:
55d613576 [Jeff Zhang] [ZEPPELIN-4714]. Flink table api doesn't work in multiple threads
### What is this PR for?
The root cause of this issue is that we didn't redirect java output to interpreter output. This PR fix it via redirect java output before interpreting python code in both PySparkInterpreter & IPySparkInterpreter. Unit test is also added to verify this feature.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4692
### How should this be tested?
* Unit test is added, also manually tested it
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3696 from zjffdu/ZEPPELIN-4692 and squashes the following commits:
e1a5ead7e [Jeff Zhang] [ZEPPELIN-4692]. zeppelin pyspark doesn't print java output
### What is this PR for?
I noticed that the Zeppelin project mixes "commons-lang" and "commons-lang3". I have (hopefully) changed all imports to `commons-lang3`
I use `commons-lang3` 3.7, because this version was used in several submodules.
### What type of PR is it?
Improvement
### Todos
* [ ] - Is the replacement of `getFullStackTrace` to `getStackTrace` right?
* [ ] - commons-lang isn't deleted from main `pom.xml`. Okay?
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4686
### How should this be tested?
* Unit-Tests withTravis CI https://travis-ci.org/github/Reamer/zeppelin/builds/663565663
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Philipp Dallig <philipp.dallig@gmail.com>
Closes#3691 from Reamer/commons-lang3 and squashes the following commits:
bf12fc07c [Philipp Dallig] Update to org.apache.commons commons-lang3
### What is this PR for?
Straightforward PR to just rename BaseZeppelinContext to ZeppelinContext
### What type of PR is it?
[ Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4616
### How should this be tested?
* Not necessary
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3646 from zjffdu/ZEPPELIN-4616 and squashes the following commits:
f4eb0f5ba [Jeff Zhang] [ZEPPELIN-4616]. Rename BaseZeppelinContext to ZeppelinContext
### What is this PR for?
This PR would replace all the special characters(\t, \n, \r\n) in table content with white space. so that it won't break the table format display.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4611
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3643 from zjffdu/ZEPPELIN-4611 and squashes the following commits:
d12eea943 [Jeff Zhang] [ZEPPELIN-4611]. Fetching rows with newline character (\n) breaks entire table
### What is this PR for?
Python Interpreter fails on Windows because of classpath issue
### What type of PR is it?
Bug Fix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4564#
THIS SOFTWARE IS CONTRIBUTED SUBJECT TO THE TERMS OF THE APACHE SOFTWARE FOUNDATION SOFTWARE GRANT AND CORPORATE CONTRIBUTOR LICENSE AGREEMENT VERSION R190612.
THIS SOFTWARE IS LICENSED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OF NON-INFRINGEMENT, ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THIS SOFTWARE MAY BE REDISTRIBUTED TO OTHERS ONLY BY EFFECTIVELY USING THIS OR ANOTHER EQUIVALENT DISCLAIMER IN ADDITION TO ANY OTHER REQUIRED LICENSE TERMS.
Author: Muhammad Taufiq <Muhammad.Taufiq@morganstanley.com>
Closes#3604 from Muhammad-ms/zeppelin9_file_separator and squashes the following commits:
a09b6463a [Muhammad Taufiq] Changed path separator to allow it to work on windows as well
### What is this PR for?
Use the SqlSplitter in `zeppelin-interpreter` to split sql and execute in SparkSqlInterpreter. Nothing changes for the previous single sql statement paragraph. But just multiple result will be displayed for multiple sql statements.
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4522
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3579 from zjffdu/ZEPPELIN-4522 and squashes the following commits:
eda573649 [Jeff Zhang] fix failed test
68d5a30c8 [Jeff Zhang] Add test for no sql but just 2 comments
4ff15e4fb [Jeff Zhang] address comment
bc3c1feff [Jeff Zhang] [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
### What is this PR for?
In ZEPPELIN-4480, we introduce zeppelin-jupyter-adapter which build a bridge between zeppelin and jupyter, zeppelin can extends this class to leverage any kernel of jupyter.
After a second thought, I think it is not enough. We can allow user to use jupyter kernel directly instead of implement one by one. So the purpose is to allow user use jupyter kernel like this
```
%jupyter(kernel=ipython)
python_code
...
```
Of course user can enhance the jupyter kernel in Zeppelin, e.g. IPythonInterpreter of Zeppelin can enhance the ipython kernel by adding support of ZeppelinContext
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4497#
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3558 from zjffdu/ZEPPELIN-4497 and squashes the following commits:
2efdc1e0e [Jeff Zhang] [ZEPPELIN-4497] Change zeppelin-jupyter-adapter to zeppelin-jupyter-interpreter
### What is this PR for?
This PR is to polish the python interpreter document.
### What type of PR is it?
[Documentation]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4437
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3538 from zjffdu/ZEPPELIN-4437 and squashes the following commits:
48163d089 [Jeff Zhang] ZEPPELIN-4437. Update python document
### What is this PR for?
This PR move the ipython code into module zeppelin-jupyter-adapter. And ipython will just depends on this module. zeppelin-jupyter-adapter could be used for connecting any jupyter kernel.
### What type of PR is it?
[ Feature | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4480
### How should this be tested?
* Ci pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3551 from zjffdu/ZEPPELIN-4480 and squashes the following commits:
78453473e [Jeff Zhang] [ZEPPELIN-4480]. Move the ipython code into a general jupyter kernel bridge
### What is this PR for?
This PR is to align all the functions in ZeppelinContext in Scala/Python/R
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4419
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3511 from zjffdu/ZEPPELIN-4419 and squashes the following commits:
7d5cdc853 [Jeff Zhang] [ZEPPELIN-4419]. Align functions in ZeppelinContext of Scala/Python/R
### What is this PR for?
This is allow user to specify resourceName when they want to save the paragraph result into ResourcePool. Before this PR, user don't have control on what name of the saving resource name. It is associated with noteId and paragraphId, but this is not a good solution. Because when you clone the note, noteId will be changed, and you have to change the noteId in code as well. This PR is trying to allow user to set resource Name for the saving paragraph result.
### What type of PR is it?
[ Feature |g]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3617
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3065 from zjffdu/ZEPPELIN-3617 and squashes the following commits:
024965e64 [Jeff Zhang] ZEPPELIN-3617. Allow to specify saving resourceName as paragraph property
### What is this PR for?
HvPlot is broken due to bug in `ipython_server.py`. IPython may produce multiple types of output for one single python object. And the orders of these output matters, we should output html type over plain text type.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4396
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3515 from zjffdu/ZEPPELIN-4396 and squashes the following commits:
f71dbf231 [Jeff Zhang] [ZEPPELIN-4396]. HvPlot is broken
### What is this PR for?
`ExecutorService` is not properly shutdown due to we didn't use the correct api. We should use `shutdownNow` instead of `shutdown`. See https://stackoverflow.com/questions/11520189/difference-between-shutdown-and-shutdownnow-of-executor-service.
The effect of this issue will cause thread resource leakage.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4390
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3491 from zjffdu/ZEPPELIN-4390 and squashes the following commits:
439221492 [Jeff Zhang] [ZEPPELIN-4390]. ExecutorService is not properly shutdown
### What is this PR for?
This PR enhance method `z.angular` in `%spark.r` and `%spark.pyspark` so that angular variables defined in frontend can be shared in backend `%spark`, `%spark.r`, `%spark.pyspark`
### What type of PR is it?
[ Improvement | Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4354
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3464 from zjffdu/ZEPPELIN-4354 and squashes the following commits:
34b11e746 [Jeff Zhang] [ZEPPELIN-4354]. Enhance z.angular for more flexiable data sharing
### What is this PR for?
There's redundant line separator when there's multiple text output. The root cause is that each `%text ` will append new line separator in frontend. This PR fix this issue by removing `%text ` is the previous output type is also `%text `.
### What type of PR is it?
[Bug Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4369
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
Before
After
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3478 from zjffdu/ZEPPELIN-4369 and squashes the following commits:
8c3d380db [Jeff Zhang] [ZEPPELIN-4369] Redundant line separator for multiple text output
### What is this PR for?
This PR is to support latest flink 1.9.0. Main features:
* Support table api
* Support batch sql
* Support streaming sql
* Support both flink planner and blink planner.
* Support pyflink
### What type of PR is it?
[ Improvement | Feature | Documentation ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4273
### How should this be tested?
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? NO
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3422 from zjffdu/ZEPPELIN-4273 and squashes the following commits:
b0c94cfd6 [Jeff Zhang] [ZEPPELIN-4273] Support Flink 1.9 for Flink Interpreter
### What is this PR for?
The pr is to fix a bug that will make the **ipython** queue listener, overuse cpu. After this fix, cpu usage should be way lower.
Also there is a bit of refactor to use only one queue to ensure message will be order even with a sleep.
Also, the refactor is reducing the number of line and code duplication
### What type of PR is it?
Bug Fix / performance improvement
### Todos
* [x] - Performance improvement
### What is the Jira issue?
It is one part of the jira issue.
https://issues.apache.org/jira/browse/ZEPPELIN-4090
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3337 from AyWa/fix/ipython-cpu and squashes the following commits:
1abe50f25 [marc hurabielle] fix error message when kernel stop
30ef7f7cb [marc hurabielle] fix identation
fe20923cf [marc hurabielle] fix rebase and fix lint
e1d37b76e [marc hurabielle] increase queue size
42f3ca097 [marc hurabielle] use one queue and use sleep to improve ipython performance
### What is this PR for?
The pr is to fix a bug that will make the **ipython** `execute_interactive` hang forever if a auto `complete` call is make at the same time. (see unit test for example that is failing on master).
For now the fix is to synchronize those method : `execute` / `complete`. It will not bring regression because anyway, the kernel does not support concurrent execute and auto complete (see https://github.com/jupyter/notebook/issues/3763)
### What type of PR is it?
Bug Fix
### Todos
* [x] - unit test failing in master / succeed on this branch
* [x] - fix with lock
### What is the Jira issue?
It is one part of the jira issue. Other fix will come soon
https://issues.apache.org/jira/browse/ZEPPELIN-4091
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3336 from AyWa/fix/concurrent-auto-complete and squashes the following commits:
86dab7345 [marc hurabielle] fix rebase
5bed19496 [marc hurabielle] fix lint
6e48c1380 [marc hurabielle] try single threaded
f14d8b242 [marc hurabielle] Revert "just test ci behavior"
be6663f89 [marc hurabielle] just test ci behavior
bc2b4f6e6 [marc hurabielle] bring back test
d43f03da9 [marc hurabielle] use initIntpProperties instead of empty one
c37414cc2 [marc hurabielle] increase timeout
f7cae9538 [marc hurabielle] move synchronize near the thread check
616f0122f [marc hurabielle] add test to ensure that autocomplete and interpret can be call concurrently
409b75f0f [marc hurabielle] add lock to ensure ipython execute will not be stuck forever when complete is call
### What is this PR for?
For now, python test is missing in travis, this PR enable it. But we have to disable some test, because some test will fail, I will fix them in a followup ticket.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4234
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3408 from zjffdu/ZEPPELIN-4234 and squashes the following commits:
42d6877aa [Jeff Zhang] ZEPPELIN-4234. Python test is missing in travis
### What is this PR for?
There're several places in zeppelin to launch processes, such as interpreter process launch, python process launch. Sometimes it is very hard to diagnose if the process is failed to launch or some error happens during its lifecycle. This PR introduce ProcessLauncher to encapsulate process launch. ProcessLauncher will do most of the process launching stuff but also allow for customization. E.g. how to check the process is started properly. And when error happens, ProcessLauncher will capture the necessary error output for further diagnose.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4066
### How should this be tested?
* Ci pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3346 from zjffdu/ZEPPELIN-4066 and squashes the following commits:
b847d6d86 [Jeff Zhang] Add state COMPLETED & also add transition method for more detail logging
97049bd55 [Jeff Zhang] [ZEPPELIN-4066]. Introduce ProcessLauncher to encapsulate process launch
### What is this PR for?
This PR will break python code execution if the python process is existed. Besides that, I also improve the error message for ipython interpreter although it doesn't have such issue.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4081
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3338 from zjffdu/ZEPPELIN-4081 and squashes the following commits:
907faacf6 [Jeff Zhang] ZEPPELIN-4081. when the python process is killed,the task state is still running
(cherry picked from commit c9514e26a0)
Signed-off-by: Jeff Zhang <zjffdu@apache.org>
### What is this PR for?
The pr is the last fix of the `ZEPPELIN-4089` It will aims to notify user when ipython kernel crash. Indeed, right now, execution will be stuck forever when ipython kernel crash.
### What type of PR is it?
Bug Fix
### Todos
* [x] - logic to check kernel status
* [ ] - decide what to do when we spot the failure
* [x] - test
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4089
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3339 from AyWa/fix/kernel-crash and squashes the following commits:
6f4910c67 [marc hurabielle] fix lint
73424d17d [marc hurabielle] Revert "example for kernel die and request stuck"
f37d5f95d [marc hurabielle] add test for kernel die
c9ec0335c [marc hurabielle] example for kernel die and request stuck
5fe84dfef [marc hurabielle] handle kernel crash
(cherry picked from commit ef5e173d3a)
Signed-off-by: Jeff Zhang <zjffdu@apache.org>
### What is this PR for?
This PR will break python code execution if the python process is existed. Besides that, I also improve the error message for ipython interpreter although it doesn't have such issue.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4081
### How should this be tested?
* Unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3338 from zjffdu/ZEPPELIN-4081 and squashes the following commits:
907faacf6 [Jeff Zhang] ZEPPELIN-4081. when the python process is killed,the task state is still running
### What is this PR for?
The pr is the last fix of the `ZEPPELIN-4089` It will aims to notify user when ipython kernel crash. Indeed, right now, execution will be stuck forever when ipython kernel crash.
### What type of PR is it?
Bug Fix
### Todos
* [x] - logic to check kernel status
* [ ] - decide what to do when we spot the failure
* [x] - test
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4089
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: marc hurabielle <marc.hurabielle@gmail.com>
Closes#3339 from AyWa/fix/kernel-crash and squashes the following commits:
6f4910c67 [marc hurabielle] fix lint
73424d17d [marc hurabielle] Revert "example for kernel die and request stuck"
f37d5f95d [marc hurabielle] add test for kernel die
c9ec0335c [marc hurabielle] example for kernel die and request stuck
5fe84dfef [marc hurabielle] handle kernel crash
### What is this PR for?
This PR will exist wait ipython process loop if the python process is failed. This can save time to exit the waiting loop earlier, otherwise it would wait until timeout (30 seconds by default)
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4062
### How should this be tested?
* Manually tested.
Before this PR
```
INFO [2019-03-13 10:06:30,070] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,175] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,281] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,386] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,489] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,595] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,700] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:30,805] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
WARN [2019-03-13 10:06:30,810] ({Exec Default Executor} IPythonInterpreter.java[onProcessFailed]:419) - Exception happens in Python Process
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:06:30,910] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:06:31,015] ({FIFOScheduler-Worker-1}
```
After this PR
```
INFO [2019-03-13 10:28:12,805] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:12,907] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:13,011] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:13,116] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:13,220] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
INFO [2019-03-13 10:28:13,325] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
WARN [2019-03-13 10:28:13,347] ({Exec Default Executor} IPythonInterpreter.java[onProcessFailed]:423) - Exception happens in Python Process
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:28:13,430] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started
WARN [2019-03-13 10:28:13,431] ({FIFOScheduler-Worker-1} PythonInterpreter.java[open]:98) - Fail to open IPythonInterpreter
java.lang.RuntimeException: Fail to open IPythonInterpreter
at org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:152)
at org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:93)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:598)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:507)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:121)
at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Fail to launch IPython Kernel as the python process is failed
at org.apache.zeppelin.python.IPythonInterpreter.launchIPythonKernel(IPythonInterpreter.java:323)
at org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:149)
... 10 more
INFO [2019-03-13 10:28:13,431] ({FIFOScheduler-Worker-1} PythonInterpreter.java[open]:104) - IPython is not available, use the native PythonInterpreter
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3332 from zjffdu/ZEPPELIN-4062 and squashes the following commits:
130c2a82a [Jeff Zhang] ZEPPELIN-4062. Don't wait ipython kernel if python process failed
### What is this PR for?
Trivial PR to fix the IPythonClient not shutdown issue. Otherwise user will get the following error
```
Channel ManagedChannelImpl{logId=6, target=127.0.0.1:43464} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:410)
...
```
### What type of PR is it?
[Bug Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4042
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3324 from zjffdu/ZEPPELIN-4042 and squashes the following commits:
72e1ceb58 [Jeff Zhang] [ZEPPELIN-4042]. IPythonClient is not shutdown properly
### What is this PR for?
This PR add local property `limit` for SparkInterpreter and JdbcInterpreter, so that users can not control the max number of returned rows in paragraph level instead of interpreter level.
### What type of PR is it?
[ Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-3975
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3301 from zjffdu/ZEPPELIN-3975 and squashes the following commits:
6350433cc [Jeff Zhang] ZEPPELIN-3975. Add limit local property for SparkInterpreter and JdbcInterpreter
### What is this PR for?
delete temporary file and close inputstream in IPythonInterpreter.checkIPythonPrerequisite()
### What type of PR is it?
[Bug Fix]
### Todos
*no
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-3910, [ZEPPELIN-3910]
### How should this be tested?
* manual.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
no
* Is there breaking changes for older versions?
no
* Does this needs documentation?
no
Author: yx91490 <yx91490@126.com>
Closes#3287 from yx91490/ipy_tmp_file and squashes the following commits:
c811e3736 [yx91490] delete temp file and close stream
### What is this PR for?
This ticket create AbstractInterpreter which do some common things:
* Support ZeppelinContext
* Support z variable replacement
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-3976
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#3293 from zjffdu/ZEPPELIN-3976 and squashes the following commits:
7b42a42f3 [Jeff Zhang] ZEPPELIN-3976. Create AbstractInterprter for common usage
### What is this PR for?
By using the Raft protocol, multiple Zeppelin-Server groups are built into a Zeppelin cluster, the cluster State Machine is maintained through the Raft protocol, and the services in the cluster are agreed upon. The Zeppelin-Server and Zeppelin-Interperter services and processes are stored in the Cluster MetaData. Metadata information;
### What type of PR is it?
[Feature]
### Todos
* [x] add raft algorithm atomix jar
* [x] add cluster state machine
* [x] add state machine query command
* [x] add state machine delete command
* [x] add state machine put command
* [x] Isolate the netty JAR package introduced by atomix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3610
### How should this be tested?
[CI pass](https://travis-ci.org/liuxunorg/zeppelin/builds/418742522)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: liuxunorg <33611720@qq.com>
Author: Jeff Zhang <zjffdu@gmail.com>
Closes#3183 from liuxunorg/ZEPPELIN-3610 and squashes the following commits:
c899b151f [liuxunorg] uncomment maven-dependency-plugin.
af728bd2c [liuxunorg] add more doc to explain this class's responsibility.
d268ee72e [Jeff Zhang] fix test (#2)
97f17acac [liuxunorg] Cluster Raft module design
### What is this PR for?
Avoiding outdated checkstyle error.
To check format
`mvn validate`
To fix non-complying files
`mvn fmt:format`
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [x] - Adopt `fmt-maven-plugin`
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3740
### How should this be tested?
CI passs
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes for contributors
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#3158 from jongyoul/ZEPPELIN-3740 and squashes the following commits:
f06c50e90 [Jongyoul Lee] Adopt `fmt-maven-plugin` for google-java-format Adopt all files
