mirror of
https://github.com/apache/zeppelin
synced 2026-05-24 09:38:26 +00:00
64 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Malay Majithia
|
84cb4b5fb9 |
ZEPPELIN-277 Add Tab as Autocomplete for Notebook non-md paragraphs
### What is this PR for? This PR will add tab as auto complete invoker if paragraph is non-md and user has not pressed the tab as a first character in the line ### What type of PR is it? [Improvement] ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-277 * https://issues.apache.org/jira/browse/ZEPPELIN-2736 ### How should this be tested? - Build: mvn clean package -Denforcer.skip -DskipTests -Drat.skip - Open a paragraph - Press tab with following options: first character, after space ### Questions: * Does the licenses files need an update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Malay Majithia <malay.majithia@gmail.com> Author: Lee moon soo <moon@apache.org> Closes #2542 from malayhm/ZEPPELIN-277 and squashes the following commits: |
||
|
Jeff Zhang
|
152538b7ef |
ZEPPELIN-2990. Matplotlib sometimes fails in IPythonInterpreter
### What is this PR for? Trivial fix for matplotlib fail in `IPythonInterpreter`. Besides that, I also make some minor changes on zeppelin code to improve logging. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2990 ### How should this be tested? UT is added. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2622 from zjffdu/ZEPPELIN-2990 and squashes the following commits: |
||
|
Jeff Zhang
|
9812e26bce |
ZEPPELIN-2685. Improvement on Interpreter class
### What is this PR for? Follow up of #2577. Main changes on Interpreter * Add throw `InterpreterException` which is checked exception for the abstract methods of `Interpreter`, this would enforce the interpreter implementation to throw `InterpreterException`. * field name refactoring. * `property` -> `properties` * `getProperty()` --> `getProperties()` * Introduce launcher layer for interpreter launching. Currently we only use shell script to launch interpreter, but it could be any other service or component to launch interpreter, such as livy server , other 3rd party tools or even we may create a separate module for interpreter launcher * abstract cass `InterpreterLauncher` * For now, only 2 implementation: `ShellScriptLauncher` & `SparkInterpreterLauncher`. We could add method in class `Interpreter` to allow interpreter to specify its own launcher class, but it could be future work. ### What type of PR is it? [Improvement | Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2685 ### How should this be tested? Unit test is covered. `ShellScriptLauncherTest` & `SparkInterpreterLauncherTest` ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2592 from zjffdu/ZEPPELIN-2685 and squashes the following commits: |
||
|
Jeff Zhang
|
4a3057fa87 |
ZEPPELIN-2982. Copy interpreter-setting.json to interpreter dir
### What is this PR for? 2 main changes. * Introduce `interpreter-parent` module, so that all the interpreter can reuse the plugin defined in `interpreter-parent` * Add new plugin for `copying interpreter-setting.json` to interpreter dir ### What type of PR is it? [ Improvement | Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2982 ### How should this be tested? * First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration * Strongly recommended: add automated unit tests for any new or changed behavior * Outline any manual steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2613 from zjffdu/ZEPPELIN-2982 and squashes the following commits: |
||
|
Prabhjyot Singh
|
e998f5e067 |
[ZEPPELIN-2970] Enhance Authentication decrypting key generation
### What is this PR for? Overview : The random number generator implemented by nextInt() cannot withstand a cryptographic attack. In the file IdHashes.java similar issues were on line numbers 74 Comments : Use of SecureRandom is recommended ### What type of PR is it? [Refactoring] ### What is the Jira issue? * [https://issues.apache.org/jira/browse/ZEPPELIN-2970](https://issues.apache.org/jira/browse/ZEPPELIN-2970) ### How should this be tested? CI should be green Author: Prabhjyot Singh <prabhjyotsingh@gmail.com> Closes #2606 from prabhjyotsingh/ZEPPELIN-2970 and squashes the following commits: |
||
|
Jeff Zhang
|
7f16fc2177 |
ZEPPELIN-2943 Streaming output will be closed in 30 seconds for IPythonInterpreter
### What is this PR for? Straightforward fix to set the timeout as 100 years so that spark streaming app won't be stopped unless 100 years later. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2943 ### How should this be tested? Tested manually, see screenshot. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2593 from zjffdu/ZEPPELIN-2943 and squashes the following commits: |
||
|
Jeff Zhang
|
5d7151097e |
ZEPPELIN-2898. Support Yarn-Cluster for Spark Interpreter
### What is this PR for?
This is the first version for supporting yarn-cluster of `SparkInterpreter`. I just delegate all the function to `spark-submit` as yarn-cluster is natively supported by spark, we don't need to reinvent the wheel. But there's still improvement to be done in future, e.g. I put some spark specific logic in `InterpreterSetting` which is not a good practise. I plan to improve it when I refactor the `Interpreter` class (ZEPPELIN-2685).
Besides that, I also add `MiniHadoopCluster` & `MiniZeppelin` which help for the integration test of yarn-client & yarn-cluster mode, otherwise I have to manually verify yarn-client & yarn-cluster mode which would easily cause regression issue in future.
To be noticed:
* SPARK_HOME must be specified for yarn-cluster mode
* HADOOP_CONF_DIR must be specified for yarn-cluster mode
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
https://github.com/zjffdu/zeppelin/tree/ZEPPELIN-2898
### How should this be tested?
System test is added in `SparkInterpreterIT`.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes #2577 from zjffdu/ZEPPELIN-2898 and squashes the following commits:
|
||
|
CloverHearts
|
84ac43328b |
[ZEPPELIN-2921] does not work conda environment in python interpreter
### What is this PR for? Fixed a problem where conda's library was not installed and reflected properly due to changed python interpreter execution flow and structures. ORIGINAL : https://github.com/apache/zeppelin/pull/2574 ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2921 ### How should this be tested? Please run the following command line for each paragraph. If the module such as scipy is normally imported, it is a success. `%python.conda create --name Hello6 python=2.7` `%python.conda activate Hello6` `%python.conda install seaborn pandas numpy scipy matplotlib` ``` %python import scipy as sp import seaborn as sns ``` `%python.conda deactivate` ### Screenshots (if appropriate) #### Before  #### After  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: CloverHearts <cloverheartsdev@gmail.com> Closes #2578 from cloverhearts/ZEPPELIN-2921 and squashes the following commits: |
||
|
Jeff Zhang
|
32517c9d9f |
[ZEPPELIN-2753] Basic Implementation of IPython Interpreter
### What is this PR for? This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well. For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter. Main features: * IPython interpreter support ** All the ipython features are available, including visualization, ipython magics. * ZeppelinContext support * Streaming output support * Support Ipython in PySpark Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here. In unit test, I only verify the following 3 popular visualization library. could add more later. * matplotlib * bokeh * ggplot ### What type of PR is it? [Feature ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2753 ### How should this be tested? Unit test is added. ### Screenshots (if appropriate) Verify bokeh in IPython Interpreter  Verify matplotlib  Verify ZeppelinContext  Verify Streaming  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits: |
||
|
Shiem Edelbrock
|
f1c48e87ee |
[ZEPPELIN-2821] Fix Missing import
### What is this PR for?
The python interpreter has a bug when trying to render matplotlib images from the z.show() function. Line 139 of `python/src/main/resources/python/zeppelin_python.py` references un-imported package `base64`. `import base64` was added to the file to prevent this error in the future.
### What type of PR is it?
Bug Fix
### Todos
* [x] - Add missing `base64` module to `python/src/main/resources/python/zeppelin_python.py`
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2821
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
Display a matplotlib image with zeppelin:
```python
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
z.show(plt)
```
### Questions:
* Does the licenses files need update?
no
* Is there breaking changes for older versions?
no
* Does this needs documentation?
no
Author: Shiem Edelbrock <shiem.edelbrock@me.com>
Closes #2511 from Sdedelbrock/patch-1 and squashes the following commits:
|
||
|
andrea
|
f238ca0e3f |
[ZEPPELIN-2766] Make online resources url configurable at compile time
### What is this PR for? At compile time Zeppelin is downloading several external resources. I want to be able to provide alternative URLs to compile when internet is not available(i.e. behind corporate proxy). ### What type of PR is it? [Improvement] ### What is the Jira issue? [ZEPPELIN-2766] ### How should this be tested? ``` mvn -DskipTests -Dplugin.frontend.downloadRoot=<alernative-url> -Dplugin.frontend.nodeDownloadRoot=<alernative-url> -Dplugin.frontend.yarnDownloadRoot=<alernative-url> -Dplugin.frontend.npmDownloadRoot=<alernative-url> -Dpypi.repo.url=<alernative-url> clean package ``` ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO Author: andrea <andrea.peruffo1982@gmail.com> Closes #2483 from andreaTP/softCodeRepos and squashes the following commits: |
||
|
Tinkoff DWH
|
155a55b560 |
[ZEPPELIN-2403] interpreter property widgets
### What is this PR for? I spoiled the previous PR #2251 Added widgets (string, text, url, password, url, checkbox) to properties of interpreters. Those are widgets for properties customization. Properties must have the ability to customize the display (for example password). ### What type of PR is it? Feature ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2403 ### How should this be tested? - remove conf/interpreter.json - Try new form (create, edit) of interpreter settings ### Screenshots (if appropriate) edit  view  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Tinkoff DWH <tinkoff.dwh@gmail.com> Author: isys.mreshetov <m.reshetov@i-sys.ru> Closes #2268 from tinkoff-dwh/ZEPPELIN-2403 and squashes the following commits: |
||
|
Jeff Zhang
|
7b585c7399 |
ZEPPELIN-2395. Refactor Input.java to make dynamic forms extensible
### What is this PR for?
Currently, zeppelin only support 3 kinds of dynamic form controls: TextBox, Select, CheckBox. All the things are in `Input.java`, this is hard to add new controls, this PR is for refactoring Input to make dynamic forms extensible. Main Changes:
* Make `Input` as the base class of dynamic forms also use it as the factory class
* All the concret dynamic forms extend `Input`
* Add method `toJson` and `fromJson` for `GUI` for `GUI`'s serialization/deserialization. I plan to do it for other classes as well, so that we can remove duplicated serde code and also make it easy to test serialization/deserialization
* Change `z.input` to `z.textbox` as I think z.input is a little misleading. But I still keep `z.input` and make `z.input` as deprecated.
* Ideally the new input forms' json should be the same as the old input form json. But there's one bug in the old input form, `type` is missing if the input forms are created in frontend for textbox and select. So I keep the old input forms for compatibility. I will load the old input forms json and convert it into new input forms, and after saving, `note.json` would have the new input forms json.
After this PR, user needs to do 3 things to add new ui controls
* Implement its UI control classes, (refer TextBox/CheckBox/Select), and specify it in `TypeAdapterFactory` of `Input` for serde.
* Add parsing logic in `Input.getInputForm` if you want to support this control in frontend.
* Add display logic in `paragraph-parameterizedQueryForm.html`
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2395
### How should this be tested?
Test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes #2245 from zjffdu/ZEPPELIN-2395 and squashes the following commits:
|
||
|
Tinkoff DWH
|
4d398ef2a6 |
[ZEPPELIN-2297] improvements to jdbc autocompleter
### What is this PR for? PR contains some improvements for completion (JDBC Interpreter): - types of completion - display of long values - refactoring of search of completions - uniqness of completions with type `keyword` - updating data in completer by pressing `Ctrl + .` - setting the schema filter to generate completions - fix highlighting code when used not default data source ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2297 ### How should this be tested? try to work with new completer ### Screenshots (if appropriate) **1. Types of completion**  **2. Display of long values** before  after  **3. Refactoring of search of completions. Updating data in completer by pressing `Ctrl + .`** before  after  **4. uniqness of completions with type keyword** before  after  **5. fix highlighting code when used not default data source** before  after  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Tinkoff DWH <tinkoff.dwh@gmail.com> Closes #2203 from tinkoff-dwh/ZEPPELIN-2297 and squashes the following commits: |
||
|
CloverHearts
|
8d03920b9b |
[Zeppelin-802] Support for Zeppelin Context redefinition on Python and Pyspark
### What is this PR for? If you override the reserved word ZeppelinContext such as `z` in the python language, the whole paragraph output problem occurred. I have taken care to avoid this issue. `z` == `_zc` == `zeppelin context` ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-802 ### How should this be tested? The error should not occur in the following situations: ``` %python z = 1 print("Hello Zeppelin") ``` ``` %pyspark z = 1 print("Hello Zeppelin") ``` ### Screenshots (if appropriate) #### before  #### after  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: CloverHearts <cloverheartsdev@gmail.com> Author: cloverhearts <cloverheartsdev@gmail.com> Closes #2207 from cloverhearts/ZEPPELIN-802-pyspark-zeppelin-context and squashes the following commits: |
||
|
AhyoungRyu
|
c87fa53a3a |
[ZEPPELIN-2298] Remove -Ppyspark build profile
### What is this PR for? Currently users who build Zeppelin from source need to include `-Ppyspark` to use `%pyspark` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Ppyspark` and make pyspark related libraries automatically downloaded when we build Zeppelin. ### What type of PR is it? Improvement ### Todos * [x] - remove the rest of `-Ppyspark` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after getting feedback ### What is the Jira issue? [ZEPPELIN-2298](https://issues.apache.org/jira/browse/ZEPPELIN-2298) ### How should this be tested? 1. Apply this patch 2. Build source with below command ``` mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies' ``` Aftr this step, there will be `pyspark` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Ppyspark` build profile. 4. Restart Zeppelin. To make sure, run any python code e.g. ``` %pyspark print("Hello "+z.input("name")) ``` It should be run successfully without any error ### Screenshots (if appropriate) tl;dr Without `-Ppyspark` profile - Before <img width="856" alt="screen shot 2017-04-02 at 2 50 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584778/0e8ec6b0-17b4-11e7-9f0d-f2599fd7bd63.png"> - After <img width="893" alt="screen shot 2017-04-02 at 2 28 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584779/10b7ed68-17b4-11e7-90d4-aa95eb9bba2d.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no I want to include `SparkR` by default(= remove `-PsparkR` build profile) like this as a next step. I want to ask how Zeppelin community think about this. Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #2213 from AhyoungRyu/ZEPPELIN-2298/includePysparkByDefault and squashes the following commits: |
||
|
Lee moon soo
|
1972a58620 |
[HOTFIX] Dynamic form in python interpreter
### What is this PR for? https://github.com/apache/zeppelin/pull/2106 rewrote python interpreter. But dynamic form feature is not rewritten correctly. ### What type of PR is it? Hot Fix ### Todos * [x] - Bring dynamic form back ### What is the Jira issue? https://github.com/apache/zeppelin/pull/2106 ### How should this be tested? run ``` %python print("Hello "+z.input("name", "sun")) ``` ``` %python print("Hello "+z.select("day", [("1","mon"), ("2","tue"), ("3","wed"), ("4","thurs"), ("5","fri"), ("6","sat"), ("7","sun")])) ``` ``` %python options = [("apple","Apple"), ("banana","Banana"), ("orange","Orange")] print("Hello "+ " and ".join(z.checkbox("fruit", options, ["apple"]))) ``` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <moon@apache.org> Closes #2155 from Leemoonsoo/python_get_interpreter_context and squashes the following commits: |
||
|
astroshim
|
287ffd50e2 |
Rewrite PythonInterpreter.
### What is this PR for? I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter. 1. not working streaming output. - https://issues.apache.org/jira/browse/ZEPPELIN-2225 2. printed "..." when there is indent in the python code. - https://issues.apache.org/jira/browse/ZEPPELIN-1929 3. very slow output of matplotlib - https://issues.apache.org/jira/browse/ZEPPELIN-1894 - https://issues.apache.org/jira/browse/ZEPPELIN-1360 4. Unexpected output of matplotlib. - https://issues.apache.org/jira/browse/ZEPPELIN-2107 so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues. and I am going to recreate conda, docker for python interpreter ASAP. ### What type of PR is it? Bug Fix | Hot Fix | Refactoring ### How should this be tested? 1. not working streaming output. ``` import time for x in range(0, 5): print x time.sleep(1) ``` 2. printed "..." when there is indent in the python code. ``` def fn(): print("hi") fn() ``` 3. very slow output of matplotlib. ``` import matplotlib import sys import matplotlib.pyplot as plt plt.plot([1,2,3]) ``` 4. Unexpected output of matplotlib. ``` import matplotlib.pyplot as plt import matplotlib as mpl # Make a figure and axes with dimensions as desired. fig = plt.figure(figsize=(8, 3)) ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15]) ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15]) ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15]) # Set the colormap and norm to correspond to the data for which # the colorbar will be used. cmap = mpl.cm.cool norm = mpl.colors.Normalize(vmin=5, vmax=10) # ColorbarBase derives from ScalarMappable and puts a colorbar # in a specified axes, so it has everything needed for a # standalone colorbar. There are many more kwargs, but the # following gives a basic continuous colorbar with ticks # and labels. cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap, norm=norm, orientation='horizontal') cb1.set_label('Some Units') # The second example illustrates the use of a ListedColormap, a # BoundaryNorm, and extended ends to show the "over" and "under" # value colors. cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c']) cmap.set_over('0.25') cmap.set_under('0.75') # If a ListedColormap is used, the length of the bounds array must be # one greater than the length of the color list. The bounds must be # monotonically increasing. bounds = [1, 2, 4, 7, 8] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap, norm=norm, # to use 'extend', you must # specify two extra boundaries: boundaries=[0] + bounds + [13], extend='both', ticks=bounds, # optional spacing='proportional', orientation='horizontal') cb2.set_label('Discrete intervals, some other units') # The third example illustrates the use of custom length colorbar # extensions, used on a colorbar with discrete intervals. cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.], [1., .8, 0.], [1., .4, 0.]]) cmap.set_over((1., 0., 0.)) cmap.set_under((0., 0., 1.)) bounds = [-1., -.5, 0., .5, 1.] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap, norm=norm, boundaries=[-10] + bounds + [10], extend='both', # Make the length of each extension # the same as the length of the # interior colors: extendfrac='auto', ticks=bounds, spacing='uniform', orientation='horizontal') cb3.set_label('Custom extension lengths, some other units') plt.show() ``` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@zepl.com> Author: Lee moon soo <moon@apache.org> Author: HyungSung <hsshim@nflabs.com> Closes #2106 from astroshim/py4jPythonInterpreter and squashes the following commits: |
||
|
Mina Lee
|
61aeeaf10f |
Bump up version to 0.8.0-SNAPSHOT
### What is this PR for? Bump up version to 0.8.0-SNAPSHOT Author: Mina Lee <minalee@apache.org> Closes #1883 from minahlee/0.8.0-SNAPSHOT and squashes the following commits: |
||
|
1ambda
|
034fdc6735 |
[ZEPPELIN-1917] Improve python.conda interpreter
### What is this PR for? Add missing commands to the `python.conda` interpreter - `conda info` - `conda list` - `conda create` - `conda install` - `conda uninstall (alias of remove)` - `conda env *` #### Implementation Detail The reason I modified `PythonProcess` is due to NPE ```java // https://github.com/apache/zeppelin/blob/master/python/src/main/java/org/apache/zeppelin/python/PythonProcess.java#L107-L118 public String sendAndGetResult(String cmd) throws IOException { writer.println(cmd); writer.println(); writer.println("\"" + STATEMENT_END + "\""); StringBuilder output = new StringBuilder(); String line = null; // NPE when line is null while (!(line = reader.readLine()).contains(STATEMENT_END)) { logger.debug("Read line from python shell : " + line); output.append(line + "\n"); } return output.toString(); } ``` ``` java.lang.NullPointerException at org.apache.zeppelin.python.PythonProcess.sendAndGetResult(PythonProcess.java:113) at org.apache.zeppelin.python.PythonInterpreter.sendCommandToPython(PythonInterpreter.java:250) at org.apache.zeppelin.python.PythonInterpreter.bootStrapInterpreter(PythonInterpreter.java:272) at org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:100) at org.apache.zeppelin.python.PythonCondaInterpreter.restartPythonProcess(PythonCondaInterpreter.java:139) at org.apache.zeppelin.python.PythonCondaInterpreter.interpret(PythonCondaInterpreter.java:88) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:494) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` ### What type of PR is it? [Improvement | Refactoring] ### Todos * [x] - info * [x] - list * [x] - create * [x] - install * [x] - uninstall (= remove) * [x] - env * ### What is the Jira issue? [ZEPPELIN-1917](https://issues.apache.org/jira/browse/ZEPPELIN-1917) ### How should this be tested? 1. Install [miniconda](http://conda.pydata.org/miniconda.html) 2. Make sure that your python interpreter can use `conda` (check the Interpreter Binding page) 3. Remove `test` conda env since we will create in the following section ```sh $ conda env remove --yes --name test ``` 4. Run these commands with `%python.conda` ``` %python.conda info %python.conda env list %python.conda create --name test # you should be able to see `test` in the list %python.conda env list %python.conda activate pymysql %python.conda install pymysql # you should be able to import %python import pymysql.cursors %python.conda uninstall pymysql %python.conda deactivate pymysql # you should be able to see `No module named pymysql.cursor` since we deactivated %python import pymysql.cursors ``` ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO Author: 1ambda <1amb4a@gmail.com> Closes #1868 from 1ambda/ZEPPELIN-1917/improve-conda-interpreter and squashes the following commits: |
||
|
astroshim
|
8e6bfb45ea |
[ZEPPELIN-1930](HotFix) PythonInterpreter syntax error.
### What is this PR for? This PR fixes syntax error of PythonInterpreter. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1930 ### How should this be tested? Please run following code on the paragraph of PythonInterpreter. ``` for x in range(0,3): print ("hi") ``` ### Screenshots (if appropriate) - before  - after  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@zepl.com> Closes #1877 from astroshim/ZEPPELIN-1930 and squashes the following commits: |
||
|
1ambda
|
ee309066a4 |
[ZEPPELIN-1695] Use shared versions in test libraries (maven)
### What is this PR for? Use shared test library versions in maven config so that lib versions do mot be fragmented. Previously we used multiple versions of - Junit (4.11, 4.12) - mockito (1.9.0, 1.10.8, ...) - powermock (...) ### What type of PR is it? [Improvement] ### What is the Jira issue? [ZEPPELIN-1695](https://issues.apache.org/jira/browse/ZEPPELIN-1695) ### How should this be tested? Use this command to see test libraries share versions or not ``` $ mvn org.apache.maven.plugins:maven-help-plugin:2.2:effective-pom | vim - ``` ### Questions: * Does the licenses files need update? - YES, I updated JUnit version to 4.12 from 4.11 * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO Author: 1ambda <1amb4a@gmail.com> Closes #1727 from 1ambda/feat/centralise-testing-libraries and squashes the following commits: |
||
|
astroshim
|
8e6f333a69 |
Add PythonDockerInterpreter to interpreter group
### What is this PR for? Recently PythonDockerInterpreter added to PythonInterpreter in https://github.com/apache/zeppelin/pull/1654. ### What type of PR is it? Bug Fix | Improvement ### Screenshots (if appropriate) - before  - after  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Closes #1707 from astroshim/add/python-docker-interpreter and squashes the following commits: |
||
|
astroshim
|
01c45d7ae6 |
[ZEPPELIN-1724] conda run command removed in 4.1.0
### What is this PR for? Because `conda run` command removed since version `4.0.9`, PythonCondaInterpreter not working after the `conda-4.0.9`. This PR fixes this issue. I tested conda-4.2.12 and conda-4.0.9 . ### What type of PR is it? Bug Fix | Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1724 ### How should this be tested? Please refer to https://github.com/apache/zeppelin/pull/1645 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Closes #1699 from astroshim/ZEPPELIN-1724 and squashes the following commits: |
||
|
Lee moon soo
|
850fd81a51 |
[ZEPPELIN-212] Multiple paragraph results
### What is this PR for?
Currently a paragraph can display only a single type of result.
For example
```
print("""
%text textout
%html htmlout
""")
```
will display only last display system detected, "htmlout".
This pr implements multiple results supports, so not only the last, but also all the display systems in a paragraph outputs can be displayed.
To do that, Note.json format need to be changed. from
```
paragraph : [
{
result : {
code: "",
type: "",
msg: ""
},
config : {
graph : {},
...
},
...
},
...
],
...
```
to
```
paragraph : [
{
results : {
{
code: "",
msg: [
{
type: "",
data: ""
},
{
type: "",
data: ""
},
...
]
},
config : {
results : [
{
graph : {},
},
{
graph : {},
},
...
]
},
...
},
...
],
...
```
### What type of PR is it?
Improvement
### Todos
* [x] - Make InterpreterResult and InterpreterOutput support multiple display system
* [x] - Automatic migration old format to new format
* [x] - Render multiple results in front-end
* [x] - Take care Importing old notebook format
* [x] - Make helium framework support multiple results
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-212
### How should this be tested?
run following code in a single spark paragraph.
```
%spark
// default output is text
(1 to 3).foreach{i=>
println(new java.util.Date())
Thread.sleep(1000)
}
// print something in html
println(s"""%html <h3><font style="color:blue">Some HTML</font></h3>""")
// create table
println(s"""%table key\tvalue
sun\t100
moon\t200
""")
// display text again
println("""%text""")
(1 to 3).foreach{i=>
println(new java.util.Date())
Thread.sleep(1000)
}
// another table
Thread.sleep(1000)
println(s"""%table key\tvalue
apple\t100
banana\t200
""")
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? yes
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #1658 from Leemoonsoo/ZEPPELIN-212 and squashes the following commits:
|
||
|
Alex Goodman
|
4ac577f711 |
[ZEPPELIN-1639] Add tests with external python dependencies to CI build
### What is this PR for? Take 2 of #1618 because I had some earlier problems with rebasing. Since, then I have added some new features, namely: - Matplotlib integration tests for pyspark - `install_external_dependencies.sh` which conditionally installs the R or python dependencies based on the specified build profile in `.travis.yml`. This saves several minutes of time for a few of the build profiles since the R dependencies are compiled from source and therefore take quite a bit of time to install. - The extra python unit tests which require external dependencies (`matplotlib` and `pandas`) are now relegated to two separate build profiles. This is done primarily to efficiently test both Python 2 and 3. - Some minor bugs in the python and pyspark interpreters (mostly with respect to python 3 compatibility) were caught as a result of these tests, and are also fixed in this PR. ### What type of PR is it? Improvement and Bugfix ### What is the Jira issue? [ZEPPELIN-1639](https://issues.apache.org/jira/browse/ZEPPELIN-1639) ### How should this be tested? CI tests should be green! ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Alex Goodman <agoodm@users.noreply.github.com> Closes #1632 from agoodm/ZEPPELIN-1639 and squashes the following commits: |
||
|
1ambda
|
9780744494 |
[MINOR] Invalid constructor call in a test causes all CI failure
### What is this PR for?
A small minor fix for CI failure
- PR A added `replName` to a constructor of `InterpreterContext` as an argument.
- and PR B was merged without rebase
This results in compilation failure.
see also
- (current master) https://travis-ci.org/apache/zeppelin/builds/178654143
- (raw log) https://api.travis-ci.org/jobs/178654145/log.txt?deansi=true
```
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/build/apache/zeppelin/python/src/test/java/org/apache/zeppelin/python/PythonDockerInterpreterTest.java:[73,12] no suitable constructor found for InterpreterContext(java.lang.String,java.lang.String,java.lang.String,java.lang.String,org.apache.zeppelin.user.AuthenticationInfo,java.util.HashMap<java.lang.String,java.lang.Object>,org.apache.zeppelin.display.GUI,<nulltype>,<nulltype>,<nulltype>,org.apache.zeppelin.interpreter.InterpreterOutput)
constructor org.apache.zeppelin.interpreter.InterpreterContext.InterpreterContext(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,org.apache.zeppelin.user.AuthenticationInfo,java.util.Map<java.lang.String,java.lang.Object>,org.apache.zeppelin.display.GUI,org.apache.zeppelin.display.AngularObjectRegistry,org.apache.zeppelin.resource.ResourcePool,java.util.List<org.apache.zeppelin.interpreter.InterpreterContextRunner>,org.apache.zeppelin.interpreter.InterpreterOutput,org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient) is not applicable
(actual and formal argument lists differ in length)
constructor org.apache.zeppelin.interpreter.InterpreterContext.InterpreterContext(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,org.apache.zeppelin.user.AuthenticationInfo,java.util.Map<java.lang.String,java.lang.Object>,org.apache.zeppelin.display.GUI,org.apache.zeppelin.display.AngularObjectRegistry,org.apache.zeppelin.resource.ResourcePool,java.util.List<org.apache.zeppelin.interpreter.InterpreterContextRunner>,org.apache.zeppelin.interpreter.InterpreterOutput) is not applicable
(actual and formal argument lists differ in length)
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Zeppelin ........................................... SUCCESS [ 4.191 s]
[INFO] Zeppelin: Interpreter .............................. SUCCESS [ 10.154 s]
[INFO] Zeppelin: Zengine .................................. SUCCESS [ 4.848 s]
[INFO] Zeppelin: Display system apis ...................... SUCCESS [ 16.617 s]
[INFO] Zeppelin: Spark dependencies ....................... SUCCESS [ 36.233 s]
[INFO] Zeppelin: Spark .................................... SUCCESS [ 22.821 s]
[INFO] Zeppelin: Markdown interpreter ..................... SUCCESS [ 0.633 s]
[INFO] Zeppelin: Angular interpreter ...................... SUCCESS [ 0.371 s]
[INFO] Zeppelin: Shell interpreter ........................ SUCCESS [ 0.523 s]
[INFO] Zeppelin: Livy interpreter ......................... SUCCESS [ 6.972 s]
[INFO] Zeppelin: HBase interpreter ........................ SUCCESS [ 3.623 s]
[INFO] Zeppelin: Apache Pig Interpreter ................... SUCCESS [ 3.531 s]
[INFO] Zeppelin: PostgreSQL interpreter ................... SUCCESS [ 0.591 s]
[INFO] Zeppelin: JDBC interpreter ......................... SUCCESS [ 1.166 s]
[INFO] Zeppelin: File System Interpreters ................. SUCCESS [ 1.054 s]
[INFO] Zeppelin: Flink .................................... SUCCESS [ 6.863 s]
[INFO] Zeppelin: Apache Ignite interpreter ................ SUCCESS [ 0.987 s]
[INFO] Zeppelin: Kylin interpreter ........................ SUCCESS [ 0.421 s]
[INFO] Zeppelin: Python interpreter ....................... FAILURE [ 0.481 s]
[INFO] Zeppelin: Lens interpreter ......................... SKIPPED
[INFO] Zeppelin: Apache Cassandra interpreter ............. SKIPPED
[INFO] Zeppelin: Elasticsearch interpreter ................ SKIPPED
[INFO] Zeppelin: BigQuery interpreter ..................... SKIPPED
[INFO] Zeppelin: Alluxio interpreter ...................... SKIPPED
[INFO] Zeppelin: Scio ..................................... SKIPPED
[INFO] Zeppelin: web Application .......................... SKIPPED
[INFO] Zeppelin: Server ................................... SKIPPED
[INFO] Zeppelin: Packaging distribution ................... SKIPPED
[INFO] Zeppelin: Scalding interpreter ..................... SKIPPED
[INFO] Zeppelin: Examples ................................. SKIPPED
[INFO] Zeppelin: Example application - Clock .............. SKIPPED
[INFO] Zeppelin: Example application - Horizontal Bar chart SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:02 min
[INFO] Finished at: 2016-11-24T17:21:08+00:00
[INFO] Final Memory: 181M/1246M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project zeppelin-python: Compilation failure
[ERROR] /home/travis/build/apache/zeppelin/python/src/test/java/org/apache/zeppelin/python/PythonDockerInterpreterTest.java:[73,12] no suitable constructor found for InterpreterContext(java.lang.String,java.lang.String,java.lang.String,java.lang.String,org.apache.zeppelin.user.AuthenticationInfo,java.util.HashMap<java.lang.String,java.lang.Object>,org.apache.zeppelin.display.GUI,<nulltype>,<nulltype>,<nulltype>,org.apache.zeppelin.interpreter.InterpreterOutput)
[ERROR] constructor org.apache.zeppelin.interpreter.InterpreterContext.InterpreterContext(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,org.apache.zeppelin.user.AuthenticationInfo,java.util.Map<java.lang.String,java.lang.Object>,org.apache.zeppelin.display.GUI,org.apache.zeppelin.display.AngularObjectRegistry,org.apache.zeppelin.resource.ResourcePool,java.util.List<org.apache.zeppelin.interpreter.InterpreterContextRunner>,org.apache.zeppelin.interpreter.InterpreterOutput,org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] constructor org.apache.zeppelin.interpreter.InterpreterContext.InterpreterContext(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,org.apache.zeppelin.user.AuthenticationInfo,java.util.Map<java.lang.String,java.lang.Object>,org.apache.zeppelin.display.GUI,org.apache.zeppelin.display.AngularObjectRegistry,org.apache.zeppelin.resource.ResourcePool,java.util.List<org.apache.zeppelin.interpreter.InterpreterContextRunner>,org.apache.zeppelin.interpreter.InterpreterOutput) is not applicable
[ERROR] (actual and formal argument lists differ in length)
```
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
NO JIRA issue. (small, emergently)
### How should this be tested?
CI will do that :)
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes #1680 from 1ambda/minor/fix-invalid-constructor-call-in-test and squashes the following commits:
|
||
|
astroshim
|
b7307d49de |
[ZEPPELIN-1567] Let JDBC interpreter use user credential information.
### What is this PR for? This PR is for the multi-tenant of JDBC Interpreter. User can create a user/password for JDBC account at the [Credential page](http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/security/datasource_authorization.html). The `Entity` of `Credential` is match with JDBC interpreter group name. If the account for JDBC is not setted in the `Interpreter property` then use `Credential`'s. ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1567 ### How should this be tested? Please refer to testMultiTenant() of JDBCInterpreterTest/ ### Screenshots (if appropriate) ### Questions: - Does the licenses files need update? no - Is there breaking changes for older versions? no - Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Closes #1539 from astroshim/jdbc-impersonation and squashes the following commits: |
||
|
Lee moon soo
|
caa664d6ee |
[ZEPPELIN-1683] Run python process in docker container
### What is this PR for? Inspired by ZEPPELIN-1671 conda interpreter. Docker can provides kind of virtual environment for python like conda does. This PR implements %python.docker interpreter that helps run python process in docker container. This PR implements feature on top of https://github.com/apache/zeppelin/pull/1645 ### What type of PR is it? Feature ### Todos * [x] - basic feature * [x] - unittest * [x] - documentation ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1683 ### How should this be tested? see screenshot ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Lee moon soo <moon@apache.org> Closes #1654 from Leemoonsoo/pydocker and squashes the following commits: |
||
|
Lee moon soo
|
3665901504 |
[ZEPPELIN-1671] Conda interpreter
### What is this PR for? Conda interpreter that manages conda environment for PythonInterpreter ### What type of PR is it? Feature ### Todos * [x] - Basic impl * [x] - update doc ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1671 ### How should this be tested? Recreate(or create new) your python interpreter setting in gui. List all conda env ``` %python.conda ``` Activate env ``` %python.conda activate [name] ``` Deactivate env ``` %python.conda deactivate ``` ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Lee moon soo <moon@apache.org> Closes #1645 from Leemoonsoo/conda and squashes the following commits: |
||
|
Alex Goodman
|
f7af21e219 |
[HOTFIX][ZEPPELIN-1656] z.show in Python interpreter does not work
### What is this PR for? There have been reports of #1534 causing the python interpreter to always show an error because `z` is not being set. As it turns out this is a result of improperly handling the case when matplotlib isn't found when initializing the interpreter. ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-1656](https://issues.apache.org/jira/browse/ZEPPELIN-1656) ### How should this be tested? Run any simple python paragraph. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Alex Goodman <agoodm@users.noreply.github.com> Closes #1628 from agoodm/patch-1 and squashes the following commits: |
||
|
Alex Goodman
|
aded8681bb |
[ZEPPELIN-1655] Dynamic forms in Python interpreter do not work
### What is this PR for?
After #1534 , Dynamic Forms were no longer working in the python interpreter. This is because the `Py4jZeppelinContext` constructor did not initialize the `_displayhook` which is always called on post-execute.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
[ZEPPELIN-1655](https://issues.apache.org/jira/browse/ZEPPELIN-1655)
### How should this be tested?
Run the following `%python` paragraph, being sure that Py4j is installed:
```python
%python
a, b, c = (1, 2, 3)
z.select("Choose a letter", ([a,"a"], [b,"b"], [c,"c"] ))
```
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Alex Goodman <agoodm@users.noreply.github.com>
Closes #1626 from agoodm/ZEPPELIN-1655 and squashes the following commits:
|
||
|
Paul Bustios
|
dbd81bf856 |
[ZEPPELIN-1358] Add support to display Pandas DataFrame index using z.show()
### What is this PR for? Add support to display optionally Pandas DataFrame index using z.show(show_index=True) in python interpreter. By default, DataFrame index will not be displayed. ### What type of PR is it? Improvement ### What is the Jira issue? [ZEPPELIN-1358](https://issues.apache.org/jira/browse/ZEPPELIN-1358) ### How should this be tested? ``` mvn -Dpython.test.exclude='' test -pl python -am ``` ### Screenshots (if appropriate)  ### Questions: - Does the licenses files need update? no - Is there breaking changes for older versions? no - Does this needs documentation? no Author: Paul Bustios <pbustios@gmail.com> Closes #1378 from bustios/ZEPPELIN-1358 and squashes the following commits: |
||
|
Alex Goodman
|
438dbca686 |
ZEPPELIN-1345 - Create a custom matplotlib backend that natively supports inline plotting in a python interpreter cell
### What is this PR for? This PR is the first of two major steps needed to improve matplotlib integration in Zeppelin (ZEPPELIN-1344). The latter, which is a plotting backend with fully interactive tools enabled, will be done afterwards in a separate PR. This PR specifically for automatically displaying output from calls to matplotlib plotting functions inline with each paragraph. Thanks to the addition of post-execute hooks (ZEPPELIN-1423), there is no need to call any `show()` function to display an inline plot, just like in Jupyter. ### What type of PR is it? Improvement ### Todos The main code has been written and anyone who reads this is encouraged to test it, but there are a few minor todos: - [x] - Add unit tests - [x] - Add documentation - [x] - Add screenshot showing iterative plotting with angular mode ### What is the Jira issue? [ZEPPELIN-1345](https://issues.apache.org/jira/browse/ZEPPELIN-1345) ### How should this be tested? In a pyspark or python paragraph, enter and run ``` python import matplotlib.pyplot as plt plt.plot([1, 2, 3]) ``` The plot should be displayed automatically without calling any `show()` function whatsoever. A special method called `configure_mpl()` can also be used to modify the inline plotting behavior. For example, ``` python z.configure_mpl(close=False, angular=True) plt.plot([1, 2, 3]) ``` allows for iterative updates to the plot provided you have PY4J installed for your python installation (which of course is always the case if you use pypsark). To clarify, this feature only currently works with pyspark (not python as there are no `angularBind()` and `angularUnbind()` methods yet). Doing something like: ``` plt.plot([3, 2, 1]) ``` will update the plot that was generated by the previous paragraph by leveraging Zeppelin's Angular Display System. However, by setting `close=False`, matplotlib will no longer automatically close figures so it is now up to the user to explicitly close each figure instance they create. There's quite a bit more options for `z.configure_mpl()`, but I will save that discussion for the documentation. ### Screenshots (if appropriate)  ### Questions: - Does the licenses files need update? No - Is there breaking changes for older versions? No - Does this needs documentation? Yes Author: Alex Goodman <agoodm@users.noreply.github.com> Closes #1534 from agoodm/ZEPPELIN-1345 and squashes the following commits: |
||
|
Mina Lee
|
60089f0f58 |
[ZEPPELIN-1566] Make paragraph editable with double click
### What is this PR for? This PR enables edit on double click for markdown/angular paragraph. Users can change `editOnDblClick` field to be `false` by editting `interpreter/md/interpreter-setting.json` or `conf/interpreter.json`. In the same context, users can set other type paragraphs to be editable on double click by setting `editOnDblClick` to be true. This PR also fixes bug that syntax highlight doesn't work on pasted code. ### What type of PR is it? Feature ### Todos * [x] Create test * [x] Update docs ### What is the Jira issue? [ZEPPELIN-1566](https://issues.apache.org/jira/browse/ZEPPELIN-1566) ### How should this be tested? 1. Create new markdown interpreter 2. Create notebook and run markdown paragraph 3. Double click markdown paragraph to edit ### Screenshots (if appropriate) **Edit on double click and hide editor on paragraph run**  **Syntax highlight on paste** Before  After  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Mina Lee <minalee@apache.org> Closes #1540 from minahlee/ZEPPELIN-1566 and squashes the following commits: |
||
|
Kai Jiang
|
1b2635cfe6 |
[Zeppelin-1555] Eliminate prefix in PythonInterpreter exception
### What is this PR for? Solve bug metioned [here]( |
||
|
Mina Lee
|
a3ca800311 |
[ZEPPELIN-1026] set syntax highlight based on default bound interpreter
### What is this PR for? This is complete work of #1148. Comments and tasks on #1148 has been handled in this PR. - Add syntax language information in `interpreter-setting.json` - When user type `%replName` in paragraph, back-end check if the interpreter name with `replName` exists, and return language information to front-end if it does - If user doesn't specify `%replName`, default interpreter's language will be used - Using alias name for paragraph syntax highlight ### What type of PR is it? [Bug Fix | Improvement] ### What is the Jira issue? [ZEPPELIN-1026](https://issues.apache.org/jira/browse/ZEPPELIN-1026) ### How should this be tested? 1. Create new note and make markdown interpreter to be default. 2. See if markdown syntax is applied. ### Screenshots (if appropriate) #### Case 1. When the default interpreter set to python interpreter. **Before** Has `scala` as syntax highlight language when %python is not set. <img width="665" alt="screen shot 2016-07-07 at 10 46 20 pm" src="https://cloud.githubusercontent.com/assets/8503346/16655312/af67a302-4494-11e6-949e-793ad0515d7a.png"> **After** Has `python` as syntax highlight language even when %python is not set. <img width="666" alt="screen shot 2016-07-07 at 10 44 39 pm" src="https://cloud.githubusercontent.com/assets/8503346/16655248/769d8ba4-4494-11e6-9b3c-dc5e026e9c53.png"> #### Case 2. When use alias name as repl name. **Before** <img width="742" alt="screen shot 2016-09-08 at 4 22 39 pm" src="https://cloud.githubusercontent.com/assets/8503346/18353471/620c5ede-75e2-11e6-9d01-0726bc900dc0.png"> **After** <img width="741" alt="screen shot 2016-09-08 at 4 34 57 pm" src="https://cloud.githubusercontent.com/assets/8503346/18353487/6cdaa406-75e2-11e6-831a-08e0fa3a85d8.png"> ### Further possible improvements There are still several cases that Zeppelin doesn't handle syntax highlight well. These can be handled with another jira ticket/PR. 1. When default bound interpreter changes, syntax highlight is not changed accordingly 2. When copy/paste code, syntax highlight won't be applied properly since Zeppelin only checks changes when cursor is in first line. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes(for creating new interpreter) Author: Mina Lee <minalee@apache.org> Closes #1415 from minahlee/ZEPPELIN-1026 and squashes the following commits: |
||
|
CloverHearts
|
66d5811365 |
[ZEPPELIN-1412] add support multiline for pythonErrorIn method on python interpreter
### What is this PR for?
currently, has not support multiline exception text on python interpreter.
for example:
```
Exception: blabla
```
is error.
but
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception: test exception
```
is sucess (now)
to resolve this issue.
### What type of PR is it?
Bug Fix
### Todos
- [x] modification pythonErrorIn method
- [x] add test case
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1412?jql=project%20%3D%20ZEPPELIN%20AND%20status%20%3D%20Open
### How should this be tested?
added test case.
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: CloverHearts <cloverheartsdev@gmail.com>
Closes #1407 from cloverhearts/dev/ZEPPELIN-1412 and squashes the following commits:
|
||
|
Paul Bustios
|
8064c54917 |
[ZEPPELIN-1327] Fix bug in z.show for Python interpreter
### What is this PR for? Currently, height parameter for z.show implementation to display PNG images in Python interpreter is not working. This PR fix that bug. ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-1327](https://issues.apache.org/jira/browse/ZEPPELIN-1327) ### How should this be tested? ```python import matplotlib.pyplot as plt x = [1,2,3,4,5] y = [6,7,8,9,0] plt.plot(x, y, marker="o") z.show(plt, height="200px") plt.close() ``` ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Paul Bustios <pbustios@gmail.com> Closes #1352 from bustios/ZEPPELIN-1327 and squashes the following commits: |
||
|
Alex Goodman
|
582981677c |
ZEPPELIN-1328 - z.show in python interpreter does not display PNG images in python 3
### What is this PR for? Support for plotting PNG images via matplotlib inline for the python interpreter was recently added (#1329). However, these changes did not work for python3 since it handles strings differently. This PR aims to make the inline plotting compatible with both python 2 and 3. ### What type of PR is it? Bug Fix ### What is the Jira issue? * [ZEPPELIN-1328](https://issues.apache.org/jira/browse/ZEPPELIN-1328) ### How should this be tested? In a python interpreteter cell, make sure the following produce an image: ```python %python import matplotlib.pyplot as plt import numpy as np x = np.arange(5) plt.plot(x) z.show(plt, fmt='png') # Repeat for fmt='svg' ``` This should be tested for both python2 and 3 interpreters (via the interpreter settings page). ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Alex Goodman <agoodm@users.noreply.github.com> Closes #1343 from agoodm/ZEPPELIN-1328 and squashes the following commits: |
||
|
Alex Goodman
|
8b40268d16 |
ZEPPELIN-1318 - Add support for matplotlib displaying png images in python interpreter
### What is this PR for? This PR adds support for plotting png images using the matplotlib helper function within a python interpreter (eg `z.show()`). The primary motivation for this is due to the overhead incurred from svg images, which can lag the notebooks if multiple, complicated images are generated (for example, multiple filled contour plots). png images are more lightweight, but of course come at a cost of image quality due to them being raster rather than vector like svg. The support for png images is incorporated through the use of a new optional argument to `z.show` called `fmt` which can be one of `'svg'` or `'png'`. The same code that is currently used in show is used for svg images while the code for png images relies on converting the image directly to a byte array and then entering the decoded byte string directly into an HTML image tag. Currently `fmt` defaults to `'png'` but I think we should consider discussing the pros and cons of each option in this PR. ### What type of PR is it? Improvement ### What is the Jira issue? [ZEPPELIN-1318](https://issues.apache.org/jira/browse/ZEPPELIN-1318) ### How should this be tested? In a notebook cell, enter: ```python %python import matplotlib.pyplot as plt import numpy as np plt.figure() plt.plot(np.arange(10)) z.show(plt, fmt=fmt) ``` Where `fmt` may be one of `'svg'` or `'png'`, and any other input should result in a `ValueError`. I would also recommend testing the example in the screenshot below. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes (if the changes to the `help()` docstring are not sufficient) Author: Alex Goodman <agoodm@users.noreply.github.com> Closes #1329 from agoodm/ZEPPELIN-1318 and squashes the following commits: |
||
|
Jeff Zhang
|
3dec4d7006 |
ZEPPELIN-1287. No need to call print to display output in PythonInterpreter
### What is this PR for? It is not necessary to call print to display output in PythonInterpreter. 2 main changes: * the root cause is the displayhook in bootstrap.py * also did some code refactoring on PythonInterpreter ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-1287 ### How should this be tested? Verify it manually ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #1278 from zjffdu/ZEPPELIN-1287 and squashes the following commits: |
||
|
Alexander Bezzubov
|
a922fd28c1 |
Small refactoring of Python interpreter
### What is this PR for? Small refactoring of Python interpreter, that is what it is. ### What type of PR is it? Refactoring ### Todos * [x] refactor `help()` * [x] impl `maxResult` fetch from JVM ### How should this be tested? `cd python && mvn -Dpython.test.exclude='' test ` pass (given that `pip install pandasql` and `pip install py4j`) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Alexander Bezzubov <bzz@apache.org> Closes #1275 from bzz/python/refactoring and squashes the following commits: |
||
|
Paul Bustios
|
9eac20d08a |
[ZEPPELIN-1261] Bug fix in z.show() for matplotlib graphs
### What is this PR for? Bug fix in z.show() for matplotlib graphs and code refactoring ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-1261](https://issues.apache.org/jira/browse/ZEPPELIN-1261) ### How should this be tested? ``` %python import matplotlib.pyplot as plt x = [1,2,3,4,5] y = [6,7,8,9,0] plt.plot(x, y, marker="o") z.show(plt, height="20em") plt.close() ``` ``` %python import matplotlib.pyplot as plt x = [1,2,3,4,5] y = [6,7,8,9,0] plt.plot(x, y, marker="o") z.show(plt, height="300px") plt.close() ``` ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Paul Bustios <pbustios@gmail.com> Closes #1267 from bustios/ZEPPELIN-1261 and squashes the following commits: |
||
|
paulbustios
|
6f867ceb0c |
[ZEPPELIN-1255] Add cast to string in z.show() for Pandas DataFrame
### What is this PR for? Casting data types in Pandas DataFrame to string in z.show() ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-1255](https://issues.apache.org/jira/browse/ZEPPELIN-1255) ### How should this be tested? ``` %python import pandas as pd df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data', header=None) df.columns=[1, 2, 3, 'PetalWidth', 'Name'] z.show(df) %python.sql SELECT * FROM df LIMIT 10 ``` ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: paulbustios <pbustios@gmail.com> Closes #1249 from bustios/ZEPPELIN-1255 and squashes the following commits: |
||
|
Mina Lee
|
04da56403b |
[MINOR] Change url in pom.xml files
### What is this PR for? Set project url to `http://zeppelin.apache.org` in pom.xml files ### What type of PR is it? Refactoring ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <minalee@apache.org> Closes #1221 from minahlee/pom_url and squashes the following commits: |
||
|
Alexander Bezzubov
|
5afd7cacdb |
ZEPPELIN-1115: add correct %python.sql interpreter name to configuration
### What is this PR for?
Hotfix for [ZEPPELIN-1244](https://issues.apache.org/jira/browse/ZEPPELIN-1244)
### What type of PR is it?
Hot Fix
### What is the Jira issue?
[ZEPPELIN-1244](https://issues.apache.org/jira/browse/ZEPPELIN-1244)
### How should this be tested?
Remove interpreter configuration and try `%python.sql` i.e over [bank.csv](http://mlr.cs.umass.edu/ml/datasets/Bank+Marketing) read as `pandas.dataframe`
```
#stop zeppelin
rm conf/interpreter-setting.json
#start zeppelin
%python
import pandas as pd
rates = pd.read_csv("http://www3.dsi.uminho.pt/pcortez/data/bank.csv", sep=";")
%python.sql
SELECT * FROM rates LIMIT 10
```
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Alexander Bezzubov <bzz@apache.org>
Closes #1238 from bzz/python/fix/ZEPPELIN-1244 and squashes the following commits:
|
||
|
Sangmin Yoon
|
80d910049b |
[ZEPPELIN-1206] fix "name 'z' is not defined" with python3
### What is this PR for?
PythonInterpreter can not use dynamic form with python3.
Fix this problem.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1206
### How should this be tested?
```
%python
print(z.input("test"))
```
Make a note above, and run it.
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? NO
Author: Sangmin Yoon <sangmin.yoon@croquis.com>
Closes #1213 from sixmen/fix_python3 and squashes the following commits:
|
||
|
Alexander Bezzubov
|
d8b54cf76d |
ZEPPELIN-1115: Python - interpreter for SQL over DataFrame
### What is this PR for? Add new interpreter to Python group: `%python.sql` for SQL over DataFrame support ### What type of PR is it? Improvement ### TODOs * [x] add new interpreter `%python.sql` * [x] add test * [x] make Python-dependant tests, excluded from CI * PythonInterpreterWithPythonInstalledTest * PythonPandasSqlInterpreterTest * run manually by `mvn -Dpython.test.exclude='' test -pl python -am` * [x] add docs `%python.sql` * [x] make `%python.sql` fail gracefully in case there is no Pandas or PandaSQL installed * [x] after #747 is merged - rebase and remove `-Dpython.test.exclude=''` from both profiles ### What is the Jira issue? [ZEPPELIN-1115](https://issues.apache.org/jira/browse/ZEPPELIN-1115) ### How should this be tested? `mvn -Dpython.test.exclude='' test -pl python -am` should pass or manually run - Given the DataFrame i.e ``` %python import pandas as pd rates = pd.read_csv("bank.csv", sep=";") ``` - SQL query it like ``` %python.sql SELECT * FROM rates LIMIT 10 ``` ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No, no dependencies were included in source or binary release * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Alexander Bezzubov <bzz@apache.org> Closes #1164 from bzz/ZEPPELIN-1115/python/add-sql-for-dataframes and squashes the following commits: |
||
|
Alexander Bezzubov
|
399c49b4e1 |
ZEPPELIN-1063: fix PythonInterpreter flaky test
### What is this PR for? fix flaky `PythonInterpreter.testClose()` test ### What type of PR is it? Hot Fix ### What is the Jira issue? [ZEPPELIN-1063](https://issues.apache.org/jira/browse/ZEPPELIN-1063) ### How should this be tested? CI should pass ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Alexander Bezzubov <bzz@apache.org> Closes #1134 from bzz/ZEPPELIN-1063/python/add-retrys and squashes the following commits: |
