zeppelin/python/pom.xml

228 lines
6.9 KiB
XML
Raw Normal View History

[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>interpreter-parent</artifactId>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
<groupId>org.apache.zeppelin</groupId>
<version>0.9.0-SNAPSHOT</version>
<relativePath>../interpreter-parent</relativePath>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
</parent>
<groupId>org.apache.zeppelin</groupId>
<artifactId>zeppelin-python</artifactId>
<packaging>jar</packaging>
<version>0.9.0-SNAPSHOT</version>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
<name>Zeppelin: Python interpreter</name>
<properties>
<interpreter.name>python</interpreter.name>
[ZEPPELIN-2298] Remove -Ppyspark build profile ### What is this PR for? Currently users who build Zeppelin from source need to include `-Ppyspark` to use `%pyspark` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Ppyspark` and make pyspark related libraries automatically downloaded when we build Zeppelin. ### What type of PR is it? Improvement ### Todos * [x] - remove the rest of `-Ppyspark` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after getting feedback ### What is the Jira issue? [ZEPPELIN-2298](https://issues.apache.org/jira/browse/ZEPPELIN-2298) ### How should this be tested? 1. Apply this patch 2. Build source with below command ``` mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies' ``` Aftr this step, there will be `pyspark` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Ppyspark` build profile. 4. Restart Zeppelin. To make sure, run any python code e.g. ``` %pyspark print("Hello "+z.input("name")) ``` It should be run successfully without any error ### Screenshots (if appropriate) tl;dr Without `-Ppyspark` profile - Before <img width="856" alt="screen shot 2017-04-02 at 2 50 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584778/0e8ec6b0-17b4-11e7-9f0d-f2599fd7bd63.png"> - After <img width="893" alt="screen shot 2017-04-02 at 2 28 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584779/10b7ed68-17b4-11e7-90d4-aa95eb9bba2d.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no I want to include `SparkR` by default(= remove `-PsparkR` build profile) like this as a next step. I want to ask how Zeppelin community think about this. Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #2213 from AhyoungRyu/ZEPPELIN-2298/includePysparkByDefault and squashes the following commits: f7bcf06 [AhyoungRyu] Remove -Ppyspark in virtual_machine.md 458ac02 [AhyoungRyu] Remove the rest of -Ppyspark in blind side of Zeppelin :) cee1e87 [AhyoungRyu] Change py4j.version -> python.py4j.version ce43158 [AhyoungRyu] Change py4j.version -> spark.py4j.version fa4fb36 [AhyoungRyu] Remove the rest of -Ppyspark 30aac81 [AhyoungRyu] Remove -Ppyspark build flag
2017-04-04 01:49:36 +00:00
<python.py4j.version>0.9.2</python.py4j.version>
ZEPPELIN-1115: Python - interpreter for SQL over DataFrame ### What is this PR for? Add new interpreter to Python group: `%python.sql` for SQL over DataFrame support ### What type of PR is it? Improvement ### TODOs * [x] add new interpreter `%python.sql` * [x] add test * [x] make Python-dependant tests, excluded from CI * PythonInterpreterWithPythonInstalledTest * PythonPandasSqlInterpreterTest * run manually by `mvn -Dpython.test.exclude='' test -pl python -am` * [x] add docs `%python.sql` * [x] make `%python.sql` fail gracefully in case there is no Pandas or PandaSQL installed * [x] after #747 is merged - rebase and remove `-Dpython.test.exclude=''` from both profiles ### What is the Jira issue? [ZEPPELIN-1115](https://issues.apache.org/jira/browse/ZEPPELIN-1115) ### How should this be tested? `mvn -Dpython.test.exclude='' test -pl python -am` should pass or manually run - Given the DataFrame i.e ``` %python import pandas as pd rates = pd.read_csv("bank.csv", sep=";") ``` - SQL query it like ``` %python.sql SELECT * FROM rates LIMIT 10 ``` ### Screenshots (if appropriate) ![screen shot 2016-07-11 at 23 56 04](https://cloud.githubusercontent.com/assets/5582506/16735171/1ebb9354-47c3-11e6-9354-6364e9374a20.png) ### Questions: * Does the licenses files need update? No, no dependencies were included in source or binary release * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Alexander Bezzubov <bzz@apache.org> Closes #1164 from bzz/ZEPPELIN-1115/python/add-sql-for-dataframes and squashes the following commits: 0f2f852 [Alexander Bezzubov] Fail SQL gracefully if no python dependencies installed aca2bdf [Alexander Bezzubov] Fix typos in docs :zap: 158ba6a [Alexander Bezzubov] Remove third-party dependant test from CI 5fe46fc [Alexander Bezzubov] Update Python Matplotlib notebook example 72884c8 [Alexander Bezzubov] Add docs for %python.sql feature e931dc4 [Alexander Bezzubov] Make test for PythonPandasSqlInterpreter usable 76bbb44 [Alexander Bezzubov] Complete implementation of the PythonPandasSqlInterpreter f6ca1eb [Alexander Bezzubov] Add %python.sql to interpreter menue 11ba490 [Alexander Bezzubov] Add draft implementation of %python.sql for DataFrames
2016-07-14 05:15:42 +00:00
<python.test.exclude>
**/PythonInterpreterWithPythonInstalledTest.java,
ZEPPELIN-1345 - Create a custom matplotlib backend that natively supports inline plotting in a python interpreter cell ### What is this PR for? This PR is the first of two major steps needed to improve matplotlib integration in Zeppelin (ZEPPELIN-1344). The latter, which is a plotting backend with fully interactive tools enabled, will be done afterwards in a separate PR. This PR specifically for automatically displaying output from calls to matplotlib plotting functions inline with each paragraph. Thanks to the addition of post-execute hooks (ZEPPELIN-1423), there is no need to call any `show()` function to display an inline plot, just like in Jupyter. ### What type of PR is it? Improvement ### Todos The main code has been written and anyone who reads this is encouraged to test it, but there are a few minor todos: - [x] - Add unit tests - [x] - Add documentation - [x] - Add screenshot showing iterative plotting with angular mode ### What is the Jira issue? [ZEPPELIN-1345](https://issues.apache.org/jira/browse/ZEPPELIN-1345) ### How should this be tested? In a pyspark or python paragraph, enter and run ``` python import matplotlib.pyplot as plt plt.plot([1, 2, 3]) ``` The plot should be displayed automatically without calling any `show()` function whatsoever. A special method called `configure_mpl()` can also be used to modify the inline plotting behavior. For example, ``` python z.configure_mpl(close=False, angular=True) plt.plot([1, 2, 3]) ``` allows for iterative updates to the plot provided you have PY4J installed for your python installation (which of course is always the case if you use pypsark). To clarify, this feature only currently works with pyspark (not python as there are no `angularBind()` and `angularUnbind()` methods yet). Doing something like: ``` plt.plot([3, 2, 1]) ``` will update the plot that was generated by the previous paragraph by leveraging Zeppelin's Angular Display System. However, by setting `close=False`, matplotlib will no longer automatically close figures so it is now up to the user to explicitly close each figure instance they create. There's quite a bit more options for `z.configure_mpl()`, but I will save that discussion for the documentation. ### Screenshots (if appropriate) ![img](http://i.imgur.com/e1xHKnV.gif) ### Questions: - Does the licenses files need update? No - Is there breaking changes for older versions? No - Does this needs documentation? Yes Author: Alex Goodman <agoodm@users.noreply.github.com> Closes #1534 from agoodm/ZEPPELIN-1345 and squashes the following commits: 9ef6ff7 [Alex Goodman] Move mpl backend files to /interpreter 24f89c6 [Alex Goodman] Catch potential NullPointerExceptions from hook registry bdb584e [Alex Goodman] Make sure expressions are printed when no plots are shown 22b6fe4 [Alex Goodman] Remove unused variable d3d1aa0 [Alex Goodman] Fix CI test failure c90d204 [Alex Goodman] Update spark.md bcf0bf3 [Alex Goodman] Update python.md for new matplotlib integration c9b65a5 [Alex Goodman] Add iterative plotting example image 8029a05 [Alex Goodman] Update python/README.md f2d9e86 [Alex Goodman] Exclude tests are excluded in python/pom.xml 86b1c90 [Alex Goodman] Fix tutorial notebook not loading c37b00f [Alex Goodman] Fix legend in tutorial notebook a321d79 [Alex Goodman] Update python.md 82350e3 [Alex Goodman] Update matplotlib tutorial notebook 9792f97 [Alex Goodman] Add unit tests 8b9b973 [Alex Goodman] Fix NullPointerExceptions in unit tests 82135ad [Alex Goodman] Removed unused variable f9c9498 [Alex Goodman] Added support for Angular Display System edf750a [Alex Goodman] Add new matplotlib backend for python/pyspark interpreters
2016-11-06 06:03:04 +00:00
**/PythonInterpreterPandasSqlTest.java,
**/PythonInterpreterMatplotlibTest.java
ZEPPELIN-1115: Python - interpreter for SQL over DataFrame ### What is this PR for? Add new interpreter to Python group: `%python.sql` for SQL over DataFrame support ### What type of PR is it? Improvement ### TODOs * [x] add new interpreter `%python.sql` * [x] add test * [x] make Python-dependant tests, excluded from CI * PythonInterpreterWithPythonInstalledTest * PythonPandasSqlInterpreterTest * run manually by `mvn -Dpython.test.exclude='' test -pl python -am` * [x] add docs `%python.sql` * [x] make `%python.sql` fail gracefully in case there is no Pandas or PandaSQL installed * [x] after #747 is merged - rebase and remove `-Dpython.test.exclude=''` from both profiles ### What is the Jira issue? [ZEPPELIN-1115](https://issues.apache.org/jira/browse/ZEPPELIN-1115) ### How should this be tested? `mvn -Dpython.test.exclude='' test -pl python -am` should pass or manually run - Given the DataFrame i.e ``` %python import pandas as pd rates = pd.read_csv("bank.csv", sep=";") ``` - SQL query it like ``` %python.sql SELECT * FROM rates LIMIT 10 ``` ### Screenshots (if appropriate) ![screen shot 2016-07-11 at 23 56 04](https://cloud.githubusercontent.com/assets/5582506/16735171/1ebb9354-47c3-11e6-9354-6364e9374a20.png) ### Questions: * Does the licenses files need update? No, no dependencies were included in source or binary release * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Alexander Bezzubov <bzz@apache.org> Closes #1164 from bzz/ZEPPELIN-1115/python/add-sql-for-dataframes and squashes the following commits: 0f2f852 [Alexander Bezzubov] Fail SQL gracefully if no python dependencies installed aca2bdf [Alexander Bezzubov] Fix typos in docs :zap: 158ba6a [Alexander Bezzubov] Remove third-party dependant test from CI 5fe46fc [Alexander Bezzubov] Update Python Matplotlib notebook example 72884c8 [Alexander Bezzubov] Add docs for %python.sql feature e931dc4 [Alexander Bezzubov] Make test for PythonPandasSqlInterpreter usable 76bbb44 [Alexander Bezzubov] Complete implementation of the PythonPandasSqlInterpreter f6ca1eb [Alexander Bezzubov] Add %python.sql to interpreter menue 11ba490 [Alexander Bezzubov] Add draft implementation of %python.sql for DataFrames
2016-07-14 05:15:42 +00:00
</python.test.exclude>
[ZEPPELIN-2753] Basic Implementation of IPython Interpreter ### What is this PR for? This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well. For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter. Main features: * IPython interpreter support ** All the ipython features are available, including visualization, ipython magics. * ZeppelinContext support * Streaming output support * Support Ipython in PySpark Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here. In unit test, I only verify the following 3 popular visualization library. could add more later. * matplotlib * bokeh * ggplot ### What type of PR is it? [Feature ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2753 ### How should this be tested? Unit test is added. ### Screenshots (if appropriate) Verify bokeh in IPython Interpreter ![image](https://user-images.githubusercontent.com/164491/27999716-756d749e-6552-11e7-90bb-4c6b08f4ab5c.png) Verify matplotlib ![image](https://user-images.githubusercontent.com/164491/28046960-e881b28e-6619-11e7-9e1f-7f4662f842f3.png) Verify ZeppelinContext ![image](https://user-images.githubusercontent.com/164491/28119378-4212620c-6747-11e7-89d5-3b5e609593ce.png) Verify Streaming ![streaming](https://user-images.githubusercontent.com/164491/28950974-8f92fe1e-78fa-11e7-841f-3174da198bb7.gif) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits: e869f31 [Jeff Zhang] address comments b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
2017-08-26 03:59:43 +00:00
<grpc.version>1.4.0</grpc.version>
<plugin.shade.version>2.4.1</plugin.shade.version>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
</properties>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>zeppelin-interpreter</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-exec</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>net.sf.py4j</groupId>
<artifactId>py4j</artifactId>
[ZEPPELIN-2298] Remove -Ppyspark build profile ### What is this PR for? Currently users who build Zeppelin from source need to include `-Ppyspark` to use `%pyspark` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Ppyspark` and make pyspark related libraries automatically downloaded when we build Zeppelin. ### What type of PR is it? Improvement ### Todos * [x] - remove the rest of `-Ppyspark` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after getting feedback ### What is the Jira issue? [ZEPPELIN-2298](https://issues.apache.org/jira/browse/ZEPPELIN-2298) ### How should this be tested? 1. Apply this patch 2. Build source with below command ``` mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies' ``` Aftr this step, there will be `pyspark` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Ppyspark` build profile. 4. Restart Zeppelin. To make sure, run any python code e.g. ``` %pyspark print("Hello "+z.input("name")) ``` It should be run successfully without any error ### Screenshots (if appropriate) tl;dr Without `-Ppyspark` profile - Before <img width="856" alt="screen shot 2017-04-02 at 2 50 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584778/0e8ec6b0-17b4-11e7-9f0d-f2599fd7bd63.png"> - After <img width="893" alt="screen shot 2017-04-02 at 2 28 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584779/10b7ed68-17b4-11e7-90d4-aa95eb9bba2d.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no I want to include `SparkR` by default(= remove `-PsparkR` build profile) like this as a next step. I want to ask how Zeppelin community think about this. Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #2213 from AhyoungRyu/ZEPPELIN-2298/includePysparkByDefault and squashes the following commits: f7bcf06 [AhyoungRyu] Remove -Ppyspark in virtual_machine.md 458ac02 [AhyoungRyu] Remove the rest of -Ppyspark in blind side of Zeppelin :) cee1e87 [AhyoungRyu] Change py4j.version -> python.py4j.version ce43158 [AhyoungRyu] Change py4j.version -> spark.py4j.version fa4fb36 [AhyoungRyu] Remove the rest of -Ppyspark 30aac81 [AhyoungRyu] Remove -Ppyspark build flag
2017-04-04 01:49:36 +00:00
<version>${python.py4j.version}</version>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</dependency>
[ZEPPELIN-2753] Basic Implementation of IPython Interpreter ### What is this PR for? This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well. For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter. Main features: * IPython interpreter support ** All the ipython features are available, including visualization, ipython magics. * ZeppelinContext support * Streaming output support * Support Ipython in PySpark Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here. In unit test, I only verify the following 3 popular visualization library. could add more later. * matplotlib * bokeh * ggplot ### What type of PR is it? [Feature ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2753 ### How should this be tested? Unit test is added. ### Screenshots (if appropriate) Verify bokeh in IPython Interpreter ![image](https://user-images.githubusercontent.com/164491/27999716-756d749e-6552-11e7-90bb-4c6b08f4ab5c.png) Verify matplotlib ![image](https://user-images.githubusercontent.com/164491/28046960-e881b28e-6619-11e7-9e1f-7f4662f842f3.png) Verify ZeppelinContext ![image](https://user-images.githubusercontent.com/164491/28119378-4212620c-6747-11e7-89d5-3b5e609593ce.png) Verify Streaming ![streaming](https://user-images.githubusercontent.com/164491/28950974-8f92fe1e-78fa-11e7-841f-3174da198bb7.gif) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits: e869f31 [Jeff Zhang] address comments b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
2017-08-26 03:59:43 +00:00
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty</artifactId>
<version>${grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>${grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
<version>${grpc.version}</version>
</dependency>
<!-- test libraries -->
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
[ZEPPELIN-2753] Basic Implementation of IPython Interpreter ### What is this PR for? This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well. For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter. Main features: * IPython interpreter support ** All the ipython features are available, including visualization, ipython magics. * ZeppelinContext support * Streaming output support * Support Ipython in PySpark Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here. In unit test, I only verify the following 3 popular visualization library. could add more later. * matplotlib * bokeh * ggplot ### What type of PR is it? [Feature ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2753 ### How should this be tested? Unit test is added. ### Screenshots (if appropriate) Verify bokeh in IPython Interpreter ![image](https://user-images.githubusercontent.com/164491/27999716-756d749e-6552-11e7-90bb-4c6b08f4ab5c.png) Verify matplotlib ![image](https://user-images.githubusercontent.com/164491/28046960-e881b28e-6619-11e7-9e1f-7f4662f842f3.png) Verify ZeppelinContext ![image](https://user-images.githubusercontent.com/164491/28119378-4212620c-6747-11e7-89d5-3b5e609593ce.png) Verify Streaming ![streaming](https://user-images.githubusercontent.com/164491/28950974-8f92fe1e-78fa-11e7-841f-3174da198bb7.gif) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits: e869f31 [Jeff Zhang] address comments b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
2017-08-26 03:59:43 +00:00
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.4.1.Final</version>
</extension>
</extensions>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
<plugins>
[ZEPPELIN-2753] Basic Implementation of IPython Interpreter ### What is this PR for? This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well. For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter. Main features: * IPython interpreter support ** All the ipython features are available, including visualization, ipython magics. * ZeppelinContext support * Streaming output support * Support Ipython in PySpark Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here. In unit test, I only verify the following 3 popular visualization library. could add more later. * matplotlib * bokeh * ggplot ### What type of PR is it? [Feature ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2753 ### How should this be tested? Unit test is added. ### Screenshots (if appropriate) Verify bokeh in IPython Interpreter ![image](https://user-images.githubusercontent.com/164491/27999716-756d749e-6552-11e7-90bb-4c6b08f4ab5c.png) Verify matplotlib ![image](https://user-images.githubusercontent.com/164491/28046960-e881b28e-6619-11e7-9e1f-7f4662f842f3.png) Verify ZeppelinContext ![image](https://user-images.githubusercontent.com/164491/28119378-4212620c-6747-11e7-89d5-3b5e609593ce.png) Verify Streaming ![streaming](https://user-images.githubusercontent.com/164491/28950974-8f92fe1e-78fa-11e7-841f-3174da198bb7.gif) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits: e869f31 [Jeff Zhang] address comments b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
2017-08-26 03:59:43 +00:00
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.5.0</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:3.3.0:exe:${os.detected.classifier}</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:1.4.0:exe:${os.detected.classifier}</pluginArtifact>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>compile-custom</goal>
</goals>
</execution>
</executions>
</plugin>
Rewrite PythonInterpreter. ### What is this PR for? I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter. 1. not working streaming output. - https://issues.apache.org/jira/browse/ZEPPELIN-2225 2. printed "..." when there is indent in the python code. - https://issues.apache.org/jira/browse/ZEPPELIN-1929 3. very slow output of matplotlib - https://issues.apache.org/jira/browse/ZEPPELIN-1894 - https://issues.apache.org/jira/browse/ZEPPELIN-1360 4. Unexpected output of matplotlib. - https://issues.apache.org/jira/browse/ZEPPELIN-2107 so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues. and I am going to recreate conda, docker for python interpreter ASAP. ### What type of PR is it? Bug Fix | Hot Fix | Refactoring ### How should this be tested? 1. not working streaming output. ``` import time for x in range(0, 5): print x time.sleep(1) ``` 2. printed "..." when there is indent in the python code. ``` def fn(): print("hi") fn() ``` 3. very slow output of matplotlib. ``` import matplotlib import sys import matplotlib.pyplot as plt plt.plot([1,2,3]) ``` 4. Unexpected output of matplotlib. ``` import matplotlib.pyplot as plt import matplotlib as mpl # Make a figure and axes with dimensions as desired. fig = plt.figure(figsize=(8, 3)) ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15]) ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15]) ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15]) # Set the colormap and norm to correspond to the data for which # the colorbar will be used. cmap = mpl.cm.cool norm = mpl.colors.Normalize(vmin=5, vmax=10) # ColorbarBase derives from ScalarMappable and puts a colorbar # in a specified axes, so it has everything needed for a # standalone colorbar. There are many more kwargs, but the # following gives a basic continuous colorbar with ticks # and labels. cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap, norm=norm, orientation='horizontal') cb1.set_label('Some Units') # The second example illustrates the use of a ListedColormap, a # BoundaryNorm, and extended ends to show the "over" and "under" # value colors. cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c']) cmap.set_over('0.25') cmap.set_under('0.75') # If a ListedColormap is used, the length of the bounds array must be # one greater than the length of the color list. The bounds must be # monotonically increasing. bounds = [1, 2, 4, 7, 8] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap, norm=norm, # to use 'extend', you must # specify two extra boundaries: boundaries=[0] + bounds + [13], extend='both', ticks=bounds, # optional spacing='proportional', orientation='horizontal') cb2.set_label('Discrete intervals, some other units') # The third example illustrates the use of custom length colorbar # extensions, used on a colorbar with discrete intervals. cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.], [1., .8, 0.], [1., .4, 0.]]) cmap.set_over((1., 0., 0.)) cmap.set_under((0., 0., 1.)) bounds = [-1., -.5, 0., .5, 1.] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap, norm=norm, boundaries=[-10] + bounds + [10], extend='both', # Make the length of each extension # the same as the length of the # interior colors: extendfrac='auto', ticks=bounds, spacing='uniform', orientation='horizontal') cb3.set_label('Custom extension lengths, some other units') plt.show() ``` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@zepl.com> Author: Lee moon soo <moon@apache.org> Author: HyungSung <hsshim@nflabs.com> Closes #2106 from astroshim/py4jPythonInterpreter and squashes the following commits: c9b195b [HyungSung] Merge pull request #16 from Leemoonsoo/py4jdocker e511ebe [Lee moon soo] add PythonDockerInterpreter to interpreter-setting.json a76b0d8 [Lee moon soo] fix test on python3 2eb5de7 [Lee moon soo] Fix PythonDockerInterpreterTest.java test 9fcf144 [Lee moon soo] Make python docker interpreter work using py4j 8a016c9 [astroshim] Merge branch 'master' into py4jPythonInterpreter aad7ee8 [astroshim] fix testcase ac92cdb [astroshim] fix python interpreter testcase e8570d2 [astroshim] fix ci for pandassql be5db4d [astroshim] fix pandas sql testcase f8e19be [astroshim] fix matplotlib testcase 046db88 [astroshim] add testcase e49ad24 [astroshim] add pandas 60e9820 [astroshim] bug fix about copying library 574bd21 [astroshim] fix interpreter-setting error a48df58 [astroshim] Merge branch 'master' into py4jPythonInterpreter 3c9585f [astroshim] update interpreter-setting.json a50179e [astroshim] add conda interpreter cbbc15c [astroshim] fix py4j path 5ae5120 [astroshim] fix interpreter-setting f17bff4 [astroshim] fix testcase failure. af097ac [astroshim] add testcase c3f5b78 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into py4jPythonInterpreter 1395875 [astroshim] removed unnecessary code. 276011e [astroshim] add py4j lib 7304919 [astroshim] initialize python interpreter using py4j
2017-03-18 09:24:16 +00:00
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
[ZEPPELIN-2753] Basic Implementation of IPython Interpreter ### What is this PR for? This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well. For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter. Main features: * IPython interpreter support ** All the ipython features are available, including visualization, ipython magics. * ZeppelinContext support * Streaming output support * Support Ipython in PySpark Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here. In unit test, I only verify the following 3 popular visualization library. could add more later. * matplotlib * bokeh * ggplot ### What type of PR is it? [Feature ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2753 ### How should this be tested? Unit test is added. ### Screenshots (if appropriate) Verify bokeh in IPython Interpreter ![image](https://user-images.githubusercontent.com/164491/27999716-756d749e-6552-11e7-90bb-4c6b08f4ab5c.png) Verify matplotlib ![image](https://user-images.githubusercontent.com/164491/28046960-e881b28e-6619-11e7-9e1f-7f4662f842f3.png) Verify ZeppelinContext ![image](https://user-images.githubusercontent.com/164491/28119378-4212620c-6747-11e7-89d5-3b5e609593ce.png) Verify Streaming ![streaming](https://user-images.githubusercontent.com/164491/28950974-8f92fe1e-78fa-11e7-841f-3174da198bb7.gif) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits: e869f31 [Jeff Zhang] address comments b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
2017-08-26 03:59:43 +00:00
<!-- publish test jar as well so that spark module can use it -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
Rewrite PythonInterpreter. ### What is this PR for? I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter. 1. not working streaming output. - https://issues.apache.org/jira/browse/ZEPPELIN-2225 2. printed "..." when there is indent in the python code. - https://issues.apache.org/jira/browse/ZEPPELIN-1929 3. very slow output of matplotlib - https://issues.apache.org/jira/browse/ZEPPELIN-1894 - https://issues.apache.org/jira/browse/ZEPPELIN-1360 4. Unexpected output of matplotlib. - https://issues.apache.org/jira/browse/ZEPPELIN-2107 so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues. and I am going to recreate conda, docker for python interpreter ASAP. ### What type of PR is it? Bug Fix | Hot Fix | Refactoring ### How should this be tested? 1. not working streaming output. ``` import time for x in range(0, 5): print x time.sleep(1) ``` 2. printed "..." when there is indent in the python code. ``` def fn(): print("hi") fn() ``` 3. very slow output of matplotlib. ``` import matplotlib import sys import matplotlib.pyplot as plt plt.plot([1,2,3]) ``` 4. Unexpected output of matplotlib. ``` import matplotlib.pyplot as plt import matplotlib as mpl # Make a figure and axes with dimensions as desired. fig = plt.figure(figsize=(8, 3)) ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15]) ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15]) ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15]) # Set the colormap and norm to correspond to the data for which # the colorbar will be used. cmap = mpl.cm.cool norm = mpl.colors.Normalize(vmin=5, vmax=10) # ColorbarBase derives from ScalarMappable and puts a colorbar # in a specified axes, so it has everything needed for a # standalone colorbar. There are many more kwargs, but the # following gives a basic continuous colorbar with ticks # and labels. cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap, norm=norm, orientation='horizontal') cb1.set_label('Some Units') # The second example illustrates the use of a ListedColormap, a # BoundaryNorm, and extended ends to show the "over" and "under" # value colors. cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c']) cmap.set_over('0.25') cmap.set_under('0.75') # If a ListedColormap is used, the length of the bounds array must be # one greater than the length of the color list. The bounds must be # monotonically increasing. bounds = [1, 2, 4, 7, 8] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap, norm=norm, # to use 'extend', you must # specify two extra boundaries: boundaries=[0] + bounds + [13], extend='both', ticks=bounds, # optional spacing='proportional', orientation='horizontal') cb2.set_label('Discrete intervals, some other units') # The third example illustrates the use of custom length colorbar # extensions, used on a colorbar with discrete intervals. cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.], [1., .8, 0.], [1., .4, 0.]]) cmap.set_over((1., 0., 0.)) cmap.set_under((0., 0., 1.)) bounds = [-1., -.5, 0., .5, 1.] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap, norm=norm, boundaries=[-10] + bounds + [10], extend='both', # Make the length of each extension # the same as the length of the # interior colors: extendfrac='auto', ticks=bounds, spacing='uniform', orientation='horizontal') cb3.set_label('Custom extension lengths, some other units') plt.show() ``` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@zepl.com> Author: Lee moon soo <moon@apache.org> Author: HyungSung <hsshim@nflabs.com> Closes #2106 from astroshim/py4jPythonInterpreter and squashes the following commits: c9b195b [HyungSung] Merge pull request #16 from Leemoonsoo/py4jdocker e511ebe [Lee moon soo] add PythonDockerInterpreter to interpreter-setting.json a76b0d8 [Lee moon soo] fix test on python3 2eb5de7 [Lee moon soo] Fix PythonDockerInterpreterTest.java test 9fcf144 [Lee moon soo] Make python docker interpreter work using py4j 8a016c9 [astroshim] Merge branch 'master' into py4jPythonInterpreter aad7ee8 [astroshim] fix testcase ac92cdb [astroshim] fix python interpreter testcase e8570d2 [astroshim] fix ci for pandassql be5db4d [astroshim] fix pandas sql testcase f8e19be [astroshim] fix matplotlib testcase 046db88 [astroshim] add testcase e49ad24 [astroshim] add pandas 60e9820 [astroshim] bug fix about copying library 574bd21 [astroshim] fix interpreter-setting error a48df58 [astroshim] Merge branch 'master' into py4jPythonInterpreter 3c9585f [astroshim] update interpreter-setting.json a50179e [astroshim] add conda interpreter cbbc15c [astroshim] fix py4j path 5ae5120 [astroshim] fix interpreter-setting f17bff4 [astroshim] fix testcase failure. af097ac [astroshim] add testcase c3f5b78 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into py4jPythonInterpreter 1395875 [astroshim] removed unnecessary code. 276011e [astroshim] add py4j lib 7304919 [astroshim] initialize python interpreter using py4j
2017-03-18 09:24:16 +00:00
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<exclude>${python.test.exclude}</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${plugin.shade.version}</version>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>org.apache.zeppelin.com.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>py4j</pattern>
<shadedPattern>org.apache.zeppelin.py4j</shadedPattern>
</relocation>
</relocations>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-enforcer-plugin</artifactId>
</plugin>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<skip>false</skip>
</configuration>
</plugin>
[ZEPPELIN-502] Python interpreter group ### What is this PR for? Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL. The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change. I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG. ### What type of PR is it? [Feature] ### Todos * [x] - Code review * [x] - Improve bootstrap.py : choose available helper functions and their names * [x] - Unit / IT tests ? * [x] documentation updates needed, that AhyoungRyu pointed out * [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license * [x] double-check that code formatting conforms project style guide * [x] the branch need to be rebased on latest master. ### What is the Jira issue? [ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22) ### How should this be tested? 1. In interpreter screen, in Python section, specify in python.path the python binary you want to use 2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities. 3. Install py4j (pip install py4j) if you want to use input form ### Screenshots ![image](https://cloud.githubusercontent.com/assets/12515751/14936724/5108fb60-0ef4-11e6-93ea-232a037f7957.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943716/98a75c4a-0fe0-11e6-9d4b-e10c39d53a15.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14936715/0eec90de-0ef4-11e6-811b-7ebe46f0d279.png) ![image](https://cloud.githubusercontent.com/assets/12515751/14943722/b89b7824-0fe0-11e6-9c73-c12f7372d487.png) ### Questions: * Does the licenses files need update? Yes, only bin-license (py4j) * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Hervé RIVIERE <hriviere@users.noreply.github.com> Closes #869 from hriviere/PR_interpreter_python and squashes the following commits: 80b6e75 [Hervé RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license a4b82a5 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 3252353 [Hervé RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention 54ec4f1 [Hervé RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review 6a831bc [Hervé RIVIERE] [ZEPPELIN-502] Add BSD py4j license 11e1b9c [Hervé RIVIERE] [ZEPPELIN-502] minor changes in python.md e5d0bdb [Hervé RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON c62ac98 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md 5008125 [Hervé RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description 7d533e1 [Hervé RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing fecaf25 [Hervé RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python 02d1320 [Hervé RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax) 60d2956 [Hervé RIVIERE] [ZEPPELIN-502] Indent as pep8 convention 9bdb192 [Hervé RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html 7142aa5 [Hervé RIVIERE] [ZEPPELIN-502] Catch exception in logger.error 1a86ad7 [Hervé RIVIERE] [ZEPPELIN-502] Python interpreter group
2016-05-30 20:07:26 +00:00
</plugins>
</build>
</project>