2016-05-30 20:07:26 +00:00
|
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
|
|
|
<!--
|
|
|
|
|
~ Licensed to the Apache Software Foundation (ASF) under one or more
|
|
|
|
|
~ contributor license agreements. See the NOTICE file distributed with
|
|
|
|
|
~ this work for additional information regarding copyright ownership.
|
|
|
|
|
~ The ASF licenses this file to You under the Apache License, Version 2.0
|
|
|
|
|
~ (the "License"); you may not use this file except in compliance with
|
|
|
|
|
~ the License. You may obtain a copy of the License at
|
|
|
|
|
~
|
|
|
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
~
|
|
|
|
|
~ Unless required by applicable law or agreed to in writing, software
|
|
|
|
|
~ distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
|
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
|
~ See the License for the specific language governing permissions and
|
|
|
|
|
~ limitations under the License.
|
|
|
|
|
-->
|
|
|
|
|
|
|
|
|
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
|
|
|
|
|
<modelVersion>4.0.0</modelVersion>
|
|
|
|
|
|
|
|
|
|
<parent>
|
2017-10-12 01:29:25 +00:00
|
|
|
<artifactId>interpreter-parent</artifactId>
|
2016-05-30 20:07:26 +00:00
|
|
|
<groupId>org.apache.zeppelin</groupId>
|
2018-01-29 02:41:14 +00:00
|
|
|
<version>0.9.0-SNAPSHOT</version>
|
2017-10-12 01:29:25 +00:00
|
|
|
<relativePath>../interpreter-parent</relativePath>
|
2016-05-30 20:07:26 +00:00
|
|
|
</parent>
|
|
|
|
|
|
|
|
|
|
<groupId>org.apache.zeppelin</groupId>
|
|
|
|
|
<artifactId>zeppelin-python</artifactId>
|
|
|
|
|
<packaging>jar</packaging>
|
2018-01-29 02:41:14 +00:00
|
|
|
<version>0.9.0-SNAPSHOT</version>
|
2016-05-30 20:07:26 +00:00
|
|
|
<name>Zeppelin: Python interpreter</name>
|
|
|
|
|
|
|
|
|
|
<properties>
|
2017-10-12 01:29:25 +00:00
|
|
|
<interpreter.name>python</interpreter.name>
|
[ZEPPELIN-2298] Remove -Ppyspark build profile
### What is this PR for?
Currently users who build Zeppelin from source need to include `-Ppyspark` to use `%pyspark` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Ppyspark` and make pyspark related libraries automatically downloaded when we build Zeppelin.
### What type of PR is it?
Improvement
### Todos
* [x] - remove the rest of `-Ppyspark` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after getting feedback
### What is the Jira issue?
[ZEPPELIN-2298](https://issues.apache.org/jira/browse/ZEPPELIN-2298)
### How should this be tested?
1. Apply this patch
2. Build source with below command
```
mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies'
```
Aftr this step, there will be `pyspark` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Ppyspark` build profile.
4. Restart Zeppelin. To make sure, run any python code e.g.
```
%pyspark
print("Hello "+z.input("name"))
```
It should be run successfully without any error
### Screenshots (if appropriate)
tl;dr Without `-Ppyspark` profile
- Before
<img width="856" alt="screen shot 2017-04-02 at 2 50 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584778/0e8ec6b0-17b4-11e7-9f0d-f2599fd7bd63.png">
- After
<img width="893" alt="screen shot 2017-04-02 at 2 28 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584779/10b7ed68-17b4-11e7-90d4-aa95eb9bba2d.png">
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
I want to include `SparkR` by default(= remove `-PsparkR` build profile) like this as a next step. I want to ask how Zeppelin community think about this.
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Closes #2213 from AhyoungRyu/ZEPPELIN-2298/includePysparkByDefault and squashes the following commits:
f7bcf06 [AhyoungRyu] Remove -Ppyspark in virtual_machine.md
458ac02 [AhyoungRyu] Remove the rest of -Ppyspark in blind side of Zeppelin :)
cee1e87 [AhyoungRyu] Change py4j.version -> python.py4j.version
ce43158 [AhyoungRyu] Change py4j.version -> spark.py4j.version
fa4fb36 [AhyoungRyu] Remove the rest of -Ppyspark
30aac81 [AhyoungRyu] Remove -Ppyspark build flag
2017-04-04 01:49:36 +00:00
|
|
|
<python.py4j.version>0.9.2</python.py4j.version>
|
2016-07-14 05:15:42 +00:00
|
|
|
<python.test.exclude>
|
|
|
|
|
**/PythonInterpreterWithPythonInstalledTest.java,
|
ZEPPELIN-1345 - Create a custom matplotlib backend that natively supports inline plotting in a python interpreter cell
### What is this PR for?
This PR is the first of two major steps needed to improve matplotlib integration in Zeppelin (ZEPPELIN-1344). The latter, which is a plotting backend with fully interactive tools enabled, will be done afterwards in a separate PR. This PR specifically for automatically displaying output from calls to matplotlib plotting functions inline with each paragraph. Thanks to the addition of post-execute hooks (ZEPPELIN-1423), there is no need to call any `show()` function to display an inline plot, just like in Jupyter.
### What type of PR is it?
Improvement
### Todos
The main code has been written and anyone who reads this is encouraged to test it, but there are a few minor todos:
- [x] - Add unit tests
- [x] - Add documentation
- [x] - Add screenshot showing iterative plotting with angular mode
### What is the Jira issue?
[ZEPPELIN-1345](https://issues.apache.org/jira/browse/ZEPPELIN-1345)
### How should this be tested?
In a pyspark or python paragraph, enter and run
``` python
import matplotlib.pyplot as plt
plt.plot([1, 2, 3])
```
The plot should be displayed automatically without calling any `show()` function whatsoever. A special method called `configure_mpl()` can also be used to modify the inline plotting behavior. For example,
``` python
z.configure_mpl(close=False, angular=True)
plt.plot([1, 2, 3])
```
allows for iterative updates to the plot provided you have PY4J installed for your python installation (which of course is always the case if you use pypsark). To clarify, this feature only currently works with pyspark (not python as there are no `angularBind()` and `angularUnbind()` methods yet). Doing something like:
```
plt.plot([3, 2, 1])
```
will update the plot that was generated by the previous paragraph by leveraging Zeppelin's Angular Display System. However, by setting `close=False`, matplotlib will no longer automatically close figures so it is now up to the user to explicitly close each figure instance they create. There's quite a bit more options for `z.configure_mpl()`, but I will save that discussion for the documentation.
### Screenshots (if appropriate)

### Questions:
- Does the licenses files need update? No
- Is there breaking changes for older versions? No
- Does this needs documentation? Yes
Author: Alex Goodman <agoodm@users.noreply.github.com>
Closes #1534 from agoodm/ZEPPELIN-1345 and squashes the following commits:
9ef6ff7 [Alex Goodman] Move mpl backend files to /interpreter
24f89c6 [Alex Goodman] Catch potential NullPointerExceptions from hook registry
bdb584e [Alex Goodman] Make sure expressions are printed when no plots are shown
22b6fe4 [Alex Goodman] Remove unused variable
d3d1aa0 [Alex Goodman] Fix CI test failure
c90d204 [Alex Goodman] Update spark.md
bcf0bf3 [Alex Goodman] Update python.md for new matplotlib integration
c9b65a5 [Alex Goodman] Add iterative plotting example image
8029a05 [Alex Goodman] Update python/README.md
f2d9e86 [Alex Goodman] Exclude tests are excluded in python/pom.xml
86b1c90 [Alex Goodman] Fix tutorial notebook not loading
c37b00f [Alex Goodman] Fix legend in tutorial notebook
a321d79 [Alex Goodman] Update python.md
82350e3 [Alex Goodman] Update matplotlib tutorial notebook
9792f97 [Alex Goodman] Add unit tests
8b9b973 [Alex Goodman] Fix NullPointerExceptions in unit tests
82135ad [Alex Goodman] Removed unused variable
f9c9498 [Alex Goodman] Added support for Angular Display System
edf750a [Alex Goodman] Add new matplotlib backend for python/pyspark interpreters
2016-11-06 06:03:04 +00:00
|
|
|
**/PythonInterpreterPandasSqlTest.java,
|
|
|
|
|
**/PythonInterpreterMatplotlibTest.java
|
2016-07-14 05:15:42 +00:00
|
|
|
</python.test.exclude>
|
2017-08-26 03:59:43 +00:00
|
|
|
<grpc.version>1.4.0</grpc.version>
|
2017-07-17 05:02:09 +00:00
|
|
|
<plugin.shade.version>2.4.1</plugin.shade.version>
|
2016-05-30 20:07:26 +00:00
|
|
|
</properties>
|
|
|
|
|
|
|
|
|
|
<dependencies>
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>${project.groupId}</groupId>
|
|
|
|
|
<artifactId>zeppelin-interpreter</artifactId>
|
|
|
|
|
<version>${project.version}</version>
|
|
|
|
|
<scope>provided</scope>
|
|
|
|
|
</dependency>
|
|
|
|
|
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>org.apache.commons</groupId>
|
|
|
|
|
<artifactId>commons-exec</artifactId>
|
|
|
|
|
<version>1.3</version>
|
|
|
|
|
</dependency>
|
|
|
|
|
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>net.sf.py4j</groupId>
|
|
|
|
|
<artifactId>py4j</artifactId>
|
[ZEPPELIN-2298] Remove -Ppyspark build profile
### What is this PR for?
Currently users who build Zeppelin from source need to include `-Ppyspark` to use `%pyspark` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Ppyspark` and make pyspark related libraries automatically downloaded when we build Zeppelin.
### What type of PR is it?
Improvement
### Todos
* [x] - remove the rest of `-Ppyspark` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after getting feedback
### What is the Jira issue?
[ZEPPELIN-2298](https://issues.apache.org/jira/browse/ZEPPELIN-2298)
### How should this be tested?
1. Apply this patch
2. Build source with below command
```
mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies'
```
Aftr this step, there will be `pyspark` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Ppyspark` build profile.
4. Restart Zeppelin. To make sure, run any python code e.g.
```
%pyspark
print("Hello "+z.input("name"))
```
It should be run successfully without any error
### Screenshots (if appropriate)
tl;dr Without `-Ppyspark` profile
- Before
<img width="856" alt="screen shot 2017-04-02 at 2 50 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584778/0e8ec6b0-17b4-11e7-9f0d-f2599fd7bd63.png">
- After
<img width="893" alt="screen shot 2017-04-02 at 2 28 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584779/10b7ed68-17b4-11e7-90d4-aa95eb9bba2d.png">
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
I want to include `SparkR` by default(= remove `-PsparkR` build profile) like this as a next step. I want to ask how Zeppelin community think about this.
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Closes #2213 from AhyoungRyu/ZEPPELIN-2298/includePysparkByDefault and squashes the following commits:
f7bcf06 [AhyoungRyu] Remove -Ppyspark in virtual_machine.md
458ac02 [AhyoungRyu] Remove the rest of -Ppyspark in blind side of Zeppelin :)
cee1e87 [AhyoungRyu] Change py4j.version -> python.py4j.version
ce43158 [AhyoungRyu] Change py4j.version -> spark.py4j.version
fa4fb36 [AhyoungRyu] Remove the rest of -Ppyspark
30aac81 [AhyoungRyu] Remove -Ppyspark build flag
2017-04-04 01:49:36 +00:00
|
|
|
<version>${python.py4j.version}</version>
|
2016-05-30 20:07:26 +00:00
|
|
|
</dependency>
|
|
|
|
|
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>org.slf4j</groupId>
|
|
|
|
|
<artifactId>slf4j-api</artifactId>
|
|
|
|
|
</dependency>
|
|
|
|
|
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>org.slf4j</groupId>
|
|
|
|
|
<artifactId>slf4j-log4j12</artifactId>
|
|
|
|
|
</dependency>
|
2016-07-04 13:35:46 +00:00
|
|
|
|
2017-08-26 03:59:43 +00:00
|
|
|
<dependency>
|
|
|
|
|
<groupId>io.grpc</groupId>
|
|
|
|
|
<artifactId>grpc-netty</artifactId>
|
|
|
|
|
<version>${grpc.version}</version>
|
|
|
|
|
</dependency>
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>io.grpc</groupId>
|
|
|
|
|
<artifactId>grpc-protobuf</artifactId>
|
|
|
|
|
<version>${grpc.version}</version>
|
|
|
|
|
</dependency>
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>io.grpc</groupId>
|
|
|
|
|
<artifactId>grpc-stub</artifactId>
|
|
|
|
|
<version>${grpc.version}</version>
|
|
|
|
|
</dependency>
|
2017-07-17 05:02:09 +00:00
|
|
|
|
2016-12-05 04:34:09 +00:00
|
|
|
<!-- test libraries -->
|
2016-05-30 20:07:26 +00:00
|
|
|
<dependency>
|
|
|
|
|
<groupId>junit</groupId>
|
|
|
|
|
<artifactId>junit</artifactId>
|
|
|
|
|
<scope>test</scope>
|
|
|
|
|
</dependency>
|
|
|
|
|
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>org.mockito</groupId>
|
|
|
|
|
<artifactId>mockito-all</artifactId>
|
|
|
|
|
<scope>test</scope>
|
|
|
|
|
</dependency>
|
|
|
|
|
</dependencies>
|
|
|
|
|
|
|
|
|
|
<build>
|
2017-08-26 03:59:43 +00:00
|
|
|
|
|
|
|
|
<extensions>
|
|
|
|
|
<extension>
|
|
|
|
|
<groupId>kr.motd.maven</groupId>
|
|
|
|
|
<artifactId>os-maven-plugin</artifactId>
|
|
|
|
|
<version>1.4.1.Final</version>
|
|
|
|
|
</extension>
|
|
|
|
|
</extensions>
|
|
|
|
|
|
2016-05-30 20:07:26 +00:00
|
|
|
<plugins>
|
2017-08-26 03:59:43 +00:00
|
|
|
|
|
|
|
|
<plugin>
|
|
|
|
|
<groupId>org.xolstice.maven.plugins</groupId>
|
|
|
|
|
<artifactId>protobuf-maven-plugin</artifactId>
|
|
|
|
|
<version>0.5.0</version>
|
|
|
|
|
<configuration>
|
|
|
|
|
<protocArtifact>com.google.protobuf:protoc:3.3.0:exe:${os.detected.classifier}</protocArtifact>
|
|
|
|
|
<pluginId>grpc-java</pluginId>
|
|
|
|
|
<pluginArtifact>io.grpc:protoc-gen-grpc-java:1.4.0:exe:${os.detected.classifier}</pluginArtifact>
|
|
|
|
|
</configuration>
|
|
|
|
|
<executions>
|
|
|
|
|
<execution>
|
|
|
|
|
<goals>
|
|
|
|
|
<goal>compile</goal>
|
|
|
|
|
<goal>compile-custom</goal>
|
|
|
|
|
</goals>
|
|
|
|
|
</execution>
|
|
|
|
|
</executions>
|
|
|
|
|
</plugin>
|
|
|
|
|
|
Rewrite PythonInterpreter.
### What is this PR for?
I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter.
1. not working streaming output.
- https://issues.apache.org/jira/browse/ZEPPELIN-2225
2. printed "..." when there is indent in the python code.
- https://issues.apache.org/jira/browse/ZEPPELIN-1929
3. very slow output of matplotlib
- https://issues.apache.org/jira/browse/ZEPPELIN-1894
- https://issues.apache.org/jira/browse/ZEPPELIN-1360
4. Unexpected output of matplotlib.
- https://issues.apache.org/jira/browse/ZEPPELIN-2107
so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues.
and I am going to recreate conda, docker for python interpreter ASAP.
### What type of PR is it?
Bug Fix | Hot Fix | Refactoring
### How should this be tested?
1. not working streaming output.
```
import time
for x in range(0, 5):
print x
time.sleep(1)
```
2. printed "..." when there is indent in the python code.
```
def fn():
print("hi")
fn()
```
3. very slow output of matplotlib.
```
import matplotlib
import sys
import matplotlib.pyplot as plt
plt.plot([1,2,3])
```
4. Unexpected output of matplotlib.
```
import matplotlib.pyplot as plt
import matplotlib as mpl
# Make a figure and axes with dimensions as desired.
fig = plt.figure(figsize=(8, 3))
ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15])
ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15])
ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15])
# Set the colormap and norm to correspond to the data for which
# the colorbar will be used.
cmap = mpl.cm.cool
norm = mpl.colors.Normalize(vmin=5, vmax=10)
# ColorbarBase derives from ScalarMappable and puts a colorbar
# in a specified axes, so it has everything needed for a
# standalone colorbar. There are many more kwargs, but the
# following gives a basic continuous colorbar with ticks
# and labels.
cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap,
norm=norm,
orientation='horizontal')
cb1.set_label('Some Units')
# The second example illustrates the use of a ListedColormap, a
# BoundaryNorm, and extended ends to show the "over" and "under"
# value colors.
cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c'])
cmap.set_over('0.25')
cmap.set_under('0.75')
# If a ListedColormap is used, the length of the bounds array must be
# one greater than the length of the color list. The bounds must be
# monotonically increasing.
bounds = [1, 2, 4, 7, 8]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap,
norm=norm,
# to use 'extend', you must
# specify two extra boundaries:
boundaries=[0] + bounds + [13],
extend='both',
ticks=bounds, # optional
spacing='proportional',
orientation='horizontal')
cb2.set_label('Discrete intervals, some other units')
# The third example illustrates the use of custom length colorbar
# extensions, used on a colorbar with discrete intervals.
cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.],
[1., .8, 0.], [1., .4, 0.]])
cmap.set_over((1., 0., 0.))
cmap.set_under((0., 0., 1.))
bounds = [-1., -.5, 0., .5, 1.]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap,
norm=norm,
boundaries=[-10] + bounds + [10],
extend='both',
# Make the length of each extension
# the same as the length of the
# interior colors:
extendfrac='auto',
ticks=bounds,
spacing='uniform',
orientation='horizontal')
cb3.set_label('Custom extension lengths, some other units')
plt.show()
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@zepl.com>
Author: Lee moon soo <moon@apache.org>
Author: HyungSung <hsshim@nflabs.com>
Closes #2106 from astroshim/py4jPythonInterpreter and squashes the following commits:
c9b195b [HyungSung] Merge pull request #16 from Leemoonsoo/py4jdocker
e511ebe [Lee moon soo] add PythonDockerInterpreter to interpreter-setting.json
a76b0d8 [Lee moon soo] fix test on python3
2eb5de7 [Lee moon soo] Fix PythonDockerInterpreterTest.java test
9fcf144 [Lee moon soo] Make python docker interpreter work using py4j
8a016c9 [astroshim] Merge branch 'master' into py4jPythonInterpreter
aad7ee8 [astroshim] fix testcase
ac92cdb [astroshim] fix python interpreter testcase
e8570d2 [astroshim] fix ci for pandassql
be5db4d [astroshim] fix pandas sql testcase
f8e19be [astroshim] fix matplotlib testcase
046db88 [astroshim] add testcase
e49ad24 [astroshim] add pandas
60e9820 [astroshim] bug fix about copying library
574bd21 [astroshim] fix interpreter-setting error
a48df58 [astroshim] Merge branch 'master' into py4jPythonInterpreter
3c9585f [astroshim] update interpreter-setting.json
a50179e [astroshim] add conda interpreter
cbbc15c [astroshim] fix py4j path
5ae5120 [astroshim] fix interpreter-setting
f17bff4 [astroshim] fix testcase failure.
af097ac [astroshim] add testcase
c3f5b78 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into py4jPythonInterpreter
1395875 [astroshim] removed unnecessary code.
276011e [astroshim] add py4j lib
7304919 [astroshim] initialize python interpreter using py4j
2017-03-18 09:24:16 +00:00
|
|
|
<plugin>
|
|
|
|
|
<artifactId>maven-antrun-plugin</artifactId>
|
|
|
|
|
<version>1.7</version>
|
|
|
|
|
<executions>
|
|
|
|
|
<execution>
|
|
|
|
|
<phase>package</phase>
|
|
|
|
|
<goals>
|
|
|
|
|
<goal>run</goal>
|
|
|
|
|
</goals>
|
|
|
|
|
</execution>
|
|
|
|
|
</executions>
|
|
|
|
|
</plugin>
|
|
|
|
|
|
2017-08-26 03:59:43 +00:00
|
|
|
<!-- publish test jar as well so that spark module can use it -->
|
|
|
|
|
<plugin>
|
|
|
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
|
|
|
<artifactId>maven-jar-plugin</artifactId>
|
|
|
|
|
<version>3.0.2</version>
|
|
|
|
|
<executions>
|
|
|
|
|
<execution>
|
|
|
|
|
<goals>
|
|
|
|
|
<goal>test-jar</goal>
|
|
|
|
|
</goals>
|
|
|
|
|
</execution>
|
|
|
|
|
</executions>
|
|
|
|
|
</plugin>
|
Rewrite PythonInterpreter.
### What is this PR for?
I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter.
1. not working streaming output.
- https://issues.apache.org/jira/browse/ZEPPELIN-2225
2. printed "..." when there is indent in the python code.
- https://issues.apache.org/jira/browse/ZEPPELIN-1929
3. very slow output of matplotlib
- https://issues.apache.org/jira/browse/ZEPPELIN-1894
- https://issues.apache.org/jira/browse/ZEPPELIN-1360
4. Unexpected output of matplotlib.
- https://issues.apache.org/jira/browse/ZEPPELIN-2107
so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues.
and I am going to recreate conda, docker for python interpreter ASAP.
### What type of PR is it?
Bug Fix | Hot Fix | Refactoring
### How should this be tested?
1. not working streaming output.
```
import time
for x in range(0, 5):
print x
time.sleep(1)
```
2. printed "..." when there is indent in the python code.
```
def fn():
print("hi")
fn()
```
3. very slow output of matplotlib.
```
import matplotlib
import sys
import matplotlib.pyplot as plt
plt.plot([1,2,3])
```
4. Unexpected output of matplotlib.
```
import matplotlib.pyplot as plt
import matplotlib as mpl
# Make a figure and axes with dimensions as desired.
fig = plt.figure(figsize=(8, 3))
ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15])
ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15])
ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15])
# Set the colormap and norm to correspond to the data for which
# the colorbar will be used.
cmap = mpl.cm.cool
norm = mpl.colors.Normalize(vmin=5, vmax=10)
# ColorbarBase derives from ScalarMappable and puts a colorbar
# in a specified axes, so it has everything needed for a
# standalone colorbar. There are many more kwargs, but the
# following gives a basic continuous colorbar with ticks
# and labels.
cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap,
norm=norm,
orientation='horizontal')
cb1.set_label('Some Units')
# The second example illustrates the use of a ListedColormap, a
# BoundaryNorm, and extended ends to show the "over" and "under"
# value colors.
cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c'])
cmap.set_over('0.25')
cmap.set_under('0.75')
# If a ListedColormap is used, the length of the bounds array must be
# one greater than the length of the color list. The bounds must be
# monotonically increasing.
bounds = [1, 2, 4, 7, 8]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap,
norm=norm,
# to use 'extend', you must
# specify two extra boundaries:
boundaries=[0] + bounds + [13],
extend='both',
ticks=bounds, # optional
spacing='proportional',
orientation='horizontal')
cb2.set_label('Discrete intervals, some other units')
# The third example illustrates the use of custom length colorbar
# extensions, used on a colorbar with discrete intervals.
cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.],
[1., .8, 0.], [1., .4, 0.]])
cmap.set_over((1., 0., 0.))
cmap.set_under((0., 0., 1.))
bounds = [-1., -.5, 0., .5, 1.]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap,
norm=norm,
boundaries=[-10] + bounds + [10],
extend='both',
# Make the length of each extension
# the same as the length of the
# interior colors:
extendfrac='auto',
ticks=bounds,
spacing='uniform',
orientation='horizontal')
cb3.set_label('Custom extension lengths, some other units')
plt.show()
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@zepl.com>
Author: Lee moon soo <moon@apache.org>
Author: HyungSung <hsshim@nflabs.com>
Closes #2106 from astroshim/py4jPythonInterpreter and squashes the following commits:
c9b195b [HyungSung] Merge pull request #16 from Leemoonsoo/py4jdocker
e511ebe [Lee moon soo] add PythonDockerInterpreter to interpreter-setting.json
a76b0d8 [Lee moon soo] fix test on python3
2eb5de7 [Lee moon soo] Fix PythonDockerInterpreterTest.java test
9fcf144 [Lee moon soo] Make python docker interpreter work using py4j
8a016c9 [astroshim] Merge branch 'master' into py4jPythonInterpreter
aad7ee8 [astroshim] fix testcase
ac92cdb [astroshim] fix python interpreter testcase
e8570d2 [astroshim] fix ci for pandassql
be5db4d [astroshim] fix pandas sql testcase
f8e19be [astroshim] fix matplotlib testcase
046db88 [astroshim] add testcase
e49ad24 [astroshim] add pandas
60e9820 [astroshim] bug fix about copying library
574bd21 [astroshim] fix interpreter-setting error
a48df58 [astroshim] Merge branch 'master' into py4jPythonInterpreter
3c9585f [astroshim] update interpreter-setting.json
a50179e [astroshim] add conda interpreter
cbbc15c [astroshim] fix py4j path
5ae5120 [astroshim] fix interpreter-setting
f17bff4 [astroshim] fix testcase failure.
af097ac [astroshim] add testcase
c3f5b78 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into py4jPythonInterpreter
1395875 [astroshim] removed unnecessary code.
276011e [astroshim] add py4j lib
7304919 [astroshim] initialize python interpreter using py4j
2017-03-18 09:24:16 +00:00
|
|
|
|
2016-07-04 13:35:46 +00:00
|
|
|
<plugin>
|
|
|
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
|
|
|
<artifactId>maven-surefire-plugin</artifactId>
|
|
|
|
|
<configuration>
|
|
|
|
|
<excludes>
|
|
|
|
|
<exclude>${python.test.exclude}</exclude>
|
|
|
|
|
</excludes>
|
|
|
|
|
</configuration>
|
|
|
|
|
</plugin>
|
|
|
|
|
|
2017-07-17 05:02:09 +00:00
|
|
|
<plugin>
|
|
|
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
|
|
|
<artifactId>maven-shade-plugin</artifactId>
|
|
|
|
|
<version>${plugin.shade.version}</version>
|
|
|
|
|
<configuration>
|
|
|
|
|
<transformers>
|
|
|
|
|
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
|
|
|
|
|
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
|
|
|
|
|
<resource>reference.conf</resource>
|
|
|
|
|
</transformer>
|
|
|
|
|
</transformers>
|
|
|
|
|
<relocations>
|
|
|
|
|
<relocation>
|
|
|
|
|
<pattern>com.google.common</pattern>
|
|
|
|
|
<shadedPattern>org.apache.zeppelin.com.google.common</shadedPattern>
|
|
|
|
|
</relocation>
|
|
|
|
|
<relocation>
|
|
|
|
|
<pattern>py4j</pattern>
|
|
|
|
|
<shadedPattern>org.apache.zeppelin.py4j</shadedPattern>
|
|
|
|
|
</relocation>
|
|
|
|
|
</relocations>
|
|
|
|
|
</configuration>
|
|
|
|
|
<executions>
|
|
|
|
|
<execution>
|
|
|
|
|
<phase>package</phase>
|
|
|
|
|
<goals>
|
|
|
|
|
<goal>shade</goal>
|
|
|
|
|
</goals>
|
|
|
|
|
</execution>
|
|
|
|
|
</executions>
|
|
|
|
|
</plugin>
|
|
|
|
|
|
2017-10-12 01:29:25 +00:00
|
|
|
<plugin>
|
|
|
|
|
<artifactId>maven-enforcer-plugin</artifactId>
|
|
|
|
|
</plugin>
|
2016-05-30 20:07:26 +00:00
|
|
|
<plugin>
|
|
|
|
|
<artifactId>maven-dependency-plugin</artifactId>
|
2017-10-12 01:29:25 +00:00
|
|
|
</plugin>
|
|
|
|
|
<plugin>
|
|
|
|
|
<artifactId>maven-resources-plugin</artifactId>
|
2016-05-30 20:07:26 +00:00
|
|
|
</plugin>
|
2016-07-04 13:35:46 +00:00
|
|
|
|
2018-06-15 07:01:49 +00:00
|
|
|
<plugin>
|
|
|
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
|
|
|
<artifactId>maven-checkstyle-plugin</artifactId>
|
|
|
|
|
<configuration>
|
|
|
|
|
<skip>false</skip>
|
|
|
|
|
</configuration>
|
|
|
|
|
</plugin>
|
2016-05-30 20:07:26 +00:00
|
|
|
</plugins>
|
|
|
|
|
</build>
|
|
|
|
|
|
|
|
|
|
</project>
|