### What is this PR for?
%spark.pyspark paragraphs keep running for more than > 5 minutes and then at the end, throw TTransportException.
```
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
```
This happens because Zeppelin refers to `matplotlib.rcParams['savefig.format']` https://github.com/apache/zeppelin/blob/master/interpreter/lib/python/mpl_config.py#L83 which is only present in matplotlib 1.2.0 (https://github.com/matplotlib/matplotlib/blob/v1.2.0/matplotlibrc.template#L355) and onwards.
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
* [ZEPPELIN-2827](https://issues.apache.org/jira/browse/ZEPPELIN-2827)
### How should this be tested?
Try running spark.pyspark on a machine with matplotlib==1.1.1 installed, pyspark should work as expected.
### Questions:
* Does the licenses files need update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? N/A
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2516 from prabhjyotsingh/ZEPPELIN-2827 and squashes the following commits:
0321fcd3e [Prabhjyot Singh] check if `savefig.format` exists in mpl_config.py
da26f5613 [Prabhjyot Singh] add a version check along with check if matplotlib is installed
### What is this PR for?
https://github.com/apache/zeppelin/pull/2106 rewrote python interpreter. But dynamic form feature is not rewritten correctly.
### What type of PR is it?
Hot Fix
### Todos
* [x] - Bring dynamic form back
### What is the Jira issue?
https://github.com/apache/zeppelin/pull/2106
### How should this be tested?
run
```
%python
print("Hello "+z.input("name", "sun"))
```
```
%python
print("Hello "+z.select("day", [("1","mon"),
("2","tue"),
("3","wed"),
("4","thurs"),
("5","fri"),
("6","sat"),
("7","sun")]))
```
```
%python
options = [("apple","Apple"), ("banana","Banana"), ("orange","Orange")]
print("Hello "+ " and ".join(z.checkbox("fruit", options, ["apple"])))
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#2155 from Leemoonsoo/python_get_interpreter_context and squashes the following commits:
c5e584a [Lee moon soo] fix matplotlib display error on python 3.4
3e6603b [Lee moon soo] correctly handle zeppelin.python property.
5be8db4 [Lee moon soo] Expose a method to get InterpreterOutput, so user can call InterpreterOutput.clear()
a405a93 [Lee moon soo] implement dynamic form
### What is this PR for?
Matplotlib 2.0.0 was just released. It has introduced some major changes including some to the `rcParams` which makes it incompatible with Zeppelin's built-in plotting backend. This PR updates the backend config for 2.0.0 as well as the CI tests.
### What type of PR is it?
Hot Fix
### What is the Jira issue?
[ZEPPELIN-1980](https://issues.apache.org/jira/browse/ZEPPELIN-1980)
### How should this be tested?
Run the matplotlib tutorial notebook with matplotlib 2.0.0 installed.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
If possible, this should be included as a hotfix for 0.7.0.
Author: Alex Goodman <agoodm@users.noreply.github.com>
Closes#1912 from agoodm/ZEPPELIN-1980 and squashes the following commits:
202548c [Alex Goodman] Use figure.dpi instead of savefig.dpi
2678359 [Alex Goodman] Make CI install latest matplotlib
### What is this PR for?
Take 2 of #1618 because I had some earlier problems with rebasing. Since, then I have added some new features, namely:
- Matplotlib integration tests for pyspark
- `install_external_dependencies.sh` which conditionally installs the R or python dependencies based on the specified build profile in `.travis.yml`. This saves several minutes of time for a few of the build profiles since the R dependencies are compiled from source and therefore take quite a bit of time to install.
- The extra python unit tests which require external dependencies (`matplotlib` and `pandas`) are now relegated to two separate build profiles. This is done primarily to efficiently test both Python 2 and 3.
- Some minor bugs in the python and pyspark interpreters (mostly with respect to python 3 compatibility) were caught as a result of these tests, and are also fixed in this PR.
### What type of PR is it?
Improvement and Bugfix
### What is the Jira issue?
[ZEPPELIN-1639](https://issues.apache.org/jira/browse/ZEPPELIN-1639)
### How should this be tested?
CI tests should be green!
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Alex Goodman <agoodm@users.noreply.github.com>
Closes#1632 from agoodm/ZEPPELIN-1639 and squashes the following commits:
01380c2 [Alex Goodman] Make sure python 3 profile uses scala 2.11
363019e [Alex Goodman] Use spark 2.0 with python 3
a4f43af [Alex Goodman] Update comments in .travis.yml
5a60181 [Alex Goodman] Isolate python tests
73663f6 [Alex Goodman] Update tests for new InterpreterContext constructor
5709c5d [Alex Goodman] Re-add pyspark to build profile
ee95d67 [Alex Goodman] Move python 3 tests to all modules with spark 2.0
3a76958 [Alex Goodman] Travis
42da31c [Alex Goodman] Shorten python version
b6b88be [Alex Goodman] Add python dependencies to .travis.yml
### What is this PR for?
This PR is the first of two major steps needed to improve matplotlib integration in Zeppelin (ZEPPELIN-1344). The latter, which is a plotting backend with fully interactive tools enabled, will be done afterwards in a separate PR. This PR specifically for automatically displaying output from calls to matplotlib plotting functions inline with each paragraph. Thanks to the addition of post-execute hooks (ZEPPELIN-1423), there is no need to call any `show()` function to display an inline plot, just like in Jupyter.
### What type of PR is it?
Improvement
### Todos
The main code has been written and anyone who reads this is encouraged to test it, but there are a few minor todos:
- [x] - Add unit tests
- [x] - Add documentation
- [x] - Add screenshot showing iterative plotting with angular mode
### What is the Jira issue?
[ZEPPELIN-1345](https://issues.apache.org/jira/browse/ZEPPELIN-1345)
### How should this be tested?
In a pyspark or python paragraph, enter and run
``` python
import matplotlib.pyplot as plt
plt.plot([1, 2, 3])
```
The plot should be displayed automatically without calling any `show()` function whatsoever. A special method called `configure_mpl()` can also be used to modify the inline plotting behavior. For example,
``` python
z.configure_mpl(close=False, angular=True)
plt.plot([1, 2, 3])
```
allows for iterative updates to the plot provided you have PY4J installed for your python installation (which of course is always the case if you use pypsark). To clarify, this feature only currently works with pyspark (not python as there are no `angularBind()` and `angularUnbind()` methods yet). Doing something like:
```
plt.plot([3, 2, 1])
```
will update the plot that was generated by the previous paragraph by leveraging Zeppelin's Angular Display System. However, by setting `close=False`, matplotlib will no longer automatically close figures so it is now up to the user to explicitly close each figure instance they create. There's quite a bit more options for `z.configure_mpl()`, but I will save that discussion for the documentation.
### Screenshots (if appropriate)
### Questions:
- Does the licenses files need update? No
- Is there breaking changes for older versions? No
- Does this needs documentation? Yes
Author: Alex Goodman <agoodm@users.noreply.github.com>
Closes#1534 from agoodm/ZEPPELIN-1345 and squashes the following commits:
9ef6ff7 [Alex Goodman] Move mpl backend files to /interpreter
24f89c6 [Alex Goodman] Catch potential NullPointerExceptions from hook registry
bdb584e [Alex Goodman] Make sure expressions are printed when no plots are shown
22b6fe4 [Alex Goodman] Remove unused variable
d3d1aa0 [Alex Goodman] Fix CI test failure
c90d204 [Alex Goodman] Update spark.md
bcf0bf3 [Alex Goodman] Update python.md for new matplotlib integration
c9b65a5 [Alex Goodman] Add iterative plotting example image
8029a05 [Alex Goodman] Update python/README.md
f2d9e86 [Alex Goodman] Exclude tests are excluded in python/pom.xml
86b1c90 [Alex Goodman] Fix tutorial notebook not loading
c37b00f [Alex Goodman] Fix legend in tutorial notebook
a321d79 [Alex Goodman] Update python.md
82350e3 [Alex Goodman] Update matplotlib tutorial notebook
9792f97 [Alex Goodman] Add unit tests
8b9b973 [Alex Goodman] Fix NullPointerExceptions in unit tests
82135ad [Alex Goodman] Removed unused variable
f9c9498 [Alex Goodman] Added support for Angular Display System
edf750a [Alex Goodman] Add new matplotlib backend for python/pyspark interpreters
