Elgato_dark/zeppelin
1
0
Fork 0
mirror of https://github.com/apache/zeppelin synced 2026-05-24 09:38:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
zeppelin/python
History
Alexander Bezzubov 7951e6c698 ZEPPELIN-1063: fix flaky python interpreter test 
### What is this PR for?
fix flaky python interpreter test

### What type of PR is it?
Bug Fix

### Todos
* [x] cleanup test code
* [x] fix flaky open port check

### What is the Jira issue?
[ZEPPELIN-1063](https://issues.apache.org/jira/browse/ZEPPELIN-1063)

### How should this be tested?
`mvn "-Dtest=org.apache.zeppelin.python.PythonInterpreterTest" test -pl python`

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Alexander Bezzubov <bzz@apache.org>

Closes #1094 from bzz/fix/python-tests/ZEPPELIN-1063 and squashes the following commits:

b9de5a2 [Alexander Bezzubov] Python: refactoring open port checking
09f3018 [Alexander Bezzubov] Python: refactoring - arrange imports
46e42e8 [Alexander Bezzubov] Python: refactoring mock() structure and JavaDocs
651d8e8 [Alexander Bezzubov] Python: refactoring loggers
d7f8cdd [Alexander Bezzubov] Python: normalize newlines in tests
2016-06-28 19:07:42 +09:00
..
src ZEPPELIN-1063: fix flaky python interpreter test 2016-06-28 19:07:42 +09:00
pom.xml [ZEPPELIN-986] Create publish release script 2016-06-22 21:22:07 -07:00
README.md Python interpreter and doc cleanup 2016-06-17 10:30:58 +09:00

README.md

Overview

Python interpreter for Apache Zeppelin

Architecture

Current interpreter implementation spawns new system python process through ProcessBuilder and re-directs it's stdin\strout to Zeppelin

Details

  • Py4j support

Py4j enables Python programs to dynamically access Java objects in a JVM. It is required in order to use Zeppelin dynamic forms feature.

  • bootstrap process

Interpreter environment is setup with thex bootstrap.py It defines help() and z convenience functions

Dev prerequisites

  • Python 2 or 3 installed with py4j (0.9.2) and matplotlib (1.31 or later) installed on each

  • Tests only checks the interpreter logic and starts any Python process! Python process is mocked with a class that simply output it input.

  • Code wrote in bootstrap.py and bootstrap_input.py should always be Python 2 and 3 compliant.

  • Use PEP8 convention for python code.

Technical overview

  • When interpreter is starting it launches a python process inside a Java ProcessBuilder. Python is started with -i (interactive mode) and -u (unbuffered stdin, stdout and stderr) options. Thus the interpreter has a "sleeping" python process.

  • Interpreter sends command to python with a Java outputStreamWiter and read from an InputStreamReader. To know when stop reading stdout, interpreter sends print "*!?flush reader!?*"after each command and reads stdout until he receives back the *!?flush reader!?*.

  • When interpreter is starting, it sends some Python code (bootstrap.py and bootstrap_input.py) to initialize default behavior and functions (help(), z.input()...). bootstrap_input.py is sent only if py4j library is detected inside Python process.

  • Py4J python and java libraries is used to load Input zeppelin Java class into the python process (make java code with python code !). Therefore the interpreter can directly create Zeppelin input form inside the Python process (and eventually with some python variable already defined). JVM opens a random open port to be accessible from python process.

  • JavaBuilder can't send SIGINT signal to interrupt paragraph execution. Therefore interpreter directly send a kill SIGINT PID to python process to interrupt execution. Python process catch SIGINT signal with some code defined in bootstrap.py

  • Matplotlib display feature is made with SVG export (in string) and then displays it with html code.