034fdc6735
### What is this PR for? Add missing commands to the `python.conda` interpreter - `conda info` - `conda list` - `conda create` - `conda install` - `conda uninstall (alias of remove)` - `conda env *` #### Implementation Detail The reason I modified `PythonProcess` is due to NPE ```java // https://github.com/apache/zeppelin/blob/master/python/src/main/java/org/apache/zeppelin/python/PythonProcess.java#L107-L118 public String sendAndGetResult(String cmd) throws IOException { writer.println(cmd); writer.println(); writer.println("\"" + STATEMENT_END + "\""); StringBuilder output = new StringBuilder(); String line = null; // NPE when line is null while (!(line = reader.readLine()).contains(STATEMENT_END)) { logger.debug("Read line from python shell : " + line); output.append(line + "\n"); } return output.toString(); } ``` ``` java.lang.NullPointerException at org.apache.zeppelin.python.PythonProcess.sendAndGetResult(PythonProcess.java:113) at org.apache.zeppelin.python.PythonInterpreter.sendCommandToPython(PythonInterpreter.java:250) at org.apache.zeppelin.python.PythonInterpreter.bootStrapInterpreter(PythonInterpreter.java:272) at org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:100) at org.apache.zeppelin.python.PythonCondaInterpreter.restartPythonProcess(PythonCondaInterpreter.java:139) at org.apache.zeppelin.python.PythonCondaInterpreter.interpret(PythonCondaInterpreter.java:88) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:494) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` ### What type of PR is it? [Improvement | Refactoring] ### Todos * [x] - info * [x] - list * [x] - create * [x] - install * [x] - uninstall (= remove) * [x] - env * ### What is the Jira issue? [ZEPPELIN-1917](https://issues.apache.org/jira/browse/ZEPPELIN-1917) ### How should this be tested? 1. Install [miniconda](http://conda.pydata.org/miniconda.html) 2. Make sure that your python interpreter can use `conda` (check the Interpreter Binding page) 3. Remove `test` conda env since we will create in the following section ```sh $ conda env remove --yes --name test ``` 4. Run these commands with `%python.conda` ``` %python.conda info %python.conda env list %python.conda create --name test # you should be able to see `test` in the list %python.conda env list %python.conda activate pymysql %python.conda install pymysql # you should be able to import %python import pymysql.cursors %python.conda uninstall pymysql %python.conda deactivate pymysql # you should be able to see `No module named pymysql.cursor` since we deactivated %python import pymysql.cursors ``` ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO Author: 1ambda <1amb4a@gmail.com> Closes #1868 from 1ambda/ZEPPELIN-1917/improve-conda-interpreter and squashes the following commits: |
||
|---|---|---|
| .. | ||
| src | ||
| pom.xml | ||
| README.md | ||
Overview
Python interpreter for Apache Zeppelin
Architecture
Current interpreter implementation spawns new system python process through ProcessBuilder and re-directs it's stdin\strout to Zeppelin
Details
- UnitTests
To run full suit of tests, including ones that depend on real Python interpreter AND external libraries installed (like Pandas, Pandasql, etc) do
mvn -Dpython.test.exclude='' test -pl python -am
- Py4j support
Py4j enables Python programs to dynamically access Java objects in a JVM. It is required in order to use Zeppelin dynamic forms feature.
- bootstrap process
Interpreter environment is setup with thex bootstrap.py
It defines help() and z convenience functions
Dev prerequisites
-
Python 2 or 3 installed with py4j (0.9.2) and matplotlib (1.31 or later) installed on each
-
Tests only checks the interpreter logic and starts any Python process! Python process is mocked with a class that simply output it input.
-
Code wrote in
bootstrap.pyandbootstrap_input.pyshould always be Python 2 and 3 compliant. -
Use PEP8 convention for python code.
Technical overview
-
When interpreter is starting it launches a python process inside a Java ProcessBuilder. Python is started with -i (interactive mode) and -u (unbuffered stdin, stdout and stderr) options. Thus the interpreter has a "sleeping" python process.
-
Interpreter sends command to python with a Java
outputStreamWiterand read from anInputStreamReader. To know when stop reading stdout, interpreter sendsprint "*!?flush reader!?*"after each command and reads stdout until he receives back the*!?flush reader!?*. -
When interpreter is starting, it sends some Python code (bootstrap.py and bootstrap_input.py) to initialize default behavior and functions (
help(), z.input()...). bootstrap_input.py is sent only if py4j library is detected inside Python process. -
Py4J python and java libraries is used to load Input zeppelin Java class into the python process (make java code with python code !). Therefore the interpreter can directly create Zeppelin input form inside the Python process (and eventually with some python variable already defined). JVM opens a random open port to be accessible from python process.
-
JavaBuilder can't send SIGINT signal to interrupt paragraph execution. Therefore interpreter directly send a
kill SIGINT PIDto python process to interrupt execution. Python process catch SIGINT signal with some code defined in bootstrap.py -
Matplotlib figures are displayed inline with the notebook automatically using a built-in backend for zeppelin in conjunction with a post-execute hook.
-
%python.sqlsupport for Pandas DataFrames is optional and provided using https://github.com/yhat/pandasql if user have one installed