[ZEPPELIN-502] Python interpreter group

This commit is contained in:
Hervé RIVIERE 2016-04-13 23:23:12 +02:00
parent 7d00af4daf
commit 1a86ad73ee
8 changed files with 528 additions and 1 deletions

View file

@ -169,7 +169,7 @@
<property>
<name>zeppelin.interpreters</name>
<value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter</value>
<value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,,org.apache.zeppelin.python.PythonInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter</value>
<description>Comma separated interpreter configurations. First interpreter become a default</description>
</property>

View file

@ -0,0 +1,74 @@
---
layout: page
title: "Python Interpreter"
description: "Python Interpreter"
group: manual
---
{% include JB/setup %}
## Python 2 & 3 Interpreter for Apache Zeppelin
## Configuration
<table class="table-configuration">
<tr>
<th>Property</th>
<th>Default</th>
<th>Description</th>
</tr>
<tr>
<td>python.path</td>
<td>/usr/bin/python</td>
<td>Path of the already installed Python binary (could be python2 or python3)</td>
</tr>
</table>
## Enabling Python Interpreter
In a notebook, to enable the **Python** interpreter, click on the **Gear** icon and select **Python**
## Using the Python Interpreter
In a paragraph, use **_%python_** to select the **Python** interpreter and then input all commands.
The interpreter can only work if you already have python installed (the interpreter doesn't bring it own python binaries).
To access the help, type **help()**
## Python modules
The interpreter can use all modules already installed (with pip, easy_install...)
## Apply Zeppelin Dynamic Forms
You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html) inside your Python code.
Example :
```bash
%python
print "${input_form(name)=defaultValue}"
```
## Matplotlib integration
The python interpreter can display matplotlib graph with the function **_zeppelin_show()_**
You need to already have matplotlib module installed to use this functionality !
```bash
%python
import matplotlib.pyplot as plt
plt.figure()
(.. ..)
zeppelin_show(plt)
plt.close()
```
zeppelin_show function can take optional parameters to adapt graph width and height
```bash
%python
zeppelin_show(plt,width='50px')
zeppelin_show(plt,height='150px')
```
[![pythonmatplotlib](/docs/interpreter/screenshots/pythonMatplotlib.png)](/docs/interpreter/screenshots/pythonMatplotlib.png)
### Use matplotlib without XServer
Matplotlib needs a X server to plot graph.
If you don't have any XServer and you don't want to install one (typically, a server environment) , you can use [Xvfb](http://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml) to simulate it.
Althought Xvfb doesn't display anything, you will have any issue to use the python interpreter, matplotlib and Xvfb. The interpreter plot the graph on X and then extracts svg path (in strings) to send it to the zeppelin webapp.

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

View file

@ -103,6 +103,7 @@
<module>flink</module>
<module>ignite</module>
<module>kylin</module>
<module>python</module>
<module>lens</module>
<module>cassandra</module>
<module>elasticsearch</module>

134
python/pom.xml Normal file
View file

@ -0,0 +1,134 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>zeppelin</artifactId>
<groupId>org.apache.zeppelin</groupId>
<version>0.6.0-incubating-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
<groupId>org.apache.zeppelin</groupId>
<artifactId>zeppelin-python</artifactId>
<packaging>jar</packaging>
<version>0.6.0-incubating-SNAPSHOT</version>
<name>Zeppelin: Python interpreter</name>
<url>http://zeppelin.incubator.apache.org</url>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>zeppelin-interpreter</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-exec</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.7</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.3.1</version>
<executions>
<execution>
<id>enforce</id>
<phase>none</phase>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.8</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/../../interpreter/python</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>false</overWriteSnapshots>
<overWriteIfNewer>true</overWriteIfNewer>
<includeScope>runtime</includeScope>
</configuration>
</execution>
<execution>
<id>copy-artifact</id>
<phase>package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/../../interpreter/python</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>false</overWriteSnapshots>
<overWriteIfNewer>true</overWriteIfNewer>
<includeScope>runtime</includeScope>
<artifactItems>
<artifactItem>
<groupId>${project.groupId}</groupId>
<artifactId>${project.artifactId}</artifactId>
<version>${project.version}</version>
<type>${project.packaging}</type>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

View file

@ -0,0 +1,236 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.python;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
import org.apache.zeppelin.scheduler.Job;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.scheduler.SchedulerFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.*;
import java.lang.reflect.Field;
import java.util.Collection;
import java.util.List;
import java.util.Properties;
/**
* Python interpreter for Zeppelin.
*/
public class PythonInterpreter extends Interpreter {
Logger logger = LoggerFactory.getLogger(PythonInterpreter.class);
public static final String PYTHON_PATH = "python.path";
public static final String DEFAULT_PYTHON_PATH = "/usr/bin/python";
private String pythonPath;
InputStream stdout;
OutputStream stdin;
BufferedWriter writer;
BufferedReader reader;
Process process = null;
private long pythonPid;
static {
Interpreter.register(
"python",
"python",
PythonInterpreter.class.getName(),
new InterpreterPropertyBuilder()
.add(PYTHON_PATH, DEFAULT_PYTHON_PATH,
"Python path. Default : /usr/bin/python")
.build()
);
}
public PythonInterpreter(Properties property) {
super(property);
}
@Override
public void open() {
logger.info("Starting Python interpreter .....");
pythonPath = getProperty(PYTHON_PATH);
logger.info("Python path is set to:" + pythonPath );
ProcessBuilder builder = new ProcessBuilder(pythonPath, "-iu");
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
logger.info(e.getMessage());
}
pythonPid = getPidOfProcess(process);
logger.info("python PID : " + pythonPid);
stdout = process.getInputStream ();
stdin = process.getOutputStream();
writer = new BufferedWriter(new OutputStreamWriter(stdin));
reader = new BufferedReader(new InputStreamReader(stdout));
try {
bootStrapInterpreter();
} catch (IOException e) {
e.printStackTrace();
}
}
@Override
public void close() {
logger.info("closing Python interpreter .....");
try {
process.destroy();
reader.close();
writer.close();
stdin.close();
stdout.close();
} catch (IOException e) {
logger.info(e.getMessage());
}
}
@Override
public InterpreterResult interpret(String cmd, InterpreterContext contextInterpreter) {
try {
logger.info("Sending : \n " + cmd);
writer.write(cmd + "\n\n");
writer.write("print (\"*!?flush reader!?*\")\n\n");
writer.flush();
} catch (IOException e) {
e.printStackTrace();
}
String output = "";
String line;
try {
while (!(line = reader.readLine ()).contains("*!?flush reader!?*")){
logger.info("Readed line from python shell : " + line);
if (line.equals("...")) {
logger.info("Syntax error ! ");
output += "Syntax error ! ";
break;
}
output += "\r" + line + "\n";
}
} catch (IOException e) {
e.printStackTrace();
}
return new InterpreterResult(Code.SUCCESS, output.replaceAll(">>>", "")
.replaceAll("\\.\\.\\.", "").trim());
}
@Override
public void cancel(InterpreterContext context) {
if (pythonPid > -1) {
try {
logger.info("Sending SIGINT signal to PID" + pythonPid);
Runtime.getRuntime().exec("kill -SIGINT " + pythonPid);
} catch (IOException e) {
e.printStackTrace();
}
}
else {
logger.info("Non UNIX/Linux system, close the interpreter");
close();
}
}
@Override
public FormType getFormType() {
return FormType.SIMPLE;
}
@Override
public int getProgress(InterpreterContext context) {
return 0;
}
@Override
public Scheduler getScheduler() {
return SchedulerFactory.singleton().createOrGetParallelScheduler(
PythonInterpreter.class.getName() + this.hashCode(), 10);
}
@Override
public List<String> completion(String buf, int cursor) {
return null;
}
private Job getRunningJob(String paragraphId) {
Job foundJob = null;
Collection<Job> jobsRunning = getScheduler().getJobsRunning();
for (Job job : jobsRunning) {
if (job.getId().equals(paragraphId)) {
foundJob = job;
}
}
return foundJob;
}
private void bootStrapInterpreter() throws IOException {
BufferedReader bootstrapReader = new BufferedReader(
new InputStreamReader(
PythonInterpreter.class.getResourceAsStream("/bootstrap.py")));
String line = null;
String bootstrapCode = "";
while ((line = bootstrapReader.readLine()) != null)
{
bootstrapCode += line + "\n";
}
logger.info("Bootstrap python interpreter with \n " + bootstrapCode);
writer.write(bootstrapCode);
writer.flush();
}
private long getPidOfProcess(Process p) {
long pid = -1;
try {
if (p.getClass().getName().equals("java.lang.UNIXProcess")) {
Field f = p.getClass().getDeclaredField("pid");
f.setAccessible(true);
pid = f.getLong(p);
f.setAccessible(false);
}
}
catch (Exception e) {
pid = -1;
}
return pid;
}
}

View file

@ -0,0 +1,81 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#PYTHON 2 / 3 comptability :
# bootstrap.py must be runnable with Python 2 and 3
#Remove interactive mode displayhook
import sys
sys.displayhook = lambda x: None
# Set the signal handler
import signal
def intHandler(signum, frame):
print ("Paragraph interrupted")
raise KeyboardInterrupt()
signal.signal(signal.SIGINT, intHandler)
def help():
print ('%html')
print ('<h2>Python Interpreter help</h2>')
print ('<h3>Python 2 & 3 comptability</h3>')
print ('<div>The interpreter is compatible with Python 2 & 3.<br/> To change Python version, change in the interpreter configuration the python.path to the desired version (example : python.path=/usr/bin/python3)</div>')
print ('<h3>Python modules</h3>')
print ('<div>The interpreter can use all modules already installed (with pip, easy_install etc) </div>')
print ('<h3>Forms</h3>')
print ('<h4>Input form</h4>')
print ('<pre> print "&#36{input_form(name)=defaultValue}"</pre>')
print ('<h4>Selection form</h4>')
print ('<pre> print "&#36{select_form(Selection Form)=op1,op1|op2(Option 2)|op3}"</pre>')
print ('<h4>Checkbox form</h4>')
print ('<pre> print "&#36{checkbox:checkbox_form=op1,op1|op2|op3}"</pre>')
print ('<h3>Matplotlib graph</h3>')
print ('<div>The interpreter can display matplotlib graph with the function zeppelin_show()</div>')
print ('<div> You need to already have matplotlib module installed to use this functionality !</div><br/>')
print ('''<pre>import matplotlib.pyplot as plt
plt.figure()
(.. ..)
zeppelin_show(plt)
plt.close()
</pre>''')
print ('<div><br/> zeppelin_show function can take optional parameters to adapt graph width and height</div>')
print ("<div><b>example </b>:")
print('''<pre>zeppelin_show(plt,width='50px')
zeppelin_show(plt,height='150px') </pre></div>''')
#Matplotlib show function
try:
import StringIO as io
except ImportError:
import io as io
def zeppelin_show(p,width="0",height="0"):
img = io.StringIO()
p.savefig(img, format='svg')
img.seek(0)
style=""
if(width!="0"):
style+='width:'+width
if(height!="0"):
if(len(style)!=0):
style+=","
style+='height:'+height
print("%html <div style='"+ style +"'>" + img.read() +"<div>")

View file

@ -482,6 +482,7 @@ public class ZeppelinConfiguration extends XMLConfiguration {
+ "org.apache.zeppelin.postgresql.PostgreSqlInterpreter,"
+ "org.apache.zeppelin.tajo.TajoInterpreter,"
+ "org.apache.zeppelin.flink.FlinkInterpreter,"
+ "org.apache.zeppelin.python.PythonInterpreter,"
+ "org.apache.zeppelin.ignite.IgniteInterpreter,"
+ "org.apache.zeppelin.ignite.IgniteSqlInterpreter,"
+ "org.apache.zeppelin.lens.LensInterpreter,"