mirror of
https://github.com/apache/zeppelin
synced 2026-05-24 09:38:26 +00:00
403c8c4e79
## What is this PR for? The PR is a interpreter for [Apache Beam](http://beam.incubator.apache.org) which is an open source unified platform for data processing pipelines. A pipeline can be build using one of the Beam SDKs. The execution of the pipeline is done by different Runners . Currently, Beam supports Apache Flink Runner, Apache Spark Runner, and Google Dataflow Runner. ### What type of PR is it? - Feature ### Todos * Test case * Review Comments * Documentation ### What is the Jira issue? * [ZEPPELIN-682] ### How should this be tested? - Start the Zeppelin server - The prefix of interpreter is `%beam` and then write your code with required imports and the runner ### Screenshots (if appropriate)   ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: mahmoudelgamal <mahmoudf.elgamal@gmail.com> Author: mfelgamal <mahmoudf.elgamal@gmail.com> Author: Fouad <fuad.assayadi@gmail.com> Closes #1334 from mfelgamal/beam-interpreter-static-repl-7 and squashes the following commits:da66c27[mahmoudelgamal] Modify condition of checking static modifier55c1322[mahmoudelgamal] set spark version to 1.6.2 and throw original exception27d7690[mahmoudelgamal] set spark version to 1.6.1 and some modifications750041c[mahmoudelgamal] Add readme file and modify pom file and travis.ymlca88f94[mahmoudelgamal] edit pom file and .travis.yml3d65427[mahmoudelgamal] update .travis.yml filef19f98d[mahmoudelgamal] Make easy example with imports ands some modifications74c14ca[mahmoudelgamal] Update the licensesacc7afb[mahmoudelgamal] Change beam to version 0.2.0e821614[mahmoudelgamal] Removing hadoop-core and print stack trace to failure5cb7c7b[mahmoudelgamal] Add some changes to doc and pom file75fc4f7[mahmoudelgamal] add interpreter to navigation.html and remove extra spaces and lines9b1b385[mahmoudelgamal] put beam in alphabetical order9c1e25d[mahmoudelgamal] Adding changes like logging and conventions and license2aa6d65[mahmoudelgamal] changing class name to StaticRepl and adding some modifications7cf25fb[mahmoudelgamal] Adding some tests3c5038f[mahmoudelgamal] Modifying the documentation5695077[mahmoudelgamal] Modifying pom file and Making documentation26fc59b[mahmoudelgamal] Refactoring of the code3a2bd85[mahmoudelgamal] Adding the beam to zeppelin 7ab7ee2d[mahmoudelgamal] beam interpreter85957ff[mfelgamal] Merge pull request #10 from apache/master852c3d3[mfelgamal] Merge pull request #9 from apache/mastera4bcc0d[mfelgamal] Merge pull request #8 from apache/master858f1e1[mfelgamal] Merge pull request #7 from apache/master03a1e80[mfelgamal] Merge pull request #4 from apache/master2586651[Fouad] Merge pull request #2 from apache/master
1,015 B
1,015 B
Overview
Beam interpreter for Apache Zeppelin
Architecture
Current interpreter implementation supports the static repl. It compiles the code in memory, execute it and redirect the output to zeppelin.
Building the Beam Interpreter
You have to first build the Beam interpreter by enable the beam profile as follows:
mvn clean package -Pbeam -DskipTests
Notice
- Flink runner comes with binary compiled for scala 2.10. So, currently we support only Scala 2.10
Technical overview
-
Upon starting an interpreter, an instance of
JavaCompileris created. -
When the user runs commands with beam, the
JavaParsergo through the code to get a class that contains the main method. -
Then it replaces the class name with random class name to avoid overriding while compilation. it creates new out & err stream to get the data in new stream instead of the console, to redirect output to zeppelin.
-
If there is any error during compilation, it can catch and redirect to zeppelin.