Elgato_dark/zeppelin
1
0
Fork 0
mirror of https://github.com/apache/zeppelin synced 2026-05-24 09:38:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge pull request #10 from apache/master

Browse source 
merge master
This commit is contained in:
mfelgamal 2016-09-23 14:47:13 +03:00 committed by GitHub
commit 85957ff037
90 changed files with 2062 additions and 1303 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.github/PULL_REQUEST_TEMPLATE vendored
View file

@ -1,6 +1,6 @@
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
### What type of PR is it?

228
CONTRIBUTING.md
View file

@ -1,228 +0,0 @@
# How to contribute
**Zeppelin** is [Apache2 License](https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md) Software.
Contributing to Zeppelin (Source code, Documents, Image, Website) means you agree to the Apache2 License.
1. Make sure your issue is not already in the [Jira issue tracker](https://issues.apache.org/jira/browse/ZEPPELIN)
2. If not, create a ticket describing the change you're proposing in the [Jira issue tracker](https://issues.apache.org/jira/browse/ZEPPELIN)
3. Contribute your patch via Pull Request.
Before you start, please read the [Code of Conduct](http://www.apache.org/foundation/policies/conduct.html) carefully, familiarize yourself with it and refer to it whenever you need it.
For those of you who are not familiar with Apache project, understanding [How it works](http://www.apache.org/foundation/how-it-works.html) would be quite helpful.
## Creating a Pull Request
In order to make the review process easier, please follow this template when making a Pull Request:
```
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md
### What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
Outline the steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
```
## Testing a Pull Request
You can also test and review a particular Pull Request. Here are two useful ways.
* Using a utility provided from Zeppelin.
```
dev/test_zeppelin_pr.py [# of PR]
```
For example, if you want to test `#513`, then the command will be:
```
dev/test_zeppelin_pr.py 513
```
* Another way is using [github/hub](https://github.com/github/hub).
```
hub checkout https://github.com/apache/zeppelin/pull/[# of PR]
```
The above two methods will help you test and review Pull Requests.
## Source Control Workflow
Zeppelin follows [Fork & Pull] (https://github.com/sevntu-checkstyle/sevntu.checkstyle/wiki/Development-workflow-with-Git:-Fork,-Branching,-Commits,-and-Pull-Request) model.
## The Review Process
When a Pull Request is submitted, it is being merged or rejected by following review process.
* Anybody can be a reviewer and may comment on the change and suggest modifications.
* Reviewer can indicate that a patch looks suitable for merging with a comment such as: "Looks good", "LGTM", "+1".
* At least one indication of suitable for merging (e.g. "LGTM") from committer is required to be merged.
* Pull request is open for 1 or 2 days for potential additional review, unless it's got enough indication of suitable for merging.
* Committer can initiate lazy consensus ("Merge if there is no more discussion") and the code can be merged after certain time (normally 24 hours) when there is no review exists.
* Contributor can ping reviewers (including committer) by commenting 'Ready to review' or suitable indication.
## Becoming a Committer
The PPMC adds new committers from the active contributors, based on their contribution to Zeppelin. The qualifications for new committers include:
1. Sustained contributions: Committers should have a history of constant contributions to Zeppelin.
2. Quality of contributions: Committers more than any other community member should submit simple, well-tested, and well-designed patches.
3. Community involvement: Committers should have a constructive and friendly attitude in all community interactions. They should also be active on the dev, user list and reviewing patches. Also help new contributors and users.
## Setting up
Here are some things you will need to build and test Zeppelin.
### Software Configuration Management (SCM)
Zeppelin uses Git for its SCM system. `http://git.apache.org/zeppelin.git` you'll need git client installed in your development machine.
For write access, `https://git-wip-us.apache.org/repos/asf/zeppelin.git`
### Integrated Development Environment (IDE)
You are free to use whatever IDE you prefer, or your favorite command line editor.
### Project Structure
Zeppelin project is based on Maven. Maven works by convention & defines [directory structure] (https://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html) for a project.
The top-level pom.xml describes the basic project structure. Currently Zeppelin has the following modules.
<module>zeppelin-interpreter</module>
<module>zeppelin-zengine</module>
<module>spark</module>
<module>markdown</module>
<module>angular</module>
<module>shell</module>
<module>flink</module>
<module>ignite</module>
<module>lens</module>
<module>cassandra</module>
<module>zeppelin-web</module>
<module>zeppelin-server</module>
<module>zeppelin-distribution</module>
### Web Project Contribution Guidelines
If you plan on making a contribution to Zeppelin's WebApplication,
please check [its own contribution guidelines](https://github.com/apache/zeppelin/blob/master/zeppelin-web/CONTRIBUTING.md)
### Code convention
We are following Google Code style:
* [Java style](https://google.github.io/styleguide/javaguide.html)
* [Shell style](https://google.github.io/styleguide/shell.xml)
Check style report location are in `${submodule}/target/site/checkstyle.html`
Test coverage report location are in `${submodule}/target/site/cobertura/index.html`
#### Build Tools
To build the code, install
* Oracle Java 7
* Apache Maven
## Getting the source code
First of all, you need the Zeppelin source code. The official location for Zeppelin is [http://git.apache.org/zeppelin.git](http://git.apache.org/zeppelin.git).
### git access
Get the source code on your development machine using git.
```
git clone git://git.apache.org/zeppelin.git zeppelin
```
You may also want to develop against a specific branch. For example, for branch-0.5.6
```
git clone -b branch-0.5.6 git://git.apache.org/zeppelin.git zeppelin
```
or with write access
```
git clone https://git-wip-us.apache.org/repos/asf/zeppelin.git
```
### Fork repository
If you want not only build Zeppelin but also make change, then you need fork Zeppelin github mirror repository (https://github.com/apache/zeppelin) and make pull request.
## Build
```
mvn install
```
To skip test
```
mvn install -DskipTests
```
To build with specific spark / hadoop version
```
mvn install -Phadoop-2.2 -Dhadoop.version=2.2.0 -Pspark-1.3 -Dspark.version=1.3.0
```
## Tests
Each new File should have its own accompanying unit tests. Each new interpreter should have come with its tests.
Zeppelin has 3 types of tests:
1. Unit Tests: The unit tests run as part of each package's build. E.g. SparkInterpeter Module's unit test is SparkInterpreterTest
2. Integration Tests: The integration tests run after all modules are build. The integration tests launch an instance of Zeppelin server. ZeppelinRestApiTest is an example integration test.
3. GUI integration tests: These tests validate the Zeppelin UI elements. These tests require a running Zeppelin server and launches a web browser to validate Notebook UI elements like Notes and their execution. See ZeppelinIT as an example.
Currently the GUI integration tests are not run in the Maven and are only run in the CI environment when the pull request is submitted to github. Make sure to watch the [CI results] (https://travis-ci.org/apache/zeppelin/pull_requests) for your pull request.
## Continuous Integration
Zeppelin uses Travis for CI. In the project root there is .travis.yml that configures CI and [publishes CI results] (https://travis-ci.org/apache/zeppelin/builds)
## Run Zeppelin server in development mode
```
cd zeppelin-server
HADOOP_HOME=YOUR_HADOOP_HOME JAVA_HOME=YOUR_JAVA_HOME mvn exec:java -Dexec.mainClass="org.apache.zeppelin.server.ZeppelinServer" -Dexec.args=""
```
or use daemon script
```
bin/zeppelin-daemon start
```
Server will be run on http://localhost:8080
## JIRA
Zeppelin manages it's issues in Jira. [https://issues.apache.org/jira/browse/ZEPPELIN](https://issues.apache.org/jira/browse/ZEPPELIN)
## Where to Start
You can find issues for [beginner](https://issues.apache.org/jira/issues/?jql=(labels%20%3D%20newbie%20or%20labels%3D%20beginner)%20and%20project%3DZeppelin)
## Stay involved
Everyone is welcome to join our mailing list:
* [users@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/zeppelin-users/) is for usage questions, help, and announcements [ [subscribe](mailto:users-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:users-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/zeppelin-users/) ]
* [dev@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/zeppelin-users/) is for people who want to contribute code to Zeppelin.[ [subscribe](mailto:dev-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:dev-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/zeppelin-dev/) ]
* [commits@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/zeppelin-commits/) is for commit messages and patches to Zeppelin. [ [subscribe](mailto:commits-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:commits-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/zeppelin-commits/) ]

10
README.md
View file

@ -3,7 +3,7 @@
**Documentation:** [User Guide](http://zeppelin.apache.org/docs/latest/index.html)<br/>
**Mailing Lists:** [User and Dev mailing list](http://zeppelin.apache.org/community.html)<br/>
**Continuous Integration:** [![Build Status](https://secure.travis-ci.org/apache/zeppelin.png?branch=master)](https://travis-ci.org/apache/zeppelin) <br/>
**Contributing:** [Contribution Guide](https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md)<br/>
**Contributing:** [Contribution Guide](https://zeppelin.apache.org/contribution/contributions.html)<br/>
**Issue Tracker:** [Jira](https://issues.apache.org/jira/browse/ZEPPELIN)<br/>
**License:** [Apache 2.0](https://github.com/apache/zeppelin/blob/master/LICENSE)
@ -18,7 +18,7 @@ Core feature:
To know more about Zeppelin, visit our web site [http://zeppelin.apache.org](http://zeppelin.apache.org)
## Requirements
* Git
* Git
* Java 1.7
* Tested on Mac OSX, Ubuntu 14.X, CentOS 6.X, Windows 7 Pro SP1
* Maven (if you want to build from the source code)
@ -67,7 +67,7 @@ First of all, set your proxy configuration on Maven `settings.xml`.
</settings>
```
Then, run these commands from shell.
Then, run these commands from shell.
```
npm config set proxy http://localhost:3128
npm config set https-proxy http://localhost:3128
@ -87,10 +87,10 @@ git config --global --unset https.proxy
git config --global --unset url."http://".insteadOf
```
_Notes:_
_Notes:_
- If you are behind NTLM proxy you can use [Cntlm Authentication Proxy](http://cntlm.sourceforge.net/).
- Replace `localhost:3128` with the standard pattern `http://user:pwd@host:port`.
#### Install maven
```
wget http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

3
bigquery/src/main/resources/interpreter-setting.json
View file

@ -22,6 +22,9 @@
"defaultValue": "100000",
"description": "Maximum number of rows to fetch from BigQuery"
}
},
"editor": {
"language": "sql"
}
}
]

5
bin/zeppelin-daemon.sh
View file

@ -161,6 +161,8 @@ function upstart() {
# where the service manager starts and stops the process
initialize_default_directories
echo "ZEPPELIN_CLASSPATH: ${ZEPPELIN_CLASSPATH_OVERRIDES}:${CLASSPATH}" >> "${ZEPPELIN_OUTFILE}"
$ZEPPELIN_RUNNER $JAVA_OPTS -cp $ZEPPELIN_CLASSPATH_OVERRIDES:$CLASSPATH $ZEPPELIN_MAIN >> "${ZEPPELIN_OUTFILE}"
}
@ -177,6 +179,8 @@ function start() {
initialize_default_directories
echo "ZEPPELIN_CLASSPATH: ${ZEPPELIN_CLASSPATH_OVERRIDES}:${CLASSPATH}" >> "${ZEPPELIN_OUTFILE}"
nohup nice -n $ZEPPELIN_NICENESS $ZEPPELIN_RUNNER $JAVA_OPTS -cp $ZEPPELIN_CLASSPATH_OVERRIDES:$CLASSPATH $ZEPPELIN_MAIN >> "${ZEPPELIN_OUTFILE}" 2>&1 < /dev/null &
pid=$!
if [[ -z "${pid}" ]]; then
@ -254,6 +258,7 @@ case "${1}" in
start
;;
restart)
echo "${ZEPPELIN_NAME} is restarting" >> "${ZEPPELIN_OUTFILE}"
stop
start
;;

94
docs/CONTRIBUTING.md
View file

@ -1,94 +0,0 @@
# Contributing to Apache Zeppelin Documentation
## Folder Structure
`docs/` folder is organized as below:
```
docs/
├── _includes/themes/zeppelin
│ ├── _navigation.html
│ └── default.html
├── _layouts
├── _plugins
├── assets/themes/zeppelin -> {ASSET_PATH}
│ ├── bootstrap
│ ├── css
│ ├── img
│ └── js
├── development/ *.md
├── displaysystem/ *.md
├── install/ *.md
├── interpreter/ *.md
├── manual/ *.md
├── quickstart/ *.md
├── rest-api/ *.md
├── security/ *.md
├── storage/ *.md
├── Gemfile
├── Gemfile.lock
├── _config.yml
├── index.md
└── ...
```
- `_navigation.html`: the dropdown menu in navbar
- `default.html` & `_layouts/`: define default HTML layout
- `_plugins/`: custom plugin `*.rb` files can be placed in this folder. See [jekyll/plugins](https://jekyllrb.com/docs/plugins/) for the further information.
- `{ASSET_PATH}/css/style.css`: extra css components can be defined
- `{ASSET_PATH}/img/docs-img/`: image files used for document pages can be placed in this folder
- `{ASSET_PATH}/js/`: extra `.js` files can be placed
- `Gemfile`: defines bundle dependencies. They will be installed by `bundle install`.
- `Gemfile.lock`: when you run `bundle install`, bundler will persist all gems name and their version to this file. For the more details, see [Bundle "The Gemfile Lock"](http://bundler.io/v1.10/man/bundle-install.1.html#THE-GEMFILE-LOCK)
- `documentation_group`: `development/`, `displaysystem/`, `install/`, `interpreter/`...
- `_config.yml`: defines configuration options for docs website. See [jekyll/configuration](https://jekyllrb.com/docs/configuration/) for the other available config variables.
- `index.md`: the main page of `http://zeppelin.apache.org/docs/<ZEPPELIN_VERSION>/`
## Markdown
Zeppelin documentation pages are written with [Markdown](http://daringfireball.net/projects/markdown/). It is possible to use [GitHub flavored syntax](https://help.github.com/categories/writing-on-github/) and intermix plain HTML.
## Front matter
Every page contains [YAML front matter](https://jekyllrb.com/docs/frontmatter/) block in their header. Don't forget to wrap the front matter list with triple-dashed lines(`---`) like below.
The document page should start this triple-dashed lines. Or you will face 404 error, since Jekyll can't find the page.
```
---
layout: page
title: "Apache Zeppelin Tutorial"
description: "This tutorial page contains a short walk-through tutorial that uses Apache Spark backend. Please note that this tutorial is valid for Spark 1.3 and higher."
group: quickstart
---
```
- `layout`: the default layout is `page` which is defined in `_layout/page.html`.
- `title`: the title for the document. Please note that if it needs to include `Zeppelin`, it should be `Apache Zeppelin`, not `Zeppelin`.
- `description`: a short description for the document. One or two sentences would be enough. This description also will be shown as an extract sentence when people search pages.
- `group`: a category of the document page
## Headings
All documents are structured with headings. From these headings, you can automatically generate a **Table of Contents**. There is a simple rule for Zeppelin docs headings.
```
# Level-1 heading <- used only for the main title
## Level-2 heading <- start with this
### Level-3 heading
#### Level-4 heading <- won't be converted in TOC from this level
```
## Table of contents(TOC)
```
<div id="toc"></div>
```
Add this line below `# main title` in order to generate a **Table of Contents**. Headings until `### (Level-3 heading)` are included to TOC.
Default setting options for TOC are definded in [here](https://github.com/apache/zeppelin/blob/master/docs/assets/themes/zeppelin/js/toc.js#L4).
## Adding new pages
If you're going to create new pages, there are some spots you need to add the location of the page.
- **Dropdown menu in navbar**: add your docs location to [_navigation.html](https://github.com/apache/zeppelin/blob/master/docs/_includes/themes/zeppelin/_navigation.html)
- **Main index**: add your docs below [What is the next?](http://zeppelin.apache.org/docs/latest/#what-is-the-next) section in [index.md](https://github.com/apache/zeppelin/blob/master/docs/index.md) with a short description. No need to do this if the page is for **Interpreters**.

8
docs/README.md
View file

@ -1,6 +1,6 @@
# Apache Zeppelin documentation
This README will walk you through building the documentation of Apache Zeppelin. The documentation is included here with Apache Zeppelin source code. The online documentation at [https://zeppelin.apache.org/docs/<ZEPPELIN_VERSION>](https://zeppelin.apache.org/docs/latest) is also generated from the files found in here.
This README will walk you through building the documentation of Apache Zeppelin. The documentation is included here with Apache Zeppelin source code. The online documentation at [https://zeppelin.apache.org/docs/<ZEPPELIN_VERSION>](https://zeppelin.apache.org/docs/latest/) is also generated from the files found in here.
## Build documentation
Zeppelin is using [Jekyll](https://jekyllrb.com/) which is a static site generator and [Github Pages](https://pages.github.com/) as a site publisher. For the more details, see [help.github.com/articles/about-github-pages-and-jekyll/](https://help.github.com/articles/about-github-pages-and-jekyll/).
@ -19,7 +19,7 @@ bundle install
For the further information about requirements, please see [here](https://help.github.com/articles/setting-up-your-github-pages-site-locally-with-jekyll/#requirements).
On OS X 10.9, you may need to do
On OS X 10.9, you may need to do
```
xcode-select --install
@ -39,7 +39,7 @@ Using the above command, Jekyll will start a web server at `http://localhost:400
## Contribute to Zeppelin documentation
If you wish to help us and contribute to Zeppelin Documentation, please look at [Zeppelin Documentation's contribution guideline](https://github.com/apache/zeppelin/blob/master/docs/CONTRIBUTING.md).
If you wish to help us and contribute to Zeppelin Documentation, please look at [Zeppelin Documentation's contribution guideline](https://zeppelin.apache.org/contribution/contributions.html).
## For committers only
@ -49,7 +49,7 @@ If you wish to help us and contribute to Zeppelin Documentation, please look at
### Deploy to ASF svnpubsub infra
1. generate static website in `./_site`
```
# go to /docs under Zeppelin source
bundle exec jekyll build --safe

BIN
docs/assets/themes/zeppelin/img/docs-img/markdown-example-markdown4j-parser.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 258 KiB

BIN
docs/assets/themes/zeppelin/img/docs-img/markdown-example-pegdown-parser.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 144 KiB

BIN
docs/assets/themes/zeppelin/img/docs-img/markdown-interpreter-setting.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 32 KiB

After

Width:  |  Height:  |  Size: 66 KiB

27
docs/development/writingzeppelininterpreter.md
View file

@ -42,13 +42,13 @@ In 'Separate Interpreter(scoped / isolated) for each note' mode which you can se
Creating a new interpreter is quite simple. Just extend [org.apache.zeppelin.interpreter](https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java) abstract class and implement some methods.
You can include `org.apache.zeppelin:zeppelin-interpreter:[VERSION]` artifact in your build system. And you should put your jars under your interpreter directory with a specific directory name. Zeppelin server reads interpreter directories recursively and initializes interpreters including your own interpreter.
There are three locations where you can store your interpreter group, name and other information. Zeppelin server tries to find the location below. Next, Zeppelin tries to find `interpreter-setting.json` in your interpreter jar.
There are three locations where you can store your interpreter group, name and other information. Zeppelin server tries to find the location below. Next, Zeppelin tries to find `interpreter-setting.json` in your interpreter jar.
```
{ZEPPELIN_INTERPRETER_DIR}/{YOUR_OWN_INTERPRETER_DIR}/interpreter-setting.json
```
Here is an example of `interpreter-setting.json` on your own interpreter.
Here is an example of `interpreter-setting.json` on your own interpreter. Note that if you don't specify editor object, your interpreter will use plain text mode for syntax highlighting.
```json
[
@ -69,11 +69,14 @@ Here is an example of `interpreter-setting.json` on your own interpreter.
"defaultValue": "property2DefaultValue",
"description": "Property 2 description"
}, ...
},
"editor": {
"language": "your-syntax-highlight-language"
}
},
{
...
}
}
]
```
@ -81,8 +84,8 @@ Finally, Zeppelin uses static initialization with the following:
```
static {
Interpreter.register("MyInterpreterName", MyClassName.class.getName());
}
Interpreter.register("MyInterpreterName", MyClassName.class.getName());
}
```
**Static initialization is deprecated and will be supported until 0.6.0.**
@ -96,15 +99,20 @@ some interpreter specific code...
```
## Programming Languages for Interpreter
If the interpreter uses a specific programming language ( like Scala, Python, SQL ), it is generally recommended to add a syntax highlighting supported for that to the notebook paragraph editor.
If the interpreter uses a specific programming language (like Scala, Python, SQL), it is generally recommended to add a syntax highlighting supported for that to the notebook paragraph editor.
To check out the list of languages supported, see the `mode-*.js` files under `zeppelin-web/bower_components/ace-builds/src-noconflict` or from [github.com/ajaxorg/ace-builds](https://github.com/ajaxorg/ace-builds/tree/master/src-noconflict).
If you want to add a new set of syntax highlighting,
1. Add the `mode-*.js` file to <code>[zeppelin-web/bower.json](https://github.com/apache/zeppelin/blob/master/zeppelin-web/bower.json)</code> ( when built, <code>[zeppelin-web/src/index.html](https://github.com/apache/zeppelin/blob/master/zeppelin-web/src/index.html)</code> will be changed automatically. ).
2. Add to the list of `editorMode` in <code>[zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js](https://github.com/apache/zeppelin/blob/master/zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js)</code> - it follows the pattern 'ace/mode/x' where x is the name.
3. Add to the code that checks for `%` prefix and calls `session.setMode(editorMode.x)` in `setParagraphMode` located in <code>[zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js](https://github.com/apache/zeppelin/blob/master/zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js)</code>.
2. Add `editor` object to `interpreter-setting.json` file. If you want to set your language to `java` for example, add:
```
"editor": {
"language": "java"
}
```
## Install your interpreter binary
@ -207,7 +215,7 @@ Checkout some interpreters released with Zeppelin by default.
We welcome contribution to a new interpreter. Please follow these few steps:
- First, check out the general contribution guide [here](https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md).
- First, check out the general contribution guide [here](https://zeppelin.apache.org/contribution/contributions.html).
- Follow the steps in [Make your own Interpreter](#make-your-own-interpreter) section above.
- Add your interpreter as in the [Configure your interpreter](#configure-your-interpreter) section above; also add it to the example template [zeppelin-site.xml.template](https://github.com/apache/zeppelin/blob/master/conf/zeppelin-site.xml.template).
- Add tests! They are run by [Travis](https://travis-ci.org/apache/zeppelin) for all changes and it is important that they are self-contained.
@ -215,4 +223,3 @@ We welcome contribution to a new interpreter. Please follow these few steps:
- Add documentation on how to use your interpreter under `docs/interpreter/`. Follow the Markdown style as this [example](https://github.com/apache/zeppelin/blob/master/docs/interpreter/elasticsearch.md). Make sure you list config settings and provide working examples on using your interpreter in code boxes in Markdown. Link to images as appropriate (images should go to `docs/assets/themes/zeppelin/img/docs-img/`). And add a link to your documentation in the navigation menu (`docs/_includes/themes/zeppelin/_navigation.html`).
- Most importantly, ensure licenses of the transitive closure of all dependencies are list in [license file](https://github.com/apache/zeppelin/blob/master/zeppelin-distribution/src/bin_license/LICENSE).
- Commit your changes and open a [Pull Request](https://github.com/apache/zeppelin/pulls) on the project [Mirror on GitHub](https://github.com/apache/zeppelin); check to make sure Travis CI build is passing.

5
docs/index.md
View file

@ -115,7 +115,7 @@ If you want to learn more about this feature, please visit [this page](./manual/
<img class="img-responsive" style="margin:0 auto; padding: 15px;" src="./assets/themes/zeppelin/img/asf_logo.png" width="250px"/>
Apache Zeppelin is Apache2 Licensed software. Please check out the [source repository](http://git.apache.org/zeppelin.git) and [how to contribute](./development/howtocontribute.html).
Apache Zeppelin is Apache2 Licensed software. Please check out the [source repository](http://git.apache.org/zeppelin.git) and [how to contribute](https://zeppelin.apache.org/contribution/contributions.html).
Apache Zeppelin has a very active development community.
Join to our [Mailing list](https://zeppelin.apache.org/community.html) and report issues on [Jira Issue tracker](https://issues.apache.org/jira/browse/ZEPPELIN).
@ -178,8 +178,7 @@ Join to our [Mailing list](https://zeppelin.apache.org/community.html) and repor
* [How to contribute (code)](./development/howtocontribute.html)
* [How to contribute (documentation website)](./development/howtocontributewebsite.html)
#### External Resources
#### External Resources
* [Mailing List](https://zeppelin.apache.org/community.html)
* [Apache Zeppelin Wiki](https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Home)
* [StackOverflow tag `apache-zeppelin`](http://stackoverflow.com/questions/tagged/apache-zeppelin)

2
docs/interpreter/elasticsearch.md
View file

@ -243,7 +243,7 @@ delete /index/type/id
```
### Apply Zeppelin Dynamic Forms
You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html) inside your queries. You can use both the `text input` and `select form` parameterization features.
You can leverage [Zeppelin Dynamic Form](../manual/dynamicform.html) inside your queries. You can use both the `text input` and `select form` parameterization features.
```bash
%elasticsearch

2
docs/interpreter/hive.md
View file

@ -151,7 +151,7 @@ select * from my_table;
You can also run multiple queries up to 10 by default. Changing these settings is not implemented yet.
### Apply Zeppelin Dynamic Forms
You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html) inside your queries. You can use both the `text input` and `select form` parameterization features.
You can leverage [Zeppelin Dynamic Form](../manual/dynamicform.html) inside your queries. You can use both the `text input` and `select form` parameterization features.
```sql
%hive

2
docs/interpreter/livy.md
View file

@ -174,7 +174,7 @@ When Zeppelin server is running with authentication enabled, then this interpret
## Apply Zeppelin Dynamic Forms
You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html). You can use both the `text input` and `select form` parameterization features.
You can leverage [Zeppelin Dynamic Form](../manual/dynamicform.html). You can use both the `text input` and `select form` parameterization features.
```
%livy.pyspark

36
docs/interpreter/markdown.md
View file

@ -25,14 +25,42 @@ limitations under the License.
## Overview
[Markdown](http://daringfireball.net/projects/markdown/) is a plain text formatting syntax designed so that it can be converted to HTML.
Apache Zeppelin uses markdown4j. For more examples and extension support, please checkout [here](https://code.google.com/p/markdown4j/).
Apache Zeppelin uses [markdown4j](https://github.com/jdcasey/markdown4j) and [pegdown](https://github.com/sirthias/pegdown) as markdown parsers.
In Zeppelin notebook, you can use ` %md ` in the beginning of a paragraph to invoke the Markdown interpreter and generate static html from Markdown plain text.
In Zeppelin, Markdown interpreter is enabled by default.
In Zeppelin, Markdown interpreter is enabled by default and uses the [markdown4j](https://github.com/jdcasey/markdown4j) parser.
<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/markdown-interpreter-setting.png" width="60%" />
<img src="../assets/themes/zeppelin/img/docs-img/markdown-interpreter-setting.png" width="60%" />
## Configuration
<table class="table-configuration">
<tr>
<th>Name</th>
<th>Default Value</th>
<th>Description</th>
</tr>
<tr>
<td>markdown.parser.type</td>
<td>markdown4j</td>
<td>Markdown Parser Type. <br/> Available values: markdown4j, pegdown.</td>
</tr>
</table>
## Example
The following example demonstrates the basic usage of Markdown in a Zeppelin notebook.
<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/markdown-example.png" width="70%" />
<img src="../assets/themes/zeppelin/img/docs-img/markdown-example.png" width="70%" />
### Markdown4j Parser
`markdown4j` parser provides [YUML](http://yuml.me/) and [Websequence](https://www.websequencediagrams.com/) extensions
<img src="../assets/themes/zeppelin/img/docs-img/markdown-example-markdown4j-parser.png" width="70%" />
### Pegdown Parser
`pegdown` parser provides github flavored markdown.
<img src="../assets/themes/zeppelin/img/docs-img/markdown-example-pegdown-parser.png" width="70%" />

4
docs/interpreter/python.md
View file

@ -108,7 +108,7 @@ z.show(plt, height='150px', fmt='svg')
## Pandas integration
Apache Zeppelin [Table Display System]({{BASE_PATH}}/displaysystem/basicdisplaysystem.html#table) provides built-in data visualization capabilities. Python interpreter leverages it to visualize Pandas DataFrames though similar `z.show()` API, same as with [Matplotlib integration](#matplotlib-integration).
Apache Zeppelin [Table Display System](../displaysystem/basicdisplaysystem.html#table) provides built-in data visualization capabilities. Python interpreter leverages it to visualize Pandas DataFrames though similar `z.show()` API, same as with [Matplotlib integration](#matplotlib-integration).
Example:
@ -120,7 +120,7 @@ z.show(rates)
## SQL over Pandas DataFrames
There is a convenience `%python.sql` interpreter that matches Apache Spark experience in Zeppelin and enables usage of SQL language to query [Pandas DataFrames](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) and visualization of results though built-in [Table Display System]({{BASE_PATH}}/displaysystem/basicdisplaysystem.html#table).
There is a convenience `%python.sql` interpreter that matches Apache Spark experience in Zeppelin and enables usage of SQL language to query [Pandas DataFrames](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) and visualization of results though built-in [Table Display System](../displaysystem/basicdisplaysystem.html#table).
**Pre-requests**

2
docs/interpreter/shell.md
View file

@ -63,6 +63,6 @@ At the "Interpreters" menu in Zeppelin dropdown menu, you can set the property v
## Example
The following example demonstrates the basic usage of Shell in a Zeppelin notebook.
<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/shell-example.png" />
<img src="../assets/themes/zeppelin/img/docs-img/shell-example.png" />
If you need further information about **Zeppelin Interpreter Setting** for using Shell interpreter, please read [What is interpreter setting?](../manual/interpreters.html#what-is-interpreter-setting) section first.

105
docs/interpreter/spark.md
View file

@ -1,7 +1,7 @@
---
layout: page
title: "Apache Spark Interpreter for Apache Zeppelin"
description: "Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs."
description: "Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution engine."
group: interpreter
---
<!--
@ -25,9 +25,8 @@ limitations under the License.
## Overview
[Apache Spark](http://spark.apache.org) is a fast and general-purpose cluster computing system.
It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs
Apache Spark is supported in Zeppelin with
Spark Interpreter group, which consists of five interpreters.
It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.
Apache Spark is supported in Zeppelin with Spark interpreter group which consists of below five interpreters.
<table class="table-configuration">
<tr>
@ -38,25 +37,25 @@ Spark Interpreter group, which consists of five interpreters.
<tr>
<td>%spark</td>
<td>SparkInterpreter</td>
<td>Creates a SparkContext and provides a scala environment</td>
<td>Creates a SparkContext and provides a Scala environment</td>
</tr>
<tr>
<td>%pyspark</td>
<td>%spark.pyspark</td>
<td>PySparkInterpreter</td>
<td>Provides a python environment</td>
<td>Provides a Python environment</td>
</tr>
<tr>
<td>%r</td>
<td>%spark.r</td>
<td>SparkRInterpreter</td>
<td>Provides an R environment with SparkR support</td>
</tr>
<tr>
<td>%sql</td>
<td>%spark.sql</td>
<td>SparkSQLInterpreter</td>
<td>Provides a SQL environment</td>
</tr>
<tr>
<td>%dep</td>
<td>%spark.dep</td>
<td>DepInterpreter</td>
<td>Dependency loader</td>
</tr>
@ -139,111 +138,113 @@ You can also set other Spark properties which are not listed in the table. For a
Without any configuration, Spark interpreter works out of box in local mode. But if you want to connect to your Spark cluster, you'll need to follow below two simple steps.
### 1. Export SPARK_HOME
In **conf/zeppelin-env.sh**, export `SPARK_HOME` environment variable with your Spark installation path.
In `conf/zeppelin-env.sh`, export `SPARK_HOME` environment variable with your Spark installation path.
for example
For example,
```bash
export SPARK_HOME=/usr/lib/spark
```
You can optionally export HADOOP\_CONF\_DIR and SPARK\_SUBMIT\_OPTIONS
You can optionally export `HADOOP_CONF_DIR` and `SPARK_SUBMIT_OPTIONS`
```bash
export HADOOP_CONF_DIR=/usr/lib/hadoop
export SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0"
```
For Windows, ensure you have `winutils.exe` in `%HADOOP_HOME%\bin`. For more details please see [Problems running Hadoop on Windows](https://wiki.apache.org/hadoop/WindowsProblems)
For Windows, ensure you have `winutils.exe` in `%HADOOP_HOME%\bin`. Please see [Problems running Hadoop on Windows](https://wiki.apache.org/hadoop/WindowsProblems) for the details.
### 2. Set master in Interpreter menu
After start Zeppelin, go to **Interpreter** menu and edit **master** property in your Spark interpreter setting. The value may vary depending on your Spark cluster deployment type.
for example,
For example,
* **local[*]** in local mode
* **spark://master:7077** in standalone cluster
* **yarn-client** in Yarn client mode
* **mesos://host:5050** in Mesos cluster
That's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding Zeppelin in this way. (Zeppelin 0.5.6-incubating release works up to Spark 1.6.1 )
That's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding Zeppelin in this way.
For the further information about Spark & Zeppelin version compatibility, please refer to "Available Interpreters" section in [Zeppelin download page](https://zeppelin.apache.org/download.html).
> Note that without exporting `SPARK_HOME`, it's running in local mode with included version of Spark. The included version may vary depending on the build profile.
## SparkContext, SQLContext, ZeppelinContext
SparkContext, SQLContext, ZeppelinContext are automatically created and exposed as variable names 'sc', 'sqlContext' and 'z', respectively, both in scala and python environments.
## SparkContext, SQLContext, SparkSession, ZeppelinContext
SparkContext, SQLContext and ZeppelinContext are automatically created and exposed as variable names `sc`, `sqlContext` and `z`, respectively, in Scala, Python and R environments.
Staring from 0.6.1 SparkSession is available as variable `spark` when you are using Spark 2.x.
> Note that scala / python environment shares the same SparkContext, SQLContext, ZeppelinContext instance.
> Note that Scala/Python/R environment shares the same SparkContext, SQLContext and ZeppelinContext instance.
<a name="dependencyloading"> </a>
## Dependency Management
There are two ways to load external library in spark interpreter. First is using Interpreter setting menu and second is loading Spark properties.
There are two ways to load external libraries in Spark interpreter. First is using interpreter setting menu and second is loading Spark properties.
### 1. Setting Dependencies via Interpreter Setting
Please see [Dependency Management](../manual/dependencymanagement.html) for the details.
### 2. Loading Spark Properties
Once `SPARK_HOME` is set in `conf/zeppelin-env.sh`, Zeppelin uses `spark-submit` as spark interpreter runner. `spark-submit` supports two ways to load configurations. The first is command line options such as --master and Zeppelin can pass these options to `spark-submit` by exporting `SPARK_SUBMIT_OPTIONS` in conf/zeppelin-env.sh. Second is reading configuration options from `SPARK_HOME/conf/spark-defaults.conf`. Spark properites that user can set to distribute libraries are:
Once `SPARK_HOME` is set in `conf/zeppelin-env.sh`, Zeppelin uses `spark-submit` as spark interpreter runner. `spark-submit` supports two ways to load configurations.
The first is command line options such as --master and Zeppelin can pass these options to `spark-submit` by exporting `SPARK_SUBMIT_OPTIONS` in `conf/zeppelin-env.sh`. Second is reading configuration options from `SPARK_HOME/conf/spark-defaults.conf`. Spark properties that user can set to distribute libraries are:
<table class="table-configuration">
<tr>
<th>spark-defaults.conf</th>
<th>SPARK_SUBMIT_OPTIONS</th>
<th>Applicable Interpreter</th>
<th>Description</th>
</tr>
<tr>
<td>spark.jars</td>
<td>--jars</td>
<td>%spark</td>
<td>Comma-separated list of local jars to include on the driver and executor classpaths.</td>
</tr>
<tr>
<td>spark.jars.packages</td>
<td>--packages</td>
<td>%spark</td>
<td>Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version.</td>
<td>Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be <code>groupId:artifactId:version</code>.</td>
</tr>
<tr>
<td>spark.files</td>
<td>--files</td>
<td>%pyspark</td>
<td>Comma-separated list of files to be placed in the working directory of each executor.</td>
</tr>
</table>
> Note that adding jar to pyspark is only availabe via `%dep` interpreter at the moment.
Here are few examples:
* SPARK\_SUBMIT\_OPTIONS in conf/zeppelin-env.sh
* `SPARK_SUBMIT_OPTIONS` in `conf/zeppelin-env.sh`
```bash
export SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0 --jars /path/mylib1.jar,/path/mylib2.jar --files /path/mylib1.py,/path/mylib2.zip,/path/mylib3.egg"
```
* `SPARK_HOME/conf/spark-defaults.conf`
* SPARK_HOME/conf/spark-defaults.conf
```
spark.jars /path/mylib1.jar,/path/mylib2.jar
spark.jars.packages com.databricks:spark-csv_2.10:1.2.0
spark.files /path/mylib1.py,/path/mylib2.egg,/path/mylib3.zip
```
### 3. Dynamic Dependency Loading via %dep interpreter
> Note: `%dep` interpreter is deprecated since v0.6.0.
`%dep` interpreter load libraries to `%spark` and `%pyspark` but not to `%spark.sql` interpreter so we recommend you to use first option instead.
### 3. Dynamic Dependency Loading via %spark.dep interpreter
> Note: `%spark.dep` interpreter is deprecated since v0.6.0.
`%spark.dep` interpreter loads libraries to `%spark` and `%spark.pyspark` but not to `%spark.sql` interpreter. So we recommend you to use the first option instead.
When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs using `%dep` interpreter.
When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs using `%spark.dep` interpreter.
* Load libraries recursively from Maven repository
* Load libraries recursively from maven repository
* Load libraries from local filesystem
* Add additional maven repository
* Automatically add libraries to SparkCluster (You can turn off)
Dep interpreter leverages scala environment. So you can write any Scala code here.
Note that `%dep` interpreter should be used before `%spark`, `%pyspark`, `%sql`.
Dep interpreter leverages Scala environment. So you can write any Scala code here.
Note that `%spark.dep` interpreter should be used before `%spark`, `%spark.pyspark`, `%spark.sql`.
Here's usages.
```scala
%dep
%spark.dep
z.reset() // clean up previously added artifact and repository
// add maven repository
@ -277,11 +278,11 @@ z.load("groupId:artifactId:version").local()
```
## ZeppelinContext
Zeppelin automatically injects ZeppelinContext as variable 'z' in your scala/python environment. ZeppelinContext provides some additional functions and utility.
Zeppelin automatically injects `ZeppelinContext` as variable `z` in your Scala/Python environment. `ZeppelinContext` provides some additional functions and utilities.
### Object Exchange
ZeppelinContext extends map and it's shared between scala, python environment.
So you can put some object from scala and read it from python, vise versa.
`ZeppelinContext` extends map and it's shared between Scala and Python environment.
So you can put some objects from Scala and read it from Python, vice versa.
<div class="codetabs">
<div data-lang="scala" markdown="1">
@ -298,7 +299,7 @@ z.put("objName", myObject)
{% highlight python %}
# Get object from python
%pyspark
%spark.pyspark
myObject = z.get("objName")
{% endhighlight %}
@ -307,8 +308,8 @@ myObject = z.get("objName")
### Form Creation
ZeppelinContext provides functions for creating forms.
In scala and python environments, you can create forms programmatically.
`ZeppelinContext` provides functions for creating forms.
In Scala and Python environments, you can create forms programmatically.
<div class="codetabs">
<div data-lang="scala" markdown="1">
@ -333,7 +334,7 @@ z.select("formName", "option1", Seq(("option1", "option1DisplayName"),
<div data-lang="python" markdown="1">
{% highlight python %}
%pyspark
%spark.pyspark
# Create text input form
z.input("formName")
@ -354,8 +355,8 @@ z.select("formName", [("option1", "option1DisplayName"),
In sql environment, you can create form in simple template.
```
%sql
```sql
%spark.sql
select * from ${table=defaultTableName} where text like '%${search}%'
```
@ -364,7 +365,7 @@ To learn more about dynamic form, checkout [Dynamic Form](../manual/dynamicform.
## Interpreter setting option
Interpreter setting can choose one of 'shared', 'scoped', 'isolated' option. Spark interpreter creates separate scala compiler per each notebook but share a single SparkContext in 'scoped' mode (experimental). It creates separate SparkContext per each notebook in 'isolated' mode.
You can choose one of `shared`, `scoped` and `isolated` options wheh you configure Spark interpreter. Spark interpreter creates separated Scala compiler per each notebook but share a single SparkContext in `scoped` mode (experimental). It creates separated SparkContext per each notebook in `isolated` mode.
## Setting up Zeppelin with Kerberos
@ -377,14 +378,14 @@ Logical setup with Zeppelin, Kerberos Key Distribution Center (KDC), and Spark o
1. On the server that Zeppelin is installed, install Kerberos client modules and configuration, krb5.conf.
This is to make the server communicate with KDC.
2. Set SPARK\_HOME in `[ZEPPELIN\_HOME]/conf/zeppelin-env.sh` to use spark-submit
(Additionally, you might have to set `export HADOOP\_CONF\_DIR=/etc/hadoop/conf`)
2. Set `SPARK_HOME` in `[ZEPPELIN_HOME]/conf/zeppelin-env.sh` to use spark-submit
(Additionally, you might have to set `export HADOOP_CONF_DIR=/etc/hadoop/conf`)
3. Add the two properties below to spark configuration (`[SPARK_HOME]/conf/spark-defaults.conf`):
3. Add the two properties below to Spark configuration (`[SPARK_HOME]/conf/spark-defaults.conf`):
spark.yarn.principal
spark.yarn.keytab
> **NOTE:** If you do not have access to the above spark-defaults.conf file, optionally, you may add the lines to the Spark Interpreter through the Interpreter tab in the Zeppelin UI.
> **NOTE:** If you do not have permission to access for the above spark-defaults.conf file, optionally, you can add the above lines to the Spark Interpreter setting through the Interpreter tab in the Zeppelin UI.
4. That's it. Play with Zeppelin!

12
docs/manual/dependencymanagement.md
View file

@ -33,8 +33,8 @@ When your code requires external library, instead of doing download/copy/restart
<hr>
<div class="row">
<div class="col-md-6">
<a data-lightbox="compiler" href="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/interpreter-dependency-loading.png">
<img class="img-responsive" src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/interpreter-dependency-loading.png" />
<a data-lightbox="compiler" href="../assets/themes/zeppelin/img/docs-img/interpreter-dependency-loading.png">
<img class="img-responsive" src="../assets/themes/zeppelin/img/docs-img/interpreter-dependency-loading.png" />
</a>
</div>
<div class="col-md-6" style="padding-top:30px">
@ -52,11 +52,11 @@ When your code requires external library, instead of doing download/copy/restart
<hr>
<div class="row">
<div class="col-md-6">
<a data-lightbox="compiler" href="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/interpreter-add-repo1.png">
<img class="img-responsive" src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/interpreter-add-repo1.png" />
<a data-lightbox="compiler" href="../assets/themes/zeppelin/img/docs-img/interpreter-add-repo1.png">
<img class="img-responsive" src="../assets/themes/zeppelin/img/docs-img/interpreter-add-repo1.png" />
</a>
<a data-lightbox="compiler" href="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/interpreter-add-repo2.png">
<img class="img-responsive" src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/interpreter-add-repo2.png" />
<a data-lightbox="compiler" href="../assets/themes/zeppelin/img/docs-img/interpreter-add-repo2.png">
<img class="img-responsive" src="../assets/themes/zeppelin/img/docs-img/interpreter-add-repo2.png" />
</a>
</div>
<div class="col-md-6" style="padding-top:30px">

2
flink/pom.xml
View file

@ -34,7 +34,7 @@
<description>Zeppelin flink support</description>
<properties>
<flink.version>1.0.3</flink.version>
<flink.version>1.1.2</flink.version>
<flink.akka.version>2.3.7</flink.akka.version>
<scala.macros.version>2.0.1</scala.macros.version>
</properties>

38
flink/src/main/java/org/apache/zeppelin/flink/FlinkInterpreter.java
View file

@ -30,6 +30,7 @@ import java.util.*;
import org.apache.flink.api.scala.FlinkILoop;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster;
import org.apache.flink.runtime.util.EnvironmentInformation;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
@ -42,6 +43,7 @@ import org.slf4j.LoggerFactory;
import scala.Console;
import scala.None;
import scala.Option;
import scala.Some;
import scala.collection.JavaConversions;
import scala.collection.immutable.Nil;
@ -83,14 +85,25 @@ public class FlinkInterpreter extends Interpreter {
startFlinkMiniCluster();
}
flinkIloop = new FlinkILoop(getHost(), getPort(), (BufferedReader) null, new PrintWriter(out));
flinkIloop = new FlinkILoop(getHost(),
getPort(),
flinkConf,
(BufferedReader) null,
new PrintWriter(out));
flinkIloop.settings_$eq(createSettings());
flinkIloop.createInterpreter();
imain = flinkIloop.intp();
org.apache.flink.api.scala.ExecutionEnvironment env = flinkIloop.scalaEnv();
env.getConfig().disableSysoutLogging();
org.apache.flink.api.scala.ExecutionEnvironment benv =
flinkIloop.scalaBenv();
//new ExecutionEnvironment(remoteBenv)
org.apache.flink.streaming.api.scala.StreamExecutionEnvironment senv =
flinkIloop.scalaSenv();
senv.getConfig().disableSysoutLogging();
benv.getConfig().disableSysoutLogging();
// prepare bindings
imain.interpret("@transient var _binder = new java.util.HashMap[String, Object]()");
@ -100,13 +113,19 @@ public class FlinkInterpreter extends Interpreter {
imain.interpret("import scala.tools.nsc.io._");
imain.interpret("import Properties.userHome");
imain.interpret("import scala.compat.Platform.EOL");
imain.interpret("import org.apache.flink.api.scala._");
imain.interpret("import org.apache.flink.api.common.functions._");
binder.put("env", env);
imain.interpret("val env = _binder.get(\"env\").asInstanceOf["
+ env.getClass().getName() + "]");
binder.put("benv", benv);
imain.interpret("val benv = _binder.get(\"benv\").asInstanceOf["
+ benv.getClass().getName() + "]");
binder.put("senv", senv);
imain.interpret("val senv = _binder.get(\"senv\").asInstanceOf["
+ senv.getClass().getName() + "]");
}
private boolean localMode() {
@ -313,8 +332,6 @@ public class FlinkInterpreter extends Interpreter {
}
}
@Override
public void cancel(InterpreterContext context) {
}
@ -354,4 +371,5 @@ public class FlinkInterpreter extends Interpreter {
static final String toString(Object o) {
return (o instanceof String) ? (String) o : "";
}
}

3
flink/src/main/resources/interpreter-setting.json
View file

@ -16,6 +16,9 @@
"defaultValue": "6123",
"description": "port of running JobManager."
}
},
"editor": {
"language": "scala"
}
}
]

2
flink/src/test/java/org/apache/zeppelin/flink/FlinkInterpreterTest.java
View file

@ -81,7 +81,7 @@ public class FlinkInterpreterTest {
@Test
public void testWordCount() {
flink.interpret("val text = env.fromElements(\"To be or not to be\")", context);
flink.interpret("val text = benv.fromElements(\"To be or not to be\")", context);
flink.interpret("val counts = text.flatMap { _.toLowerCase.split(\" \") }.map { (_, 1) }.groupBy(0).sum(1)", context);
InterpreterResult result = flink.interpret("counts.print()", context);
assertEquals(Code.SUCCESS, result.code());

85
jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
View file

@ -27,7 +27,7 @@ import java.sql.SQLException;
import java.sql.Statement;
import java.util.*;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
@ -168,7 +168,7 @@ public class JDBCInterpreter extends Interpreter {
logger.debug("propertiesMap: {}", propertiesMap);
if (!StringUtils.isAnyEmpty(property.getProperty("zeppelin.jdbc.auth.type"))) {
if (!StringUtils.isEmpty(property.getProperty("zeppelin.jdbc.auth.type"))) {
JDBCSecurityImpl.createSecureConfiguration(property);
}
for (String propertyKey : propertiesMap.keySet()) {
@ -210,54 +210,57 @@ public class JDBCInterpreter extends Interpreter {
}
}
if (null == connection) {
final Properties properties = propertiesMap.get(propertyKey);
final Properties properties = (Properties) propertiesMap.get(propertyKey).clone();
logger.info(properties.getProperty(DRIVER_KEY));
Class.forName(properties.getProperty(DRIVER_KEY));
final String url = properties.getProperty(URL_KEY);
UserGroupInformation.AuthenticationMethod authType = JDBCSecurityImpl.getAuthtype(property);
switch (authType) {
case KERBEROS:
if (user == null) {
connection = DriverManager.getConnection(url, properties);
} else {
if ("hive".equalsIgnoreCase(propertyKey)) {
connection = DriverManager.getConnection(url + ";hive.server2.proxy.user=" + user,
properties);
if (StringUtils.isEmpty(property.getProperty("zeppelin.jdbc.auth.type"))) {
connection = DriverManager.getConnection(url, properties);
} else {
UserGroupInformation.AuthenticationMethod authType = JDBCSecurityImpl.getAuthtype(property);
switch (authType) {
case KERBEROS:
if (user == null) {
connection = DriverManager.getConnection(url, properties);
} else {
UserGroupInformation ugi = null;
try {
ugi = UserGroupInformation.createProxyUser(user,
UserGroupInformation.getCurrentUser());
} catch (Exception e) {
logger.error("Error in createProxyUser", e);
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append(e.getMessage()).append("\n");
stringBuilder.append(e.getCause());
throw new InterpreterException(stringBuilder.toString());
}
try {
connection = ugi.doAs(new PrivilegedExceptionAction<Connection>() {
@Override
public Connection run() throws Exception {
return DriverManager.getConnection(url, properties);
}
});
} catch (Exception e) {
logger.error("Error in doAs", e);
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append(e.getMessage()).append("\n");
stringBuilder.append(e.getCause());
throw new InterpreterException(stringBuilder.toString());
if ("hive".equalsIgnoreCase(propertyKey)) {
connection = DriverManager.getConnection(url + ";hive.server2.proxy.user=" + user,
properties);
} else {
UserGroupInformation ugi = null;
try {
ugi = UserGroupInformation.createProxyUser(user,
UserGroupInformation.getCurrentUser());
} catch (Exception e) {
logger.error("Error in createProxyUser", e);
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append(e.getMessage()).append("\n");
stringBuilder.append(e.getCause());
throw new InterpreterException(stringBuilder.toString());
}
try {
connection = ugi.doAs(new PrivilegedExceptionAction<Connection>() {
@Override
public Connection run() throws Exception {
return DriverManager.getConnection(url, properties);
}
});
} catch (Exception e) {
logger.error("Error in doAs", e);
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append(e.getMessage()).append("\n");
stringBuilder.append(e.getCause());
throw new InterpreterException(stringBuilder.toString());
}
}
}
}
break;
break;
default:
connection = DriverManager.getConnection(url, properties);
default:
connection = DriverManager.getConnection(url, properties);
}
}
}
propertyKeySqlCompleterMap.put(propertyKey, createSqlCompleter(connection));
return connection;

1
jdbc/src/main/java/org/apache/zeppelin/jdbc/security/JDBCSecurityImpl.java
View file

@ -18,7 +18,6 @@ package org.apache.zeppelin.jdbc.security;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.zeppelin.jdbc.SqlCompleter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

3
jdbc/src/main/resources/interpreter-setting.json
View file

@ -154,6 +154,9 @@
"defaultValue": "org.postgresql.Driver",
"description": ""
}
},
"editor": {
"language": "sql"
}
}
]

12
livy/src/main/resources/interpreter-setting.json
View file

@ -87,6 +87,9 @@
"defaultValue": "",
"description": "Adding extra libraries to livy interpreter"
}
},
"editor": {
"language": "scala"
}
},
{
@ -105,6 +108,9 @@
"defaultValue": "false",
"description": "Execute multiple SQL concurrently if set true."
}
},
"editor": {
"language": "sql"
}
},
{
@ -112,6 +118,9 @@
"name": "pyspark",
"className": "org.apache.zeppelin.livy.LivyPySparkInterpreter",
"properties": {
},
"editor": {
"language": "python"
}
},
{
@ -119,6 +128,9 @@
"name": "sparkr",
"className": "org.apache.zeppelin.livy.LivySparkRInterpreter",
"properties": {
},
"editor": {
"language": "r"
}
}
]

10
markdown/pom.xml
View file

@ -40,6 +40,11 @@
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.commonjava.googlecode.markdown4j</groupId>
<artifactId>markdown4j</artifactId>
@ -47,8 +52,9 @@
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<groupId>org.pegdown</groupId>
<artifactId>pegdown</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>

47
markdown/src/main/java/org/apache/zeppelin/markdown/Markdown4jParser.java Normal file
View file

@ -0,0 +1,47 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.markdown;
import org.markdown4j.Markdown4jProcessor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
/** Markdown Parser using markdown4j processor . */
public class Markdown4jParser implements MarkdownParser {
private Markdown4jProcessor processor;
public Markdown4jParser() {
processor = new Markdown4jProcessor();
}
@Override
public String render(String markdownText) {
String html = "";
try {
html = processor.process(markdownText);
} catch (IOException e) {
// convert checked exception to non-checked exception
throw new RuntimeException(e);
}
return html;
}
}

64
markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java → markdown/src/main/java/org/apache/zeppelin/markdown/MarkdownInterpreter.java
View file

@ -23,44 +23,78 @@ import java.util.Properties;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
import org.apache.zeppelin.interpreter.InterpreterUtils;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.scheduler.SchedulerFactory;
import org.markdown4j.Markdown4jProcessor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Markdown interpreter for Zeppelin.
*/
public class Markdown extends Interpreter {
private Markdown4jProcessor md;
static final Logger LOGGER = LoggerFactory.getLogger(Markdown.class);
/** MarkdownInterpreter interpreter for Zeppelin. */
public class MarkdownInterpreter extends Interpreter {
private static final Logger LOGGER = LoggerFactory.getLogger(MarkdownInterpreter.class);
public Markdown(Properties property) {
private MarkdownParser parser;
/** Markdown Parser Type. */
public enum MarkdownParserType {
PEGDOWN {
@Override
public String toString() {
return PARSER_TYPE_PEGDOWN;
}
},
MARKDOWN4j {
@Override
public String toString() {
return PARSER_TYPE_MARKDOWN4J;
}
}
}
public static final String MARKDOWN_PARSER_TYPE = "markdown.parser.type";
public static final String PARSER_TYPE_PEGDOWN = "pegdown";
public static final String PARSER_TYPE_MARKDOWN4J = "markdown4j";
public MarkdownInterpreter(Properties property) {
super(property);
}
public static MarkdownParser createMarkdownParser(String parserType) {
LOGGER.debug("Creating " + parserType + " markdown interpreter");
if (MarkdownParserType.PEGDOWN.toString().equals(parserType)) {
return new PegdownParser();
} else {
/** default parser. */
return new Markdown4jParser();
}
}
@Override
public void open() {
md = new Markdown4jProcessor();
String parserType = getProperty(MARKDOWN_PARSER_TYPE);
parser = createMarkdownParser(parserType);
}
@Override
public void close() {}
@Override
public InterpreterResult interpret(String st, InterpreterContext interpreterContext) {
public InterpreterResult interpret(String markdownText, InterpreterContext interpreterContext) {
String html;
try {
html = md.process(st);
} catch (IOException | java.lang.RuntimeException e) {
LOGGER.error("Exception in Markdown while interpret ", e);
html = parser.render(markdownText);
} catch (RuntimeException e) {
LOGGER.error("Exception in MarkdownInterpreter while interpret ", e);
return new InterpreterResult(Code.ERROR, InterpreterUtils.getMostRelevantMessage(e));
}
return new InterpreterResult(Code.SUCCESS, "%html " + html);
}
@ -79,8 +113,8 @@ public class Markdown extends Interpreter {
@Override
public Scheduler getScheduler() {
return SchedulerFactory.singleton().createOrGetParallelScheduler(
Markdown.class.getName() + this.hashCode(), 5);
return SchedulerFactory.singleton()
.createOrGetParallelScheduler(MarkdownInterpreter.class.getName() + this.hashCode(), 5);
}
@Override

23
markdown/src/main/java/org/apache/zeppelin/markdown/MarkdownParser.java Normal file
View file

@ -0,0 +1,23 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.markdown;
/** Abstract Markdown Parser. */
public interface MarkdownParser {
String render(String markdownText);
}

56
markdown/src/main/java/org/apache/zeppelin/markdown/PegdownParser.java Normal file
View file

@ -0,0 +1,56 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.markdown;
import org.pegdown.Extensions;
import org.pegdown.PegDownProcessor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** Markdown Parser using pegdown processor. */
public class PegdownParser implements MarkdownParser {
private PegDownProcessor processor;
public PegdownParser() {
int pegdownOptions = Extensions.ALL_WITH_OPTIONALS - Extensions.ANCHORLINKS;
int parsingTimeoutAsMillis = 5000;
processor = new PegDownProcessor(pegdownOptions, parsingTimeoutAsMillis);
}
@Override
public String render(String markdownText) {
String html = "";
String parsed = processor.markdownToHtml(markdownText);
if (null == parsed) {
throw new RuntimeException("Cannot parse markdown text to HTML using pegdown");
}
html = wrapWithMarkdownClassDiv(parsed);
return html;
}
/** wrap with markdown class div to styling DOM using css. */
public static String wrapWithMarkdownClassDiv(String html) {
return new StringBuilder()
.append("<div class=\"markdown-body\">\n")
.append(html)
.append("\n</div>")
.toString();
}
}

14
markdown/src/main/resources/interpreter-setting.json
View file

@ -2,7 +2,17 @@
{
"group": "md",
"name": "md",
"className": "org.apache.zeppelin.markdown.Markdown",
"properties": null
"className": "org.apache.zeppelin.markdown.MarkdownInterpreter",
"properties": {
"markdown.parser.type": {
"envName": "MARKDOWN_PARSER_TYPE",
"propertyName": "markdown.parser.type",
"defaultValue": "markdown4j",
"description": "Markdown Parser Type. Available values: markdown4j, pegdown. Default = markdown4j"
}
},
"editor": {
"language": "markdown"
}
}
]

41
markdown/src/test/java/org/apache/zeppelin/markdown/MarkdownTest.java → markdown/src/test/java/org/apache/zeppelin/markdown/Markdown4jParserTest.java
View file

@ -17,32 +17,35 @@
package org.apache.zeppelin.markdown;
import static org.junit.Assert.assertEquals;
import java.util.Properties;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.markdown.Markdown;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
public class MarkdownTest {
import java.util.Properties;
@Before
public void setUp() throws Exception {
}
import static org.junit.Assert.assertEquals;
@After
public void tearDown() throws Exception {
}
public class Markdown4jParserTest {
@Test
public void test() {
Markdown md = new Markdown(new Properties());
md.open();
InterpreterResult result = md.interpret("This is ~~deleted~~ text", null);
assertEquals("<p>This is <s>deleted</s> text</p>\n", result.message());
}
MarkdownInterpreter md;
@Before
public void setUp() throws Exception {
Properties props = new Properties();
props.put(MarkdownInterpreter.MARKDOWN_PARSER_TYPE, MarkdownInterpreter.PARSER_TYPE_MARKDOWN4J);
md = new MarkdownInterpreter(props);
md.open();
}
@After
public void tearDown() throws Exception {
md.close();
}
@Test
public void testStrikethrough() {
InterpreterResult result = md.interpret("This is ~~deleted~~ text", null);
assertEquals("<p>This is <s>deleted</s> text</p>\n", result.message());
}
}

302
markdown/src/test/java/org/apache/zeppelin/markdown/PegdownParserTest.java Normal file
View file

@ -0,0 +1,302 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.markdown;
import static org.junit.Assert.assertEquals;
import java.util.Properties;
import org.apache.zeppelin.interpreter.InterpreterResult;
import static org.apache.zeppelin.markdown.PegdownParser.wrapWithMarkdownClassDiv;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
public class PegdownParserTest {
MarkdownInterpreter md;
@Before
public void setUp() throws Exception {
Properties props = new Properties();
props.put(MarkdownInterpreter.MARKDOWN_PARSER_TYPE, MarkdownInterpreter.PARSER_TYPE_PEGDOWN);
md = new MarkdownInterpreter(props);
md.open();
}
@After
public void tearDown() throws Exception {
md.close();
}
@Test
public void testHeader() {
InterpreterResult r1 = md.interpret("# H1", null);
assertEquals(wrapWithMarkdownClassDiv("<h1>H1</h1>"), r1.message());
InterpreterResult r2 = md.interpret("## H2", null);
assertEquals(wrapWithMarkdownClassDiv("<h2>H2</h2>"), r2.message());
InterpreterResult r3 = md.interpret("### H3", null);
assertEquals(wrapWithMarkdownClassDiv("<h3>H3</h3>"), r3.message());
InterpreterResult r4 = md.interpret("#### H4", null);
assertEquals(wrapWithMarkdownClassDiv("<h4>H4</h4>"), r4.message());
InterpreterResult r5 = md.interpret("##### H5", null);
assertEquals(wrapWithMarkdownClassDiv("<h5>H5</h5>"), r5.message());
InterpreterResult r6 = md.interpret("###### H6", null);
assertEquals(wrapWithMarkdownClassDiv("<h6>H6</h6>"), r6.message());
InterpreterResult r7 = md.interpret("Alt-H1\n" + "======", null);
assertEquals(wrapWithMarkdownClassDiv("<h1>Alt-H1</h1>"), r7.message());
InterpreterResult r8 = md.interpret("Alt-H2\n" + "------", null);
assertEquals(wrapWithMarkdownClassDiv("<h2>Alt-H2</h2>"), r8.message());
}
@Test
public void testStrikethrough() {
InterpreterResult result = md.interpret("This is ~~deleted~~ text", null);
assertEquals(
wrapWithMarkdownClassDiv("<p>This is <del>deleted</del> text</p>"), result.message());
}
@Test
public void testItalics() {
InterpreterResult result = md.interpret("This is *italics* text", null);
assertEquals(
wrapWithMarkdownClassDiv("<p>This is <em>italics</em> text</p>"), result.message());
}
@Test
public void testStrongEmphasis() {
InterpreterResult result = md.interpret("This is **strong emphasis** text", null);
assertEquals(
wrapWithMarkdownClassDiv("<p>This is <strong>strong emphasis</strong> text</p>"),
result.message());
}
@Test
public void testOrderedList() {
String input =
new StringBuilder()
.append("1. First ordered list item\n")
.append("2. Another item")
.toString();
String expected =
new StringBuilder()
.append("<ol>\n")
.append(" <li>First ordered list item</li>\n")
.append(" <li>Another item</li>\n")
.append("</ol>")
.toString();
InterpreterResult result = md.interpret(input, null);
assertEquals(wrapWithMarkdownClassDiv(expected), result.message());
}
@Test
public void testUnorderedList() {
String input =
new StringBuilder()
.append("* Unordered list can use asterisks\n")
.append("- Or minuses\n")
.append("+ Or pluses")
.toString();
String expected =
new StringBuilder()
.append("<ul>\n")
.append(" <li>Unordered list can use asterisks</li>\n")
.append(" <li>Or minuses</li>\n")
.append(" <li>Or pluses</li>\n")
.append("</ul>")
.toString();
InterpreterResult result = md.interpret(input, null);
assertEquals(wrapWithMarkdownClassDiv(expected), result.message());
}
@Test
public void testLinks() {
String input =
new StringBuilder()
.append("[I'm an inline-style link](https://www.google.com)\n")
.append("\n")
.append(
"[I'm an inline-style link with title](https://www.google.com \"Google's Homepage\")\n")
.append("\n")
.append("[I'm a reference-style link][Arbitrary case-insensitive reference text]\n")
.append("\n")
.append("[I'm a relative reference to a repository file](../blob/master/LICENSE)\n")
.append("\n")
.append("[You can use numbers for reference-style link definitions][1]\n")
.append("\n")
.append("Or leave it empty and use the [link text itself].\n")
.append("\n")
.append("URLs and URLs in angle brackets will automatically get turned into links. \n")
.append("http://www.example.com or <http://www.example.com> and sometimes \n")
.append("example.com (but not on Github, for example).\n")
.append("\n")
.append("Some text to show that the reference links can follow later.\n")
.append("\n")
.append("[arbitrary case-insensitive reference text]: https://www.mozilla.org\n")
.append("[1]: http://slashdot.org\n")
.append("[link text itself]: http://www.reddit.com")
.toString();
String expected =
new StringBuilder()
.append(
"<p><a href=\"https://www.google.com\">I&rsquo;m an inline-style link</a></p>\n")
.append(
"<p><a href=\"https://www.google.com\" title=\"Google&#39;s Homepage\">I&rsquo;m an inline-style link with title</a></p>\n")
.append(
"<p><a href=\"https://www.mozilla.org\">I&rsquo;m a reference-style link</a></p>\n")
.append(
"<p><a href=\"../blob/master/LICENSE\">I&rsquo;m a relative reference to a repository file</a></p>\n")
.append(
"<p><a href=\"http://slashdot.org\">You can use numbers for reference-style link definitions</a></p>\n")
.append(
"<p>Or leave it empty and use the <a href=\"http://www.reddit.com\">link text itself</a>.</p>\n")
.append(
"<p>URLs and URLs in angle brackets will automatically get turned into links.<br/><a href=\"http://www.example.com\">http://www.example.com</a> or <a href=\"http://www.example.com\">http://www.example.com</a> and sometimes<br/>example.com (but not on Github, for example).</p>\n")
.append("<p>Some text to show that the reference links can follow later.</p>")
.toString();
InterpreterResult result = md.interpret(input, null);
assertEquals(wrapWithMarkdownClassDiv(expected), result.message());
}
@Test
public void testInlineCode() {
InterpreterResult result = md.interpret("Inline `code` has `back-ticks around` it.", null);
assertEquals(
wrapWithMarkdownClassDiv(
"<p>Inline <code>code</code> has <code>back-ticks around</code> it.</p>"),
result.message());
}
@Test
public void testBlockQuotes() {
InterpreterResult r1 =
md.interpret(
"> Blockquotes are very handy in email to emulate reply text.\n"
+ "> This line is part of the same quote.",
null);
assertEquals(
wrapWithMarkdownClassDiv(
"<blockquote>\n"
+ " <p>Blockquotes are very handy in email to emulate reply text.<br/>This line is part of the same quote.</p>\n"
+ "</blockquote>"),
r1.message());
InterpreterResult r2 =
md.interpret(
"> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **MarkdownInterpreter** into a blockquote. ",
null);
assertEquals(
wrapWithMarkdownClassDiv(
"<blockquote>\n"
+ " <p>This is a very long line that will still be quoted properly when it wraps. Oh boy let&rsquo;s keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can <em>put</em> <strong>MarkdownInterpreter</strong> into a blockquote. </p>\n"
+ "</blockquote>"),
r2.message());
}
@Test
public void testSimpleTable() {
String input =
new StringBuilder()
.append("MarkdownInterpreter | Less | Pretty\n")
.append("--- | --- | ---\n")
.append("*Still* | `renders` | **nicely**\n")
.append("1 | 2 | 3")
.toString();
String expected =
new StringBuilder()
.append("<table>\n")
.append(" <thead>\n")
.append(" <tr>\n")
.append(" <th>MarkdownInterpreter </th>\n")
.append(" <th>Less </th>\n")
.append(" <th>Pretty</th>\n")
.append(" </tr>\n")
.append(" </thead>\n")
.append(" <tbody>\n")
.append(" <tr>\n")
.append(" <td><em>Still</em> </td>\n")
.append(" <td><code>renders</code> </td>\n")
.append(" <td><strong>nicely</strong></td>\n")
.append(" </tr>\n")
.append(" <tr>\n")
.append(" <td>1 </td>\n")
.append(" <td>2 </td>\n")
.append(" <td>3</td>\n")
.append(" </tr>\n")
.append(" </tbody>\n")
.append("</table>")
.toString();
InterpreterResult result = md.interpret(input, null);
assertEquals(wrapWithMarkdownClassDiv(expected), result.message());
}
@Test
public void testAlignedTable() {
String input =
new StringBuilder()
.append("| First Header | Second Header | Third Header |\n")
.append("| :----------- | :-----------: | -------------------: |\n")
.append("| First row | Data | Very long data entry |\n")
.append("| Second row | **Cell** | *Cell* |")
.toString();
String expected =
new StringBuilder()
.append("<table>\n")
.append(" <thead>\n")
.append(" <tr>\n")
.append(" <th align=\"left\">First Header </th>\n")
.append(" <th align=\"center\">Second Header </th>\n")
.append(" <th align=\"right\">Third Header </th>\n")
.append(" </tr>\n")
.append(" </thead>\n")
.append(" <tbody>\n")
.append(" <tr>\n")
.append(" <td align=\"left\">First row </td>\n")
.append(" <td align=\"center\">Data </td>\n")
.append(" <td align=\"right\">Very long data entry </td>\n")
.append(" </tr>\n")
.append(" <tr>\n")
.append(" <td align=\"left\">Second row </td>\n")
.append(" <td align=\"center\"><strong>Cell</strong> </td>\n")
.append(" <td align=\"right\"><em>Cell</em> </td>\n")
.append(" </tr>\n")
.append(" </tbody>\n")
.append("</table>")
.toString();
InterpreterResult result = md.interpret(input, null);
assertEquals(wrapWithMarkdownClassDiv(expected), result.message());
}
}

14
pom.xml
View file

@ -384,7 +384,7 @@
</plugin>
<!--TODO(alex): make part of the build and reconcile conflicts
<plugin>
<plugin>
<groupId>com.ning.maven.plugins</groupId>
<artifactId>maven-duplicate-finder-plugin</artifactId>
<version>1.0.4</version>
@ -393,12 +393,12 @@
<id>default</id>
<phase>verify</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
<goal>check</goal>
</goals>
</execution>
</executions>
<configuration>
<failBuildInCaseOfConflict>true</failBuildInCaseOfConflict>
<failBuildInCaseOfConflict>true</failBuildInCaseOfConflict>
</configuration>
</plugin>
-->
@ -734,7 +734,6 @@
<exclude>**/README.md</exclude>
<exclude>DEPENDENCIES</exclude>
<exclude>DEPLOY.md</exclude>
<exclude>CONTRIBUTING.md</exclude>
<exclude>STYLE.md</exclude>
<exclude>Roadmap.md</exclude>
<exclude>**/licenses/**</exclude>
@ -772,7 +771,6 @@
<exclude>docs/sitemap.txt</exclude>
<exclude>docs/search_data.json</exclude>
<exclude>**/dependency-reduced-pom.xml</exclude>
<exclude>docs/CONTRIBUTING.md</exclude>
<!-- bundled from anchor -->
<exclude>docs/assets/themes/zeppelin/js/anchor.min.js</exclude>

3
python/src/main/resources/interpreter-setting.json
View file

@ -16,6 +16,9 @@
"defaultValue": "1000",
"description": "Max number of dataframe rows to display."
}
},
"editor": {
"language": "python"
}
},
{

5
shell/src/main/resources/interpreter-setting.json
View file

@ -28,6 +28,9 @@
"defaultValue": "",
"description": "Kerberos principal"
}
},
"editor": {
"language": "sh"
}
}
]
]

47
spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
View file

@ -118,7 +118,7 @@ public class SparkInterpreter extends Interpreter {
private Map<String, Object> binder;
private SparkVersion sparkVersion;
private File outputDir; // class outputdir for scala 2.11
private static File outputDir; // class outputdir for scala 2.11
private Object classServer; // classserver for scala 2.11
@ -329,6 +329,7 @@ public class SparkInterpreter extends Interpreter {
}
setupConfForPySpark(conf);
setupConfForSparkR(conf);
Class SparkSession = Utils.findClass("org.apache.spark.sql.SparkSession");
Object builder = Utils.invokeStaticMethod(SparkSession, "builder");
Utils.invokeMethod(builder, "config", new Class[]{ SparkConf.class }, new Object[]{ conf });
@ -443,6 +444,7 @@ public class SparkInterpreter extends Interpreter {
}
}
setupConfForPySpark(conf);
setupConfForSparkR(conf);
SparkContext sparkContext = new SparkContext(conf);
return sparkContext;
}
@ -494,6 +496,35 @@ public class SparkInterpreter extends Interpreter {
}
}
private void setupConfForSparkR(SparkConf conf) {
String sparkRBasePath = new InterpreterProperty("SPARK_HOME", null, null, null).getValue();
File sparkRPath;
if (null == sparkRBasePath) {
sparkRBasePath =
new InterpreterProperty("ZEPPELIN_HOME", "zeppelin.home", "../", null).getValue();
sparkRPath = new File(sparkRBasePath,
"interpreter" + File.separator + "spark" + File.separator + "R");
} else {
sparkRPath = new File(sparkRBasePath, "R" + File.separator + "lib");
}
sparkRPath = new File(sparkRPath, "sparkr.zip");
if (sparkRPath.exists() && sparkRPath.isFile()) {
String archives = null;
if (conf.contains("spark.yarn.dist.archives")) {
archives = conf.get("spark.yarn.dist.archives");
}
if (archives != null) {
archives = archives + "," + sparkRPath + "#sparkr";
} else {
archives = sparkRPath + "#sparkr";
}
conf.set("spark.yarn.dist.archives", archives);
} else {
logger.warn("sparkr.zip is not found, sparkr may not work.");
}
}
static final String toString(Object o) {
return (o instanceof String) ? (String) o : "";
}
@ -572,8 +603,11 @@ public class SparkInterpreter extends Interpreter {
sparkReplClassDir = System.getProperty("java.io.tmpdir");
}
outputDir = createTempDir(sparkReplClassDir);
synchronized (sharedInterpreterLock) {
if (outputDir == null) {
outputDir = createTempDir(sparkReplClassDir);
}
}
argList.add("-Yrepl-class-based");
argList.add("-Yrepl-outdir");
argList.add(outputDir.getAbsolutePath());
@ -1276,7 +1310,12 @@ public class SparkInterpreter extends Interpreter {
logger.info("Close interpreter");
if (numReferenceOfSparkContext.decrementAndGet() == 0) {
sc.stop();
if (sparkSession != null) {
Utils.invokeMethod(sparkSession, "stop");
} else if (sc != null){
sc.stop();
}
sparkSession = null;
sc = null;
if (classServer != null) {
Utils.invokeMethod(classServer, "stop");

10
spark/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
View file

@ -52,13 +52,19 @@ public class SparkVersion {
if (pos > 0) {
numberPart = versionString.substring(0, pos);
}
version = Integer.parseInt(numberPart.replaceAll("\\.", ""));
String versions[] = numberPart.split("\\.");
int major = Integer.parseInt(versions[0]);
int minor = Integer.parseInt(versions[1]);
int patch = Integer.parseInt(versions[2]);
// version is always 5 digits. (e.g. 2.0.0 -> 20000, 1.6.2 -> 10602)
version = Integer.parseInt(String.format("%d%02d%02d", major, minor, patch));
} catch (Exception e) {
logger.error("Can not recognize Spark version " + versionString +
". Assume it's a future release", e);
// assume it is future release
version = 999;
version = 99999;
}
}

2
spark/src/main/resources/R/zeppelin_sparkr.R
View file

@ -42,7 +42,7 @@ assign(".scStartTime", as.integer(Sys.time()), envir = SparkR:::.sparkREnv)
# setup spark env
assign(".sc", SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinRContext", "getSparkContext"), envir = SparkR:::.sparkREnv)
assign("sc", get(".sc", envir = SparkR:::.sparkREnv), envir=.GlobalEnv)
if (version >= 200) {
if (version >= 20000) {
assign(".sparkRsession", SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinRContext", "getSparkSession"), envir = SparkR:::.sparkREnv)
assign("spark", get(".sparkRsession", envir = SparkR:::.sparkREnv), envir = .GlobalEnv)
}

12
spark/src/main/resources/interpreter-setting.json
View file

@ -54,6 +54,9 @@
"defaultValue": "local[*]",
"description": "Spark master uri. ex) spark://masterhost:7077"
}
},
"editor": {
"language": "scala"
}
},
{
@ -85,6 +88,9 @@
"defaultValue": "true",
"description": "Import implicits, UDF collection, and sql if set true. true by default."
}
},
"editor": {
"language": "sql"
}
},
{
@ -104,6 +110,9 @@
"defaultValue": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
"description": "A list of 'id,remote-repository-URL,is-snapshot;' for each remote repository."
}
},
"editor": {
"language": "scala"
}
},
{
@ -117,6 +126,9 @@
"defaultValue": "python",
"description": "Python command to run pyspark with"
}
},
"editor": {
"language": "python"
}
}
]

11
spark/src/main/resources/python/zeppelin_pyspark.py
View file

@ -107,9 +107,9 @@ class PyZeppelinContext(dict):
class SparkVersion(object):
SPARK_1_4_0 = 140
SPARK_1_3_0 = 130
SPARK_2_0_0 = 200
SPARK_1_4_0 = 10400
SPARK_1_3_0 = 10300
SPARK_2_0_0 = 20000
def __init__(self, versionNumber):
self.version = versionNumber
@ -218,7 +218,10 @@ java_import(gateway.jvm, "scala.Tuple2")
jconf = intp.getSparkConf()
conf = SparkConf(_jvm = gateway.jvm, _jconf = jconf)
sc = SparkContext(jsc=jsc, gateway=gateway, conf=conf)
sqlc = SQLContext(sc, intp.getSQLContext())
if sparkVersion.isSpark2():
sqlc = SQLContext(sparkContext=sc, jsqlContext=intp.getSQLContext())
else:
sqlc = SQLContext(sparkContext=sc, sqlContext=intp.getSQLContext())
sqlContext = sqlc
if sparkVersion.isSpark2():

15
spark/src/main/sparkr-resources/interpreter-setting.json
View file

@ -54,6 +54,9 @@
"defaultValue": "local[*]",
"description": "Spark master uri. ex) spark://masterhost:7077"
}
},
"editor": {
"language": "scala"
}
},
{
@ -85,6 +88,9 @@
"defaultValue": "true",
"description": "Import implicits, UDF collection, and sql if set true. true by default."
}
},
"editor": {
"language": "sql"
}
},
{
@ -104,6 +110,9 @@
"defaultValue": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
"description": "A list of 'id,remote-repository-URL,is-snapshot;' for each remote repository."
}
},
"editor": {
"language": "scala"
}
},
{
@ -117,6 +126,9 @@
"defaultValue": "python",
"description": "Python command to run pyspark with"
}
},
"editor": {
"language": "python"
}
},
{
@ -148,6 +160,9 @@
"defaultValue": "out.format = 'html', comment = NA, echo = FALSE, results = 'asis', message = F, warning = F",
"description": ""
}
},
"editor": {
"language": "r"
}
}
]

9
spark/src/test/java/org/apache/zeppelin/spark/SparkVersionTest.java
View file

@ -24,7 +24,7 @@ public class SparkVersionTest {
@Test
public void testUnknownSparkVersion() {
assertEquals(999, SparkVersion.fromVersionString("DEV-10.10").toNumber());
assertEquals(99999, SparkVersion.fromVersionString("DEV-10.10").toNumber());
}
@Test
@ -33,6 +33,8 @@ public class SparkVersionTest {
assertFalse(SparkVersion.fromVersionString("1.5.9").isUnsupportedVersion());
assertTrue(SparkVersion.fromVersionString("0.9.0").isUnsupportedVersion());
assertTrue(SparkVersion.UNSUPPORTED_FUTURE_VERSION.isUnsupportedVersion());
// should support spark2 version of HDP 2.5
assertFalse(SparkVersion.fromVersionString("2.0.0.2.5.0.0-1245").isUnsupportedVersion());
}
@Test
@ -40,6 +42,9 @@ public class SparkVersionTest {
// test equals
assertEquals(SparkVersion.SPARK_1_2_0, SparkVersion.fromVersionString("1.2.0"));
assertEquals(SparkVersion.SPARK_1_5_0, SparkVersion.fromVersionString("1.5.0-SNAPSHOT"));
assertEquals(SparkVersion.SPARK_1_5_0, SparkVersion.fromVersionString("1.5.0-SNAPSHOT"));
// test spark2 version of HDP 2.5
assertEquals(SparkVersion.SPARK_2_0_0, SparkVersion.fromVersionString("2.0.0.2.5.0.0-1245"));
// test newer than
assertFalse(SparkVersion.SPARK_1_2_0.newerThan(SparkVersion.SPARK_1_2_0));
@ -60,7 +65,7 @@ public class SparkVersionTest {
assertTrue(SparkVersion.SPARK_1_2_0.olderThanEquals(SparkVersion.SPARK_1_3_0));
// conversion
assertEquals(120, SparkVersion.SPARK_1_2_0.toNumber());
assertEquals(10200, SparkVersion.SPARK_1_2_0.toNumber());
assertEquals("1.2.0", SparkVersion.SPARK_1_2_0.toString());
}
}

11
zeppelin-distribution/src/bin_license/LICENSE
View file

@ -115,6 +115,9 @@ The following components are provided under Apache License.
(Apache 2.0) Servlet API (org.mortbay.jetty:servlet-api:2.5-20081211 - https://en.wikipedia.org/wiki/Jetty_(web_server))
(Apache 2.0) Google HTTP Client Library for Java (com.google.http-client:google-http-client-jackson2:1.21.0 - https://github.com/google/google-http-java-client/tree/dev/google-http-client-jackson2)
(Apache 2.0) angular-esri-map (https://github.com/Esri/angular-esri-map)
(Apache 2.0) pegdown (org.pegdown:pegdown:1.6.0 - https://github.com/sirthias/pegdown)
(Apache 2.0) parboiled-java (org.parboiled:parboiled-java:1.1.7 - https://github.com/sirthias/parboiled)
(Apache 2.0) parboiled-core (org.parboiled:parboiled-core:1.1.7 - https://github.com/sirthias/parboiled)
========================================================================
MIT licenses
@ -151,6 +154,7 @@ The text of each license is also included at licenses/LICENSE-[project]-[version
(The MIT License) bcprov-jdk15on v1.51 (org.bouncycastle:bcprov-jdk15on:jar:1.51 - http://www.bouncycastle.org/java.html) - http://www.bouncycastle.org/licence.html
(The MIT License) AnchorJS (https://github.com/bryanbraun/anchorjs) - https://github.com/bryanbraun/anchorjs/blob/master/README.md#license
(The MIT License) moment-duration-format v1.3.0 (https://github.com/jsmreese/moment-duration-format) - https://github.com/jsmreese/moment-duration-format/blob/master/LICENSE
(The MIT License) github-markdown-css 2.4.0 (https://github.com/sindresorhus/github-markdown-css) - https://github.com/sindresorhus/github-markdown-css/blob/gh-pages/license
The following components are provided under the MIT License.
@ -191,10 +195,13 @@ The following components are provided under the BSD-style License.
(BSD-like) Scala Actors library (org.scala-lang:scala-actors:2.11.7 - http://www.scala-lang.org/)
(BSD-like) Scala Compiler (org.scala-lang:scala-compiler:2.11.7 - http://www.scala-lang.org/)
(BSD-like) Scala Compiler (org.scala-lang:scala-reflect:2.11.7 - http://www.scala-lang.org/)
(BSD-like) ASM (asm:asm:jar:3.1 - http://asm.ow2.org/) - Copyright (c) 2000-2011 INRIA, France Telecom
(BSD-like) ASM asm (asm:asm:jar:3.1 - http://asm.ow2.org/) - Copyright (c) 2000-2011 INRIA, France Telecom
(BSD-like) ASM asm-tree (org.ow2.asm:asm-tree:5.0.3 - http://asm.ow2.org/) - Copyright (c) 2000-2011 INRIA, France Telecom
(BSD-like) ASM asm-analysis (org.ow2.asm:asm-analysis:5.0.3 - http://asm.ow2.org/) - Copyright (c) 2000-2011 INRIA, France Telecom
(BSD-like) ASM asm-utils (org.ow2.asm:asm-utils:5.0.3 - http://asm.ow2.org/) - Copyright (c) 2000-2011 INRIA, France Telecom
(New BSD License) Markdown4j (org.commonjava.googlecode.markdown4j:markdown4j:jar:2.2-cj-1.0 - https://code.google.com/p/markdown4j/)
(New BSD License) Py4J (net.sf.py4j:py4j:0.9 - http://py4j.sourceforge.net/)
(New BSD License) Py4J (net.sf.py4j:py4j:0.10.1 - http://py4j.sourceforge.net/) - https://github.com/bartdag/py4j/blob/0.10.1/LICENSE.txt
(New BSD License) Markdown4j (org.commonjava.googlecode.markdown4j:markdown4j:jar:2.2-cj-1.0 - https://code.google.com/p/markdown4j/)
(BSD 3 Clause) Paranamer (com.thoughtworks.paranamer:paranamer:jar:2.6) - https://github.com/paul-hammant/paranamer/blob/paranamer-parent-2.6/LICENSE.txt
(BSD 3 Clause) netlib core (com.github.fommil.netlib:core:1.1.2 - https://github.com/fommil/netlib-java/core)
(BSD 3 Clause) JPMML-Model (org.jpmml:pmml-model:1.2.7 - https://github.com/jpmml/jpmml-model)

21
zeppelin-distribution/src/bin_license/licenses/LICENSE-github-markdown-css-2.4.0 Normal file
View file

@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

12
zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java
View file

@ -251,6 +251,7 @@ public abstract class Interpreter {
private String className;
private boolean defaultInterpreter;
private Map<String, InterpreterProperty> properties;
private Map<String, Object> editor;
private String path;
public RegisteredInterpreter(String name, String group, String className,
@ -266,6 +267,7 @@ public abstract class Interpreter {
this.className = className;
this.defaultInterpreter = defaultInterpreter;
this.properties = properties;
this.editor = new HashMap<>();
}
public String getName() {
@ -292,6 +294,10 @@ public abstract class Interpreter {
return properties;
}
public Map<String, Object> getEditor() {
return editor;
}
public void setPath(String path) {
this.path = path;
}
@ -338,9 +344,9 @@ public abstract class Interpreter {
@Deprecated
public static void register(String name, String group, String className,
boolean defaultInterpreter, Map<String, InterpreterProperty> properties) {
logger.error("Static initialization is deprecated. You should change it to use " +
"interpreter-setting.json in your jar or " +
"interpreter/{interpreter}/interpreter-setting.json");
logger.warn("Static initialization is deprecated for interpreter {}, You should change it " +
"to use interpreter-setting.json in your jar or " +
"interpreter/{interpreter}/interpreter-setting.json", name);
register(new RegisteredInterpreter(name, group, className, defaultInterpreter, properties));
}

1
zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterProperty.java
View file

@ -75,6 +75,7 @@ public class InterpreterProperty {
}
public boolean equals(Object o) {
if (o == null) return false;
return this.toString().equals(o.toString());
}

7
zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java
View file

@ -527,4 +527,11 @@ public class RemoteInterpreter extends Interpreter {
public void setEnv(Map<String, String> env) {
this.env = env;
}
public void addEnv(Map<String, String> env) {
if (this.env == null) {
this.env = new HashMap<>();
}
this.env.putAll(env);
}
}

158
zeppelin-interpreter/src/test/java/org/apache/zeppelin/interpreter/InterpreterResultTest.java
View file

@ -23,91 +23,89 @@ import static org.junit.Assert.fail;
import org.junit.Test;
public class InterpreterResultTest {
@Test
public void testTextType() {
InterpreterResult result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"this is a TEXT type");
assertEquals("No magic",InterpreterResult.Type.TEXT, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%this is a TEXT type");
assertEquals("No magic",InterpreterResult.Type.TEXT, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%\n");
assertEquals("No magic",InterpreterResult.Type.TEXT, result.type());
}
@Test
public void testSimpleMagicType() {
InterpreterResult result = null;
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%table col1\tcol2\naaa\t123\n");
assertEquals(InterpreterResult.Type.TABLE, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%table\ncol1\tcol2\naaa\t123\n");
assertEquals(InterpreterResult.Type.TABLE, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"some text before magic word %table col1\tcol2\naaa\t123\n");
assertEquals(InterpreterResult.Type.TABLE, result.type());
}
@Test
public void testTextType() {
public void testComplexMagicType() {
InterpreterResult result = null;
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"some text before %table col1\tcol2\naaa\t123\n");
assertEquals("some text before magic return magic",InterpreterResult.Type.TABLE, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%html <h3> This is a hack </h3> %table\n col1\tcol2\naaa\t123\n");
assertEquals("magic A before magic B return magic A",InterpreterResult.Type.HTML, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"some text before magic word %table col1\tcol2\naaa\t123\n %html <h3> This is a hack </h3>");
assertEquals("text & magic A before magic B return magic A" ,InterpreterResult.Type.TABLE, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%table col1\tcol2\naaa\t123\n %html <h3> This is a hack </h3> %table col1\naaa\n123\n");
assertEquals("magic A, magic B, magic a' return magic A" ,InterpreterResult.Type.TABLE, result.type());
InterpreterResult result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "this is a TEXT type");
assertEquals("No magic", InterpreterResult.Type.TEXT, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%this is a TEXT type");
assertEquals("No magic", InterpreterResult.Type.TEXT, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%\n");
assertEquals("No magic", InterpreterResult.Type.TEXT, result.type());
}
}
@Test
public void testSimpleMagicType() {
InterpreterResult result = null;
@Test
public void testSimpleMagicData() {
InterpreterResult result = null;
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%table col1\tcol2\naaa\t123\n");
assertEquals("%table col1\tcol2\naaa\t123\n","col1\tcol2\naaa\t123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%table\ncol1\tcol2\naaa\t123\n");
assertEquals("%table\ncol1\tcol2\naaa\t123\n","col1\tcol2\naaa\t123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"some text before magic word %table col1\tcol2\naaa\t123\n");
assertEquals("some text before magic word %table col1\tcol2\naaa\t123\n","col1\tcol2\naaa\t123\n", result.message());
}
@Test
public void testComplexMagicData() {
InterpreterResult result = null;
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"some text before %table col1\tcol2\naaa\t123\n");
assertEquals("text before %table return %table","col1\tcol2\naaa\t123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%html <h3> This is a hack </h3> %table\ncol1\tcol2\naaa\t123\n");
assertEquals("%html before %table return %html"," <h3> This is a hack </h3> %table\n" +
"col1\tcol2\n" +
"aaa\t123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"some text before magic word %table col1\tcol2\naaa\t123\n %html <h3> This is a hack </h3>");
assertEquals("text & %table befoe %html return %table","col1\tcol2\n" +
"aaa\t123\n" +
" %html <h3> This is a hack </h3>", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%table col1\tcol2\naaa\t123\n %html <h3> This is a hack </h3> %table col1\naaa\n123\n");
assertEquals("%table, %html before %table return first %table","col1\tcol2\n" +
"aaa\t123\n" +
" %html <h3> This is a hack </h3> %table col1\n" +
"aaa\n" +
"123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS,"%table col1\tcol2\naaa\t123\n %table col1\naaa\n123\n");
assertEquals("%table before %table return first %table","col1\tcol2\n" +
"aaa\t123\n" +
" %table col1\n" +
"aaa\n" +
"123\n", result.message());
}
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%table col1\tcol2\naaa\t123\n");
assertEquals(InterpreterResult.Type.TABLE, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%table\ncol1\tcol2\naaa\t123\n");
assertEquals(InterpreterResult.Type.TABLE, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "some text before magic word %table col1\tcol2\naaa\t123\n");
assertEquals(InterpreterResult.Type.TABLE, result.type());
}
@Test
public void testToString() {
assertEquals("%html hello", new InterpreterResult(InterpreterResult.Code.SUCCESS, "%html hello").toString());
}
public void testComplexMagicType() {
InterpreterResult result = null;
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "some text before %table col1\tcol2\naaa\t123\n");
assertEquals("some text before magic return magic", InterpreterResult.Type.TABLE, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%html <h3> This is a hack </h3> %table\n col1\tcol2\naaa\t123\n");
assertEquals("magic A before magic B return magic A", InterpreterResult.Type.HTML, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "some text before magic word %table col1\tcol2\naaa\t123\n %html <h3> This is a hack </h3>");
assertEquals("text & magic A before magic B return magic A", InterpreterResult.Type.TABLE, result.type());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%table col1\tcol2\naaa\t123\n %html <h3> This is a hack </h3> %table col1\naaa\n123\n");
assertEquals("magic A, magic B, magic a' return magic A", InterpreterResult.Type.TABLE, result.type());
}
@Test
public void testSimpleMagicData() {
InterpreterResult result = null;
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%table col1\tcol2\naaa\t123\n");
assertEquals("%table col1\tcol2\naaa\t123\n", "col1\tcol2\naaa\t123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%table\ncol1\tcol2\naaa\t123\n");
assertEquals("%table\ncol1\tcol2\naaa\t123\n", "col1\tcol2\naaa\t123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "some text before magic word %table col1\tcol2\naaa\t123\n");
assertEquals("some text before magic word %table col1\tcol2\naaa\t123\n", "col1\tcol2\naaa\t123\n", result.message());
}
@Test
public void testComplexMagicData() {
InterpreterResult result = null;
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "some text before %table col1\tcol2\naaa\t123\n");
assertEquals("text before %table return %table", "col1\tcol2\naaa\t123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%html <h3> This is a hack </h3> %table\ncol1\tcol2\naaa\t123\n");
assertEquals("%html before %table return %html", " <h3> This is a hack </h3> %table\n" +
"col1\tcol2\n" +
"aaa\t123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "some text before magic word %table col1\tcol2\naaa\t123\n %html <h3> This is a hack </h3>");
assertEquals("text & %table befoe %html return %table", "col1\tcol2\n" +
"aaa\t123\n" +
" %html <h3> This is a hack </h3>", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%table col1\tcol2\naaa\t123\n %html <h3> This is a hack </h3> %table col1\naaa\n123\n");
assertEquals("%table, %html before %table return first %table", "col1\tcol2\n" +
"aaa\t123\n" +
" %html <h3> This is a hack </h3> %table col1\n" +
"aaa\n" +
"123\n", result.message());
result = new InterpreterResult(InterpreterResult.Code.SUCCESS, "%table col1\tcol2\naaa\t123\n %table col1\naaa\n123\n");
assertEquals("%table before %table return first %table", "col1\tcol2\n" +
"aaa\t123\n" +
" %table col1\n" +
"aaa\n" +
"123\n", result.message());
}
@Test
public void testToString() {
assertEquals("%html hello", new InterpreterResult(InterpreterResult.Code.SUCCESS, "%html hello").toString());
}
}

8
zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java
View file

@ -779,7 +779,8 @@ public class NotebookRestApi {
LOG.info("Get notebook jobs for job manager");
AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
List<Map<String, Object>> notebookJobs = notebook.getJobListforNotebook(false, 0, subject);
List<Map<String, Object>> notebookJobs = notebook
.getJobListByUnixTime(false, 0, subject);
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
@ -791,6 +792,8 @@ public class NotebookRestApi {
/**
* Get updated notebook jobs for job manager
*
* Return the `Note` change information within the post unix timestamp.
*
* @return JSON with status.OK
* @throws IOException, IllegalArgumentException
*/
@ -804,7 +807,7 @@ public class NotebookRestApi {
List<Map<String, Object>> notebookJobs;
AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
notebookJobs = notebook.getJobListforNotebook(false, lastUpdateUnixTime, subject);
notebookJobs = notebook.getJobListByUnixTime(false, lastUpdateUnixTime, subject);
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
@ -852,7 +855,6 @@ public class NotebookRestApi {
if (paramsForUpdating != null) {
paragraph.settings.getParams().putAll(paramsForUpdating);
AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
note.setLastReplName(paragraph.getId());
note.persist(subject);
}
}

1
zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java
View file

@ -104,6 +104,7 @@ public class ZeppelinServer extends Application {
heliumApplicationFactory.setApplicationEventListener(notebookWsServer);
notebook.addNotebookEventListener(heliumApplicationFactory);
notebook.addNotebookEventListener(notebookWsServer.getNotebookInformationListener());
}
public static void main(String[] args) throws InterruptedException {

187
zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
View file

@ -241,8 +241,8 @@ public class NotebookServer extends WebSocketServlet implements
case LIST_NOTEBOOK_JOBS:
unicastNotebookJobInfo(conn, messagereceived);
break;
case LIST_UPDATE_NOTEBOOK_JOBS:
unicastUpdateNotebookJobInfo(conn, messagereceived);
case UNSUBSCRIBE_UPDATE_NOTEBOOK_JOBS:
unsubscribeNotebookJobInfo(conn);
break;
case GET_INTERPRETER_BINDINGS:
getInterpreterBindings(conn, messagereceived);
@ -250,6 +250,9 @@ public class NotebookServer extends WebSocketServlet implements
case SAVE_INTERPRETER_BINDINGS:
saveInterpreterBindings(conn, messagereceived);
break;
case EDITOR_SETTING:
getEditorSetting(conn, messagereceived);
break;
default:
break;
}
@ -398,9 +401,10 @@ public class NotebookServer extends WebSocketServlet implements
}
public void unicastNotebookJobInfo(NotebookSocket conn, Message fromMessage) throws IOException {
addConnectionToNote(JOB_MANAGER_SERVICE.JOB_MANAGER_PAGE.getKey(), conn);
AuthenticationInfo subject = new AuthenticationInfo(fromMessage.principal);
List<Map<String, Object>> notebookJobs = notebook().getJobListforNotebook(false, 0, subject);
List<Map<String, Object>> notebookJobs = notebook()
.getJobListByUnixTime(false, 0, subject);
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
@ -410,21 +414,25 @@ public class NotebookServer extends WebSocketServlet implements
.put("notebookJobs", response)));
}
public void unicastUpdateNotebookJobInfo(NotebookSocket conn, Message fromMessage)
throws IOException {
double lastUpdateUnixTimeRaw = (double) fromMessage.get("lastUpdateUnixTime");
long lastUpdateUnixTime = new Double(lastUpdateUnixTimeRaw).longValue();
List<Map<String, Object>> notebookJobs;
AuthenticationInfo subject = new AuthenticationInfo(fromMessage.principal);
notebookJobs = notebook().getJobListforNotebook(false, lastUpdateUnixTime, subject);
public void broadcastUpdateNotebookJobInfo(long lastUpdateUnixTime) throws IOException {
List<Map<String, Object>> notebookJobs = new LinkedList<>();
Notebook notebookObject = notebook();
List<Map<String, Object>> jobNotes = null;
if (notebookObject != null) {
jobNotes = notebook().getJobListByUnixTime(false, lastUpdateUnixTime, null);
notebookJobs = jobNotes == null ? notebookJobs : jobNotes;
}
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
response.put("jobs", notebookJobs);
response.put("jobs", notebookJobs != null ? notebookJobs : new LinkedList<>());
conn.send(serializeMessage(new Message(OP.LIST_UPDATE_NOTEBOOK_JOBS)
.put("notebookRunningJobs", response)));
broadcast(JOB_MANAGER_SERVICE.JOB_MANAGER_PAGE.getKey(),
new Message(OP.LIST_UPDATE_NOTEBOOK_JOBS).put("notebookRunningJobs", response));
}
public void unsubscribeNotebookJobInfo(NotebookSocket conn) {
removeConnectionFromNote(JOB_MANAGER_SERVICE.JOB_MANAGER_PAGE.getKey(), conn);
}
public void saveInterpreterBindings(NotebookSocket conn, Message fromMessage) {
@ -467,8 +475,7 @@ public class NotebookServer extends WebSocketServlet implements
LOG.error("Fail to reload notes from repository", e);
}
}
List<Note> notes = notebook.getAllNotes();
List<Note> notes = notebook.getAllNotes(subject);
List<Map<String, String>> notesInfo = new LinkedList<>();
for (Note note : notes) {
Map<String, String> info = new HashMap<>();
@ -719,7 +726,10 @@ public class NotebookServer extends WebSocketServlet implements
if (fromMessage != null) {
String noteName = (String) ((Map) fromMessage.get("notebook")).get("name");
String noteJson = gson.toJson(fromMessage.get("notebook"));
AuthenticationInfo subject = new AuthenticationInfo(fromMessage.principal);
AuthenticationInfo subject = null;
if (fromMessage.principal != null) {
subject = new AuthenticationInfo(fromMessage.principal);
}
note = notebook.importNote(noteJson, noteName, subject);
note.persist(subject);
broadcastNote(note);
@ -1130,10 +1140,8 @@ public class NotebookServer extends WebSocketServlet implements
.get("config");
p.setConfig(config);
// if it's the last paragraph, let's add a new one
boolean isTheLastParagraph = note.getLastParagraph().getId()
.equals(p.getId());
note.setLastReplName(paragraphId);
if (!(text.equals(note.getLastInterpreterName() + " ") || Strings.isNullOrEmpty(text)) &&
boolean isTheLastParagraph = note.isLastParagraph(p.getId());
if (!(text.trim().equals(p.getMagic()) || Strings.isNullOrEmpty(text)) &&
isTheLastParagraph) {
note.addParagraph();
}
@ -1293,6 +1301,113 @@ public class NotebookServer extends WebSocketServlet implements
broadcast(noteId, msg);
}
/**
* Notebook Information Change event
*/
public static class NotebookInformationListener implements NotebookEventListener {
private NotebookServer notebookServer;
public NotebookInformationListener(NotebookServer notebookServer) {
this.notebookServer = notebookServer;
}
@Override
public void onParagraphRemove(Paragraph p) {
try {
notebookServer.broadcastUpdateNotebookJobInfo(System.currentTimeMillis() - 5000);
} catch (IOException ioe) {
LOG.error("can not broadcast for job manager {}", ioe.getMessage());
}
}
@Override
public void onNoteRemove(Note note) {
try {
notebookServer.broadcastUpdateNotebookJobInfo(System.currentTimeMillis() - 5000);
} catch (IOException ioe) {
LOG.error("can not broadcast for job manager {}", ioe.getMessage());
}
List<Map<String, Object>> notesInfo = new LinkedList<>();
Map<String, Object> info = new HashMap<>();
info.put("notebookId", note.getId());
// set paragraphs
List<Map<String, Object>> paragraphsInfo = new LinkedList<>();
// notebook json object root information.
info.put("isRunningJob", false);
info.put("unixTimeLastRun", 0);
info.put("isRemoved", true);
info.put("paragraphs", paragraphsInfo);
notesInfo.add(info);
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
response.put("jobs", notesInfo);
notebookServer.broadcast(JOB_MANAGER_SERVICE.JOB_MANAGER_PAGE.getKey(),
new Message(OP.LIST_UPDATE_NOTEBOOK_JOBS).put("notebookRunningJobs", response));
}
@Override
public void onParagraphCreate(Paragraph p) {
Notebook notebook = notebookServer.notebook();
List<Map<String, Object>> notebookJobs = notebook.getJobListByParagraphId(
p.getId()
);
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
response.put("jobs", notebookJobs);
notebookServer.broadcast(JOB_MANAGER_SERVICE.JOB_MANAGER_PAGE.getKey(),
new Message(OP.LIST_UPDATE_NOTEBOOK_JOBS).put("notebookRunningJobs", response));
}
@Override
public void onNoteCreate(Note note) {
Notebook notebook = notebookServer.notebook();
List<Map<String, Object>> notebookJobs = notebook.getJobListBymNotebookId(
note.getId()
);
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
response.put("jobs", notebookJobs);
notebookServer.broadcast(JOB_MANAGER_SERVICE.JOB_MANAGER_PAGE.getKey(),
new Message(OP.LIST_UPDATE_NOTEBOOK_JOBS).put("notebookRunningJobs", response));
}
@Override
public void onParagraphStatusChange(Paragraph p, Status status) {
Notebook notebook = notebookServer.notebook();
List<Map<String, Object>> notebookJobs = notebook.getJobListByParagraphId(
p.getId()
);
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
response.put("jobs", notebookJobs);
notebookServer.broadcast(JOB_MANAGER_SERVICE.JOB_MANAGER_PAGE.getKey(),
new Message(OP.LIST_UPDATE_NOTEBOOK_JOBS).put("notebookRunningJobs", response));
}
@Override
public void onUnbindInterpreter(Note note, InterpreterSetting setting) {
Notebook notebook = notebookServer.notebook();
List<Map<String, Object>> notebookJobs = notebook.getJobListBymNotebookId(
note.getId()
);
Map<String, Object> response = new HashMap<>();
response.put("lastResponseUnixTime", System.currentTimeMillis());
response.put("jobs", notebookJobs);
notebookServer.broadcast(JOB_MANAGER_SERVICE.JOB_MANAGER_PAGE.getKey(),
new Message(OP.LIST_UPDATE_NOTEBOOK_JOBS).put("notebookRunningJobs", response));
}
}
/**
* Need description here.
*
@ -1336,10 +1451,16 @@ public class NotebookServer extends WebSocketServlet implements
}
}
notebookServer.broadcastNote(note);
try {
notebookServer.broadcastUpdateNotebookJobInfo(System.currentTimeMillis() - 5000);
} catch (IOException e) {
LOG.error("can not broadcast for job manager {}", e);
}
}
/**
* This callback is for praragraph that runs on RemoteInterpreterProcess
* This callback is for paragraph that runs on RemoteInterpreterProcess
* @param paragraph
* @param out
* @param output
@ -1376,6 +1497,10 @@ public class NotebookServer extends WebSocketServlet implements
return new ParagraphListenerImpl(this, note);
}
public NotebookEventListener getNotebookInformationListener() {
return new NotebookInformationListener(this);
}
private void sendAllAngularObjects(Note note, NotebookSocket conn) throws IOException {
List<InterpreterSetting> settings =
notebook().getInterpreterFactory().getInterpreterSettings(note.getId());
@ -1447,11 +1572,23 @@ public class NotebookServer extends WebSocketServlet implements
if (id.equals(interpreterGroupId)) {
broadcast(
note.getId(),
new Message(OP.ANGULAR_OBJECT_REMOVE).put("name", name).put(
"noteId", noteId).put("paragraphId", paragraphId));
new Message(OP.ANGULAR_OBJECT_REMOVE)
.put("name", name)
.put("noteId", noteId)
.put("paragraphId", paragraphId));
}
}
}
}
private void getEditorSetting(NotebookSocket conn, Message fromMessage)
throws IOException {
String replName = (String) fromMessage.get("magic");
String noteId = getOpenNoteId(conn);
Message resp = new Message(OP.EDITOR_SETTING);
resp.put("editor", notebook().getInterpreterFactory().getEditorSetting(noteId, replName));
conn.send(serializeMessage(resp));
return;
}
}

25
zeppelin-server/src/test/java/org/apache/zeppelin/integration/AuthenticationIT.java
View file

@ -36,6 +36,7 @@ import org.slf4j.LoggerFactory;
import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.util.List;
@ -176,8 +177,16 @@ public class AuthenticationIT extends AbstractZeppelinIT {
authenticationIT.logoutUser("finance1");
authenticationIT.authenticationUser("hr1", "hr1");
pollingWait(By.xpath("//*[@id='notebook-names']//a[contains(@href, '" + noteId + "')]"),
MAX_BROWSER_TIMEOUT_SEC).click();
try {
WebElement element = pollingWait(By.xpath("//*[@id='notebook-names']//a[contains(@href, '" + noteId + "')]"),
MAX_BROWSER_TIMEOUT_SEC);
collector.checkThat("Check is user has permission to view this notebook link", false,
CoreMatchers.equalTo(element.isDisplayed()));
} catch (Exception e) {
//This should have failed, nothing to worry.
}
driver.get(new URI(driver.getCurrentUrl()).resolve("/#/notebook/" + noteId).toString());
List<WebElement> privilegesModal = driver.findElements(
By.xpath("//div[@class='modal-content']//div[@class='bootstrap-dialog-header']" +
@ -190,8 +199,16 @@ public class AuthenticationIT extends AbstractZeppelinIT {
authenticationIT.logoutUser("hr1");
authenticationIT.authenticationUser("finance2", "finance2");
pollingWait(By.xpath("//*[@id='notebook-names']//a[contains(@href, '" + noteId + "')]"),
MAX_BROWSER_TIMEOUT_SEC).click();
try {
WebElement element = pollingWait(By.xpath("//*[@id='notebook-names']//a[contains(@href, '" + noteId + "')]"),
MAX_BROWSER_TIMEOUT_SEC);
collector.checkThat("Check is user has permission to view this notebook link", false,
CoreMatchers.equalTo(element.isDisplayed()));
} catch (Exception e) {
//This should have failed, nothing to worry.
}
driver.get(new URI(driver.getCurrentUrl()).resolve("/#/notebook/" + noteId).toString());
privilegesModal = driver.findElements(
By.xpath("//div[@class='modal-content']//div[@class='bootstrap-dialog-header']" +

11
zeppelin-server/src/test/java/org/apache/zeppelin/integration/ParagraphActionsIT.java
View file

@ -95,28 +95,21 @@ public class ParagraphActionsIT extends AbstractZeppelinIT {
ZeppelinITUtils.sleep(1000, false);
waitForParagraph(1, "READY");
String oldIntpTag = driver.findElement(By.xpath(getParagraphXPath(1) + "//div[contains(@class, 'editor')]")).getText();
collector.checkThat("Paragraph is created above",
driver.findElement(By.xpath(getParagraphXPath(1) + "//div[contains(@class, 'editor')]")).getText(),
CoreMatchers.not(StringUtils.EMPTY));
CoreMatchers.equalTo(StringUtils.EMPTY));
setTextOfParagraph(1, " this is above ");
newPara = driver.findElement(By.xpath(getParagraphXPath(2) + "//div[contains(@class,'new-paragraph')][2]"));
action.moveToElement(newPara).click().build().perform();
waitForParagraph(3, "READY");
String lastIntpTag = driver.findElement(By.xpath(getParagraphXPath(3) + "//div[contains(@class, 'editor')]")).getText();
collector.checkThat("Paragraph is created below",
driver.findElement(By.xpath(getParagraphXPath(3) + "//div[contains(@class, 'editor')]")).getText(),
CoreMatchers.not(StringUtils.EMPTY));
CoreMatchers.equalTo(StringUtils.EMPTY));
setTextOfParagraph(3, " this is below ");
collector.checkThat("Compare interpreter name tag", oldIntpTag, CoreMatchers.equalTo(lastIntpTag));
collector.checkThat("The output field of paragraph1 contains",
driver.findElement(By.xpath(getParagraphXPath(1) + "//div[contains(@class, 'editor')]")).getText(),
CoreMatchers.equalTo(" this is above "));

16
zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
View file

@ -154,7 +154,7 @@ public abstract class AbstractTestRestApi {
// set spark master and other properties
sparkIntpSetting.getProperties().setProperty("master", "spark://" + getHostname() + ":7071");
sparkIntpSetting.getProperties().setProperty("spark.cores.max", "2");
sparkIntpSetting.getProperties().setProperty("zeppelin.spark.useHiveContext", "false");
// set spark home for pyspark
sparkIntpSetting.getProperties().setProperty("spark.home", getSparkHome());
pySpark = true;
@ -171,10 +171,16 @@ public abstract class AbstractTestRestApi {
String sparkHome = getSparkHome();
if (sparkHome != null) {
sparkIntpSetting.getProperties().setProperty("master", "spark://" + getHostname() + ":7071");
if (System.getenv("SPARK_MASTER") != null) {
sparkIntpSetting.getProperties().setProperty("master", System.getenv("SPARK_MASTER"));
} else {
sparkIntpSetting.getProperties()
.setProperty("master", "spark://" + getHostname() + ":7071");
}
sparkIntpSetting.getProperties().setProperty("spark.cores.max", "2");
// set spark home for pyspark
sparkIntpSetting.getProperties().setProperty("spark.home", sparkHome);
sparkIntpSetting.getProperties().setProperty("zeppelin.spark.useHiveContext", "false");
pySpark = true;
sparkR = true;
}
@ -194,7 +200,11 @@ public abstract class AbstractTestRestApi {
}
private static String getSparkHome() {
String sparkHome = getSparkHomeRecursively(new File(System.getProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_HOME.getVarName())));
String sparkHome = System.getenv("SPARK_HOME");
if (sparkHome != null) {
return sparkHome;
}
sparkHome = getSparkHomeRecursively(new File(System.getProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_HOME.getVarName())));
System.out.println("SPARK HOME detected " + sparkHome);
return sparkHome;
}

3
zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinRestApiTest.java
View file

@ -146,8 +146,7 @@ public class ZeppelinRestApiTest extends AbstractTestRestApi {
assertEquals("compare note name", expectedNoteName, newNoteName);
assertEquals("initial paragraph check failed", 3, newNote.getParagraphs().size());
for (Paragraph p : newNote.getParagraphs()) {
if (StringUtils.isEmpty(p.getText()) ||
p.getText().trim().equals(newNote.getLastInterpreterName())) {
if (StringUtils.isEmpty(p.getText())) {
continue;
}
assertTrue("paragraph title check failed", p.getTitle().startsWith("title"));

32
zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
View file

@ -135,11 +135,38 @@ public class ZeppelinSparkClusterTest extends AbstractTestRestApi {
config.put("enabled", true);
p.setConfig(config);
p.setText("%pyspark print(sc.parallelize(range(1, 11)).reduce(lambda a, b: a + b))");
// p.getRepl("org.apache.zeppelin.spark.SparkInterpreter").open();
note.run(p.getId());
waitForFinish(p);
assertEquals(Status.FINISHED, p.getStatus());
assertEquals("55\n", p.getResult().message());
if (sparkVersion >= 13) {
// run sqlContext test
p = note.addParagraph();
config = p.getConfig();
config.put("enabled", true);
p.setConfig(config);
p.setText("%pyspark from pyspark.sql import Row\n" +
"df=sqlContext.createDataFrame([Row(id=1, age=20)])\n" +
"df.collect()");
note.run(p.getId());
waitForFinish(p);
assertEquals(Status.FINISHED, p.getStatus());
assertEquals("[Row(age=20, id=1)]\n", p.getResult().message());
}
if (sparkVersion >= 20) {
// run SparkSession test
p = note.addParagraph();
config = p.getConfig();
config.put("enabled", true);
p.setConfig(config);
p.setText("%pyspark from pyspark.sql import Row\n" +
"df=sqlContext.createDataFrame([Row(id=1, age=20)])\n" +
"df.collect()");
note.run(p.getId());
waitForFinish(p);
assertEquals(Status.FINISHED, p.getStatus());
assertEquals("[Row(age=20, id=1)]\n", p.getResult().message());
}
}
ZeppelinServer.notebook.removeNote(note.getId(), null);
}
@ -166,7 +193,6 @@ public class ZeppelinSparkClusterTest extends AbstractTestRestApi {
p.setText("%pyspark\nfrom pyspark.sql.functions import *\n"
+ "print(" + sqlContextName + ".range(0, 10).withColumn('uniform', rand(seed=10) * 3.14).count())");
// p.getRepl("org.apache.zeppelin.spark.SparkInterpreter").open();
note.run(p.getId());
waitForFinish(p);
assertEquals(Status.FINISHED, p.getStatus());
@ -257,6 +283,7 @@ public class ZeppelinSparkClusterTest extends AbstractTestRestApi {
assertEquals(Status.FINISHED, p1.getStatus());
assertEquals("2\n", p1.getResult().message());
}
ZeppelinServer.notebook.removeNote(note.getId(), null);
}
/**
@ -270,7 +297,6 @@ public class ZeppelinSparkClusterTest extends AbstractTestRestApi {
config.put("enabled", true);
p.setConfig(config);
p.setText("%spark print(sc.version)");
// p.getRepl("org.apache.zeppelin.spark.SparkInterpreter").open();
note.run(p.getId());
waitForFinish(p);
assertEquals(Status.FINISHED, p.getStatus());

2
zeppelin-server/src/test/resources/log4j.properties
View file

@ -43,4 +43,4 @@ log4j.logger.DataNucleus.Datastore=ERROR
# Log all JDBC parameters
log4j.logger.org.hibernate.type=ALL
log4j.logger.org.apache.zeppelin.interpreter.remote.RemoteInterpreter=DEBUG

127
zeppelin-web/CONTRIBUTING.md
View file

@ -1,127 +0,0 @@
# Contributing to Zeppelin-Web
## Dev Mode
When working on Zeppelin's WebApplication, it is recommended to run in dev mode.
For that, start Zeppelin server normally, then use ``./grunt serve`` in _zeppelin-web_ directory.
This will launch a Zeppelin WebApplication on port **9000** that will update on code changes.
## Technologies
Zeppelin WebApplication is using **AngularJS** as main Framework, and **Grunt** and **Bower** as helpers.
So you might want to get familiar with it.
[Here is a good start](http://www.sitepoint.com/kickstart-your-angularjs-development-with-yeoman-grunt-and-bower/)
(There is obviously plenty more ressources to learn)
## Coding style
* We follow mainly the [Google Javascript Guide](https://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml)
* We use a 2 spaces indentation
* We use single quotes
But don't worry, Eslint and Jscs will make you remember it for the most part.
We try not to have **JQuery except in directives**, If you want to include a library,
please search for its **angularJS** directive first.
If you still need to use it, then please use ``angular.element()`` instead of ``$()``
## Folder Structure & Code Organization
* `src` folder: Contains the Source code for Zeppelin WebApplication
* `dist` folder: Contains the compiled code after using **grunt build**
### Src and Code Organization
The `src` folder is organized as such:
<pre>
src/
├── app/
│ ├── name/
│ │ ├── name.controller.js
| | ├── name.html
| | ├── subComponent1/
| | | ├── subComponent1.html
| | | ├── subComponent1.css
│ | | └── subComponent1.controller.js
│ │ └── name.css
│ └── app.js
├── assets/
│ ├── images/
│ └── styles/
| ├── looknfeel/
│ └── printMode.css
├── components/
│ ├── component1/
| | ├── component1.html
│ | └── component1.controller.js
│ └── component2/
├── fonts/
| ├── *.{eot,svg,ttf,woff,otf}
│ └── *.css
├── favico.ico
├── index.html
└── 404.html
</pre>
The code is now organized in a component type of architecture, where everything is logically grouped.
#### File type name convention
In order to understand what is contained inside the .js files without opening it, we use some name conventions:
* .controller.js
* .directive.js
* .service.js
### Component Architecture
When we talk about Component architecture, we think about grouping files together in a logical way.
A component can then be made of multiple files like `.html`, `.css` or any other file type mentioned above.
Related components can be grouped as sub-component as long as they are used in that component only.
#### App folder
Contains the application `app.js` and page related components.
* Home Page
* Interpreter Page
* Notebook Page
etc...
The only resctiction being that a component in the `app` folder is **not used anywhere else**
#### Components folder
The `components` folder is here to contains any reusable component (used more than once)
### Fonts
Fonts files and their css are mixed together in the `fonts` folder
## New files includes
As we do not use yeoman to generate controllers or other type of files with this new structure,
we need to do some includes manually in `index.html` in order to use dev mode and compile correctly.
* Non-bower `.js` files needs to be injected between the tag `<!-- build:js({.tmp,src}) scripts/scripts.js -->`
* Css files needs to be injected between the tag `<!-- build:css(.tmp) styles/main.css -->`
## Add plugins with Bower
```
bower install <plugin> --save
```
The file index.html will automatically update with the new bower_component
<br/>
**Example**: `./bower install angular-nvd3`
You should find that line in the index.html file
```
<script src="bower_components/angular-nvd3/dist/angular-nvd3.js"></script>
````

2
zeppelin-web/README.md
View file

@ -61,4 +61,4 @@ also try to add proxy info to npm install command:
and retry to build again.
## Contribute on Zeppelin Web
If you wish to help us and contribute to Zeppelin WebApplication, please look at [Zeppelin WebApplication's contribution guideline](CONTRIBUTING.md).
If you wish to help us and contribute to Zeppelin WebApplication, please look at the overall project [contribution guidelines](https://zeppelin.apache.org/contribution/contributions.html) and the more focused [Zeppelin WebApplication's documentation](https://zeppelin.apache.org/contribution/webapplication.html).

7
zeppelin-web/bower.json
View file

@ -24,7 +24,7 @@
"angular-elastic": "~2.4.2",
"angular-elastic-input": "~2.2.0",
"angular-xeditable": "0.1.12",
"highlightjs": "^9.2.0",
"highlightjs": "^9.4.0",
"lodash": "~3.9.3",
"angular-filter": "~0.5.4",
"ngtoast": "~2.0.0",
@ -33,7 +33,8 @@
"handsontable": "~0.24.2",
"moment-duration-format": "^1.3.0",
"select2": "^4.0.3",
"angular-esri-map": "~2.0.0"
"angular-esri-map": "~2.0.0",
"github-markdown-css": "^2.4.0"
},
"devDependencies": {
"angular-mocks": "1.5.0"
@ -61,7 +62,7 @@
"highlight.pack.js",
"styles/github.css"
],
"version": "8.4.0",
"version": "9.4.0",
"name": "highlightjs"
}
}

232
zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html
View file

@ -35,8 +35,8 @@ limitations under the License.
</select>
</div>
<b>Option</b>
<div>
<h5>Option</h5>
<span class="btn-group">
<button type="button" class="btn btn-default btn-xs dropdown-toggle"
data-toggle="dropdown">
@ -68,130 +68,144 @@ limitations under the License.
</span>
<span>Interpreter for note</span>
</div>
<br />
<div class="col-md-12" style="padding-left:0px">
<div class="checkbox">
<span class="input-group" style="line-height:30px;">
<label><input type="checkbox" style="width:20px" ng-model="newInterpreterSetting.option.isExistingProcess"/> Connect to existing process </label>
</span>
</div>
</div>
<div ng-show="newInterpreterSetting.option.isExistingProcess" class="form-group" style="width:200px">
<b>Host</b>
<input id="newInterpreterSettingHost" input pu-elastic-input
pu-elastic-input-minwidth="180px" ng-model="newInterpreterSetting.option.host" />
</div>
<div ng-show="newInterpreterSetting.option.isExistingProcess" class="form-group" style="width:200px">
<b>Port</b>
<input id="newInterpreterSettingPort" input pu-elastic-input
pu-elastic-input-minwidth="180px" ng-model="newInterpreterSetting.option.port" />
</div>
<div class="col-md-12">
<div class="checkbox">
<div class="row interpreter" style="margin-top: 5px;">
<div class="col-md-12">
<div class="checkbox remove-margin-top-bottom">
<span class="input-group" style="line-height:30px;">
<label><input type="checkbox" style="width:18px !important" id="idShowPermission" ng-click="togglePermissions('newInterpreter')" ng-model="newInterpreterSetting.option.setPermission"/>
Set permission </label>
<label>
<input type="checkbox" style="width:20px" id="isExistingProcess" ng-model="newInterpreterSetting.option.isExistingProcess"/>
Connect to existing process
</label>
</span>
</div>
</div>
</div>
<div class="row interpreter" ng-show="newInterpreterSetting.option.isExistingProcess" >
<div class="col-md-12">
<b>Host</b>
<input id="newInterpreterSettingHost" input pu-elastic-input
pu-elastic-input-minwidth="180px" ng-model="newInterpreterSetting.option.host"/>
</div>
<div class="col-md-12">
<b>Port</b>
<input id="newInterpreterSettingPort" input pu-elastic-input
pu-elastic-input-minwidth="180px" ng-model="newInterpreterSetting.option.port"/>
</div>
</div>
<div class="row interpreter">
<div class="col-md-12">
<div class="checkbox remove-margin-top-bottom">
<span class="input-group" style="line-height:30px;">
<label>
<input type="checkbox" style="width:20px !important" id="idShowPermission" ng-click="togglePermissions('newInterpreter')" ng-model="newInterpreterSetting.option.setPermission"/>
Set permission
</label>
</span>
</div>
</div>
</div>
<br/>
<div class="col-md-12">
<!-- permissions -->
<div ng-show="newInterpreterSetting.option.setPermission" class="permissionsForm">
<div>
<p>
Enter comma separated users in the fields. <br />
Empty field (*) implies anyone can run this interpreter.
</p>
<div class="row interpreter">
<div class="col-md-12">
<!-- permissions -->
<div ng-show="newInterpreterSetting.option.setPermission" class="permissionsForm">
<div>
<span class="owners">Owners </span>
<select id="newInterpreterUsers" class="form-control" multiple="multiple">
<option ng-repeat="user in newInterpreterSetting.option.users" selected="selected">{{user}}</option>
</select>
<p>
Enter comma separated users in the fields. <br />
Empty field (*) implies anyone can run this interpreter.
</p>
<div>
<span class="owners">Owners </span>
<select id="newInterpreterUsers" class="form-control" multiple="multiple">
<option ng-repeat="user in newInterpreterSetting.option.users" selected="selected">{{user}}</option>
</select>
</div>
</div>
</div>
</div>
</div>
<div>
<h5>Properties</h5>
<table class="table table-striped properties">
<tr>
<th>name</th>
<th>value</th>
<th>description</th>
<th>action</th>
</tr>
<tr ng-repeat="(key, value) in newInterpreterSetting.properties">
<td>{{key}}</td>
<td><textarea msd-elastic ng-model="value.value"></textarea></td>
<td>{{value.description}}</td>
<td>
<button class="btn btn-default btn-sm fa fa-remove" ng-click="removeInterpreterProperty(key)">
</button>
</td>
</tr>
<b>Properties</b>
<table class="table table-striped properties">
<tr>
<th>name</th>
<th>value</th>
<th>description</th>
<th>action</th>
</tr>
<tr ng-repeat="(key, value) in newInterpreterSetting.properties">
<td>{{key}}</td>
<td><textarea msd-elastic ng-model="value.value"></textarea></td>
<td>{{value.description}}</td>
<td>
<button class="btn btn-default btn-sm fa fa-remove" ng-click="removeInterpreterProperty(key)">
</button>
</td>
</tr>
<tr>
<td>
<input pu-elastic-input pu-elastic-input-minwidth="180px"
ng-model="newInterpreterSetting.propertyKey" />
</td>
<td><textarea msd-elastic ng-model="newInterpreterSetting.propertyValue"></textarea></td>
<td></td>
<td>
<button class="btn btn-default btn-sm fa fa-plus" ng-click="addNewInterpreterProperty()">
</button>
</td>
</tr>
</table>
</div>
<tr>
<td>
<input pu-elastic-input pu-elastic-input-minwidth="180px"
ng-model="newInterpreterSetting.propertyKey" />
</td>
<td><textarea msd-elastic ng-model="newInterpreterSetting.propertyValue"></textarea></td>
<td></td>
<td>
<button class="btn btn-default btn-sm fa fa-plus" ng-click="addNewInterpreterProperty()">
</button>
</td>
</tr>
</table>
<div>
<h5>Dependencies</h5>
<table class="table table-striped properties">
<tr>
<th>artifact</th>
<th>exclude</th>
<th>action</th>
</tr>
<b>Dependencies</b>
<table class="table table-striped properties">
<tr>
<th>artifact</th>
<th>exclude</th>
<th>action</th>
</tr>
<tr ng-repeat="dep in newInterpreterSetting.dependencies">
<td>
<input ng-model="dep.groupArtifactVersion" style="width:100%" />
</td>
<td>
<textarea msd-elastic ng-model="dep.exclusions"
ng-list
placeholder="(Optional) comma separated groupId:artifactId list">
</textarea>
</td>
<td>
<button class="btn btn-default btn-sm fa fa-remove"
ng-click="removeInterpreterDependency(dep.groupArtifactVersion)">
</button>
</td>
</tr>
<tr ng-repeat="dep in newInterpreterSetting.dependencies">
<td>
<input ng-model="dep.groupArtifactVersion" style="width:100%" />
</td>
<td>
<textarea msd-elastic ng-model="dep.exclusions"
ng-list
placeholder="(Optional) comma separated groupId:artifactId list">
</textarea>
</td>
<td>
<button class="btn btn-default btn-sm fa fa-remove"
ng-click="removeInterpreterDependency(dep.groupArtifactVersion)">
</button>
</td>
</tr>
<tr>
<td>
<input ng-model="newInterpreterSetting.depArtifact"
placeholder="groupId:artifactId:version or local file path"
style="width: 100%" />
</td>
<td>
<textarea msd-elastic ng-model="newInterpreterSetting.depExclude"
ng-list
placeholder="(Optional) comma separated groupId:artifactId list">
</textarea>
</td>
<td>
<button class="btn btn-default btn-sm fa fa-plus" ng-click="addNewInterpreterDependency()">
</button>
</td>
</tr>
</table>
<tr>
<td>
<input ng-model="newInterpreterSetting.depArtifact"
placeholder="groupId:artifactId:version or local file path"
style="width: 100%" />
</td>
<td>
<textarea msd-elastic ng-model="newInterpreterSetting.depExclude"
ng-list
placeholder="(Optional) comma separated groupId:artifactId list">
</textarea>
</td>
<td>
<button class="btn btn-default btn-sm fa fa-plus" ng-click="addNewInterpreterDependency()">
</button>
</td>
</tr>
</table>
</div>
<span class="btn btn-primary" ng-click="addNewInterpreterSetting()">
Save

5
zeppelin-web/src/app/interpreter/interpreter.css
View file

@ -109,3 +109,8 @@
color: #000;
opacity: .5;
}
.remove-margin-top-bottom {
margin-top: 0px !important;
margin-bottom: 0px !important;
}

79
zeppelin-web/src/app/interpreter/interpreter.html
View file

@ -38,13 +38,17 @@ limitations under the License.
</div>
</div>
<div class="input-group col-md-4" style="margin-top: 10px">
<input type="text" ng-model="searchInterpreter" class="form-control ng-pristine ng-untouched ng-valid ng-empty" placeholder="Search interpreters"/>
<span class="input-group-btn">
<button type="submit" class="btn btn-default" ng-disabled="!navbar.connected">
<i class="glyphicon glyphicon-search"></i>
</button>
</span>
<div class="row">
<div class="col-md-4">
<div class="input-group" style="margin-top: 10px">
<input type="text" ng-model="searchInterpreter" class="form-control ng-pristine ng-untouched ng-valid ng-empty" placeholder="Search interpreters"/>
<span class="input-group-btn">
<button type="submit" class="btn btn-default" ng-disabled="!navbar.connected">
<i class="glyphicon glyphicon-search"></i>
</button>
</span>
</div>
</div>
</div>
</div>
@ -93,10 +97,9 @@ limitations under the License.
<small>
<span style="display:inline-block" ng-repeat="interpreter in setting.interpreterGroup"
title="{{interpreter.class}}">
<span ng-show="!$first">, </span>
%<span ng-show="!$parent.$first || $first">{{setting.name}}</span
><span ng-show="(!$parent.$first || $first) && !$first">.</span
><span ng-show="!$first">{{interpreter.name}}</span>
<span ng-show="!$first">, </span>%<span ng-show="!$parent.$first || $first">{{setting.name}}</span>
<span ng-show="(!$parent.$first || $first) && !$first">.</span>
<span ng-show="!$first">{{interpreter.name}}</span>
<span ng-show="$parent.$first && $first">(default)</span>
</span>
</small>
@ -121,7 +124,7 @@ limitations under the License.
</small>
</h3>
<span style="float:right">
<span style="float:right" ng-show="!valueform.$visible" >
<button class="btn btn-default btn-xs"
ng-click="valueform.$show();
copyOriginInterpreterSettingProperties(setting.id)">
@ -131,7 +134,7 @@ limitations under the License.
<span class="fa fa-refresh"></span> restart</button>
<button class="btn btn-default btn-xs"
ng-click="removeInterpreterSetting(setting.id)">
<span class="fa fa-remove"></span> remove</button>
<span class="fa fa-trash"></span> remove</button>
</span>
</div>
</div>
@ -170,37 +173,44 @@ limitations under the License.
</span>
<span>Interpreter for note</span>
</div>
<br />
</div>
<div class="row interpreter" style="margin-top: 5px;">
<div class="col-md-12">
<div class="checkbox">
<div class="checkbox remove-margin-top-bottom">
<span class="input-group" style="line-height:30px;">
<label><input type="checkbox" style="width:20px" id="isExistingProcess" ng-model="setting.option.isExistingProcess" ng-disabled="!valueform.$visible"/>
Connect to existing process </label>
<label>
<input type="checkbox" style="width:20px" id="isExistingProcess" ng-model="setting.option.isExistingProcess" ng-disabled="!valueform.$visible"/>
Connect to existing process
</label>
</span>
</div>
</div>
<div class="col-md-12" ng-show="setting.option.isExistingProcess">
</div>
<div class="row interpreter" ng-if="setting.option.isExistingProcess">
<div class="col-md-12">
<b>Host</b>
<input id="newInterpreterSettingHost" input pu-elastic-input
pu-elastic-input-minwidth="180px" ng-model="setting.option.host" ng-disabled="!valueform.$visible" />
</div>
<div class="col-md-12" ng-show="setting.option.isExistingProcess">
<div class="col-md-12">
<b>Port</b>
<input id="newInterpreterSettingPort" input pu-elastic-input
pu-elastic-input-minwidth="180px" ng-model="setting.option.port" ng-disabled="!valueform.$visible" />
</div>
</div>
<div class="row interpreter">
<div class="col-md-12">
<div class="checkbox">
<div class="checkbox remove-margin-top-bottom">
<span class="input-group" style="line-height:30px;">
<label><input type="checkbox" style="width:18px !important" id="idShowPermission" ng-click="togglePermissions(setting.name)" ng-model="setting.option.setPermission" ng-disabled="!valueform.$visible"/>
Set permission </label>
<label>
<input type="checkbox" style="width:20px !important" id="idShowPermission" ng-click="togglePermissions(setting.name)" ng-model="setting.option.setPermission" ng-disabled="!valueform.$visible"/>
Set permission
</label>
</span>
</div>
</div>
</div>
<div class="row interpreter">
<div class="col-md-12">
<!-- permissions -->
<div ng-show="setting.option.setPermission" class="permissionsForm">
@ -219,8 +229,10 @@ limitations under the License.
</div>
</div>
</div>
</div>
<div ng-show="_.isEmpty(setting.properties) && _.isEmpty(setting.dependencies) || valueform.$hidden" class="col-md-12 gray40-message">
<div class="row interpreter" style="margin-top:20px !important">
<div ng-show="_.isEmpty(setting.properties) && _.isEmpty(setting.dependencies) && !valueform.$visible" class="col-md-12 gray40-message">
<em>Currently there are no properties and dependencies set for this interpreter</em>
</div>
<div class="col-md-12" ng-show="!_.isEmpty(setting.properties) || valueform.$visible">
@ -228,9 +240,9 @@ limitations under the License.
<table class="table table-striped">
<thead>
<tr>
<th style="width:30%">name</th>
<th>value</th>
<th ng-if="valueform.$visible">action</th>
<th style="width:40%">name</th>
<th style="width:40%">value</th>
<th style="width:20%" ng-if="valueform.$visible">action</th>
</tr>
</thead>
<tr ng-repeat="key in setting.properties | sortByKey" >
@ -263,7 +275,8 @@ limitations under the License.
</tr>
</table>
</div>
</div>
<div class="row interpreter">
<div class="col-md-12" ng-show="!_.isEmpty(setting.dependencies) || valueform.$visible">
<h5>Dependencies</h5>
<p class="gray40-message" style="font-size:12px" ng-show="valueform.$visible">
@ -272,8 +285,8 @@ limitations under the License.
<thead>
<tr>
<th style="width:40%">artifact</th>
<th>exclude</th>
<th ng-if="valueform.$visible">action</th>
<th style="width:40%">exclude</th>
<th style="width:20%" ng-if="valueform.$visible">action</th>
</tr>
</thead>
<tr ng-repeat="dep in setting.dependencies">

6
zeppelin-web/src/app/jobmanager/jobmanager.controller.js
View file

@ -32,14 +32,8 @@ angular.module('zeppelinWebApp')
$scope.JobInfomationsByFilter = $scope.jobInfomations;
websocketMsgSrv.getNotebookJobsList();
var refreshObj = $interval(function() {
if ($scope.lastJobServerUnixTime !== undefined) {
websocketMsgSrv.getUpdateNotebookJobsList($scope.lastJobServerUnixTime);
}
}, 1000);
$scope.$on('$destroy', function() {
$interval.cancel(refreshObj);
websocketMsgSrv.unsubscribeJobManager();
});
};

2
zeppelin-web/src/app/notebook/notebook.html
View file

@ -26,7 +26,7 @@ limitations under the License.
Bind interpreter for this note.
Click to Bind/Unbind interpreter.
Drag and drop to reorder interpreters. <br />
The first interpreter on the list becomes default. To create/remove interpreters, go to <a href="/#/interpreter">Interpreter</a> menu.
The first interpreter on the list becomes default. To create/remove interpreters, go to <a href="#/interpreter">Interpreter</a> menu.
</p>
<div class="interpreterSettings"

96
zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js
View file

@ -15,13 +15,14 @@
angular.module('zeppelinWebApp').controller('ParagraphCtrl', function($scope, $rootScope, $route, $window,
$routeParams, $location, $timeout, $compile,
$http, websocketMsgSrv, baseUrlSrv, ngToast,
$http, $q, websocketMsgSrv, baseUrlSrv, ngToast,
saveAsService, esriLoader) {
var ANGULAR_FUNCTION_OBJECT_NAME_PREFIX = '_Z_ANGULAR_FUNC_';
$scope.parentNote = null;
$scope.paragraph = null;
$scope.originalText = '';
$scope.editor = null;
$scope.magic = null;
var paragraphScope = $rootScope.$new(true, $rootScope);
@ -78,15 +79,6 @@ angular.module('zeppelinWebApp').controller('ParagraphCtrl', function($scope, $r
var angularObjectRegistry = {};
var editorModes = {
'ace/mode/python': /^%(\w*\.)?(pyspark|python)\s*$/,
'ace/mode/scala': /^%(\w*\.)?spark\s*$/,
'ace/mode/r': /^%(\w*\.)?(r|sparkr|knitr)\s*$/,
'ace/mode/sql': /^%(\w*\.)?\wql/,
'ace/mode/markdown': /^%md/,
'ace/mode/sh': /^%sh/
};
// Controller init
$scope.init = function(newParagraph, note) {
$scope.paragraph = newParagraph;
@ -538,7 +530,7 @@ angular.module('zeppelinWebApp').controller('ParagraphCtrl', function($scope, $r
$scope.aceChanged = function() {
$scope.dirtyText = $scope.editor.getSession().getValue();
$scope.startSaveTimer();
$scope.setParagraphMode($scope.editor.getSession(), $scope.dirtyText, $scope.editor.getCursorPosition());
setParagraphMode($scope.editor.getSession(), $scope.dirtyText, $scope.editor.getCursorPosition());
};
$scope.aceLoaded = function(_editor) {
@ -576,37 +568,11 @@ angular.module('zeppelinWebApp').controller('ParagraphCtrl', function($scope, $r
// not applying emacs key binding while the binding override Ctrl-v. default behavior of paste text on windows.
}
$scope.setParagraphMode = function(session, paragraphText, pos) {
// Evaluate the mode only if the first 30 characters of the paragraph have been modified or the the position is undefined.
if ((typeof pos === 'undefined') || (pos.row === 0 && pos.column < 30)) {
// If paragraph loading, use config value if exists
if ((typeof pos === 'undefined') && $scope.paragraph.config.editorMode) {
session.setMode($scope.paragraph.config.editorMode);
} else {
// Defaults to spark mode
var newMode = 'ace/mode/scala';
// Test first against current mode
var oldMode = session.getMode().$id;
if (!editorModes[oldMode] || !editorModes[oldMode].test(paragraphText)) {
for (var key in editorModes) {
if (key !== oldMode) {
if (editorModes[key].test(paragraphText)) {
$scope.paragraph.config.editorMode = key;
session.setMode(key);
return true;
}
}
}
$scope.paragraph.config.editorMode = newMode;
session.setMode(newMode);
}
}
}
};
var remoteCompleter = {
getCompletions: function(editor, session, pos, prefix, callback) {
if (!$scope.editor.isFocused()) { return;}
if (!$scope.editor.isFocused()) {
return;
}
pos = session.getTextRange(new Range(0, 0, pos.row, pos.column)).length;
var buf = session.getValue();
@ -662,7 +628,7 @@ angular.module('zeppelinWebApp').controller('ParagraphCtrl', function($scope, $r
autoAdjustEditorHeight(_editor.container.id);
});
$scope.setParagraphMode($scope.editor.getSession(), $scope.editor.getSession().getValue());
setParagraphMode($scope.editor.getSession(), $scope.editor.getSession().getValue());
// autocomplete on '.'
/*
@ -737,6 +703,53 @@ angular.module('zeppelinWebApp').controller('ParagraphCtrl', function($scope, $r
}
};
var getAndSetEditorSetting = function(session, interpreterName) {
var deferred = $q.defer();
websocketMsgSrv.getEditorSetting(interpreterName);
$timeout(
$scope.$on('editorSetting', function(event, data) {
deferred.resolve(data);
}
), 1000);
deferred.promise.then(function(editorSetting) {
if (!_.isEmpty(editorSetting.editor)) {
var mode = 'ace/mode/' + editorSetting.editor.language;
$scope.paragraph.config.editorMode = mode;
session.setMode(mode);
}
});
};
var setParagraphMode = function(session, paragraphText, pos) {
// Evaluate the mode only if the the position is undefined
// or the first 30 characters of the paragraph have been modified
// or cursor position is at beginning of second line.(in case user hit enter after typing %magic)
if ((typeof pos === 'undefined') || (pos.row === 0 && pos.column < 30) || (pos.row === 1 && pos.column === 0)) {
// If paragraph loading, use config value if exists
if ((typeof pos === 'undefined') && $scope.paragraph.config.editorMode) {
session.setMode($scope.paragraph.config.editorMode);
} else {
var magic;
// set editor mode to default interpreter syntax if paragraph text doesn't start with '%'
// TODO(mina): dig into the cause what makes interpreterBindings has no element
if (!paragraphText.startsWith('%') && ((typeof pos !== 'undefined') && pos.row === 0 && pos.column === 1) ||
(typeof pos === 'undefined') && $scope.$parent.interpreterBindings.length !== 0) {
magic = $scope.$parent.interpreterBindings[0].name;
getAndSetEditorSetting(session, magic);
} else {
var replNameRegexp = /%(.+?)\s/g;
var match = replNameRegexp.exec(paragraphText);
if (match && $scope.magic !== match[1]) {
magic = match[1].trim();
$scope.magic = magic;
getAndSetEditorSetting(session, magic);
}
}
}
}
};
var autoAdjustEditorHeight = function(id) {
var editor = $scope.editor;
var height = editor.getSession().getScreenLength() * editor.renderer.lineHeight +
@ -2259,7 +2272,6 @@ angular.module('zeppelinWebApp').controller('ParagraphCtrl', function($scope, $r
var noteId = $route.current.pathParams.noteId;
$http.get(baseUrlSrv.getRestApiBase() + '/helium/suggest/' + noteId + '/' + $scope.paragraph.id)
.success(function(data, status, headers, config) {
console.log('Suggested apps %o', data);
$scope.suggestion = data.body;
})
.error(function(err, status, headers, config) {

1
zeppelin-web/src/app/notebook/paragraph/paragraph.css
View file

@ -502,7 +502,6 @@ div.esri-view {
table.table-striped {
border-top: 1px solid #ddd;
margin-top: 20px;
}
.scroll-paragraph-down {

6
zeppelin-web/src/components/websocketEvents/websocketEvents.factory.js
View file

@ -76,8 +76,8 @@ angular.module('zeppelinWebApp').factory('websocketEvents',
action: function(dialog) {
dialog.close();
angular.element('#loginModal').modal({
show: 'true'
});
show: 'true'
});
}
}, {
label: 'Cancel',
@ -97,6 +97,8 @@ angular.module('zeppelinWebApp').factory('websocketEvents',
$rootScope.$broadcast('updateProgress', data);
} else if (op === 'COMPLETION_LIST') {
$rootScope.$broadcast('completionList', data);
} else if (op === 'EDITOR_SETTING') {
$rootScope.$broadcast('editorSetting', data);
} else if (op === 'ANGULAR_OBJECT_UPDATE') {
$rootScope.$broadcast('angularObjectUpdate', data);
} else if (op === 'ANGULAR_OBJECT_REMOVE') {

11
zeppelin-web/src/components/websocketEvents/websocketMsg.service.js
View file

@ -180,6 +180,15 @@ angular.module('zeppelinWebApp').service('websocketMsgSrv', function($rootScope,
});
},
getEditorSetting: function(replName) {
websocketEvents.sendNewEvent({
op: 'EDITOR_SETTING',
data: {
magic: replName
}
});
},
isConnected: function() {
return websocketEvents.isConnected();
},
@ -195,7 +204,7 @@ angular.module('zeppelinWebApp').service('websocketMsgSrv', function($rootScope,
},
unsubscribeJobManager: function() {
websocketEvents.sendNewEvent({op: 'UNSUBSCRIBE_JOBMANAGER'});
websocketEvents.sendNewEvent({op: 'UNSUBSCRIBE_UPDATE_NOTEBOOK_JOBS'});
},
getInterpreterBindings: function(noteID) {

1
zeppelin-web/src/index.html
View file

@ -46,6 +46,7 @@ limitations under the License.
<link rel="stylesheet" href="bower_components/bootstrap3-dialog/dist/css/bootstrap-dialog.min.css" />
<link rel="stylesheet" href="bower_components/pikaday/css/pikaday.css" />
<link rel="stylesheet" href="bower_components/handsontable/dist/handsontable.css" />
<link rel="stylesheet" href="bower_components/github-markdown-css/github-markdown.css" />
<!-- endbower -->
<link rel="stylesheet" href="bower_components/jquery-ui/themes/base/jquery-ui.css" />
<link rel="stylesheet" href="bower_components/select2/dist/css/select2.css" />

9
zeppelin-zengine/pom.xml
View file

@ -252,5 +252,14 @@
<filtering>true</filtering>
</resource>
</resources>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration combine.children="append">
<forkMode>always</forkMode>
</configuration>
</plugin>
</plugins>
</build>
</project>

6
zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
View file

@ -27,6 +27,7 @@ import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.XMLConfiguration;
import org.apache.commons.configuration.tree.ConfigurationNode;
import org.apache.zeppelin.notebook.repo.VFSNotebookRepo;
import org.apache.zeppelin.util.Util;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -102,6 +103,11 @@ public class ZeppelinConfiguration extends XMLConfiguration {
conf = new ZeppelinConfiguration();
}
}
LOG.info("Server Host: " + conf.getServerAddress());
LOG.info("Server Port: " + conf.getServerPort());
LOG.info("Context Path: " + conf.getServerContextPath());
LOG.info("Zeppelin Version: " + Util.getVersion());
return conf;
}

177
zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java
View file

@ -49,6 +49,8 @@ import java.util.Properties;
import java.util.Set;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Maps;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.reflect.TypeToken;
@ -171,32 +173,42 @@ public class InterpreterFactory implements InterpreterGroupFactory {
})) {
String interpreterDirString = interpreterDir.toString();
registerInterpreterFromPath(interpreterDirString, interpreterJson);
registerInterpreterFromResource(cl, interpreterDirString, interpreterJson);
/*
* TODO(jongyoul)
* - Remove these codes below because of legacy code
* - Support ThreadInterpreter
/**
* Register interpreter by the following ordering
* 1. Register it from path {ZEPPELIN_HOME}/interpreter/{interpreter_name}/
* interpreter-setting.json
* 2. Register it from interpreter-setting.json in classpath
* {ZEPPELIN_HOME}/interpreter/{interpreter_name}
* 3. Register it by Interpreter.register
*/
URLClassLoader ccl = new URLClassLoader(recursiveBuildLibList(interpreterDir.toFile()), cl);
for (String className : interpreterClassList) {
try {
// Load classes
Class.forName(className, true, ccl);
Set<String> interpreterKeys = Interpreter.registeredInterpreters.keySet();
for (String interpreterKey : interpreterKeys) {
if (className
.equals(Interpreter.registeredInterpreters.get(interpreterKey).getClassName())) {
Interpreter.registeredInterpreters.get(interpreterKey)
.setPath(interpreterDirString);
logger.info("Interpreter " + interpreterKey + " found. class=" + className);
cleanCl.put(interpreterDirString, ccl);
if (!registerInterpreterFromPath(interpreterDirString, interpreterJson)) {
if (!registerInterpreterFromResource(cl, interpreterDirString, interpreterJson)) {
/*
* TODO(jongyoul)
* - Remove these codes below because of legacy code
* - Support ThreadInterpreter
*/
URLClassLoader ccl = new URLClassLoader(
recursiveBuildLibList(interpreterDir.toFile()), cl);
for (String className : interpreterClassList) {
try {
// Load classes
Class.forName(className, true, ccl);
Set<String> interpreterKeys = Interpreter.registeredInterpreters.keySet();
for (String interpreterKey : interpreterKeys) {
if (className
.equals(Interpreter.registeredInterpreters.get(interpreterKey)
.getClassName())) {
Interpreter.registeredInterpreters.get(interpreterKey)
.setPath(interpreterDirString);
logger.info("Interpreter " + interpreterKey + " found. class=" + className);
cleanCl.put(interpreterDirString, ccl);
}
}
} catch (Throwable t) {
// nothing to do
}
}
} catch (Throwable t) {
// nothing to do
}
}
}
@ -213,7 +225,8 @@ public class InterpreterFactory implements InterpreterGroupFactory {
InterpreterInfo interpreterInfo;
for (RegisteredInterpreter r : Interpreter.registeredInterpreters.values()) {
interpreterInfo =
new InterpreterInfo(r.getClassName(), r.getName(), r.isDefaultInterpreter());
new InterpreterInfo(r.getClassName(), r.getName(), r.isDefaultInterpreter(),
r.getEditor());
add(r.getGroup(), interpreterInfo, convertInterpreterProperties(r.getProperties()),
r.getPath());
}
@ -277,7 +290,7 @@ public class InterpreterFactory implements InterpreterGroupFactory {
return properties;
}
private void registerInterpreterFromResource(ClassLoader cl, String interpreterDir,
private boolean registerInterpreterFromResource(ClassLoader cl, String interpreterDir,
String interpreterJson) throws IOException, RepositoryException {
URL[] urls = recursiveBuildLibList(new File(interpreterDir));
ClassLoader tempClassLoader = new URLClassLoader(urls, cl);
@ -289,10 +302,12 @@ public class InterpreterFactory implements InterpreterGroupFactory {
List<RegisteredInterpreter> registeredInterpreterList =
getInterpreterListFromJson(inputStream);
registerInterpreters(registeredInterpreterList, interpreterDir);
return true;
}
return false;
}
private void registerInterpreterFromPath(String interpreterDir, String interpreterJson)
private boolean registerInterpreterFromPath(String interpreterDir, String interpreterJson)
throws IOException, RepositoryException {
Path interpreterJsonPath = Paths.get(interpreterDir, interpreterJson);
@ -301,7 +316,9 @@ public class InterpreterFactory implements InterpreterGroupFactory {
List<RegisteredInterpreter> registeredInterpreterList =
getInterpreterListFromJson(interpreterJsonPath);
registerInterpreters(registeredInterpreterList, interpreterDir);
return true;
}
return false;
}
private List<RegisteredInterpreter> getInterpreterListFromJson(Path filename)
@ -321,7 +338,7 @@ public class InterpreterFactory implements InterpreterGroupFactory {
for (RegisteredInterpreter registeredInterpreter : registeredInterpreters) {
InterpreterInfo interpreterInfo =
new InterpreterInfo(registeredInterpreter.getClassName(), registeredInterpreter.getName(),
registeredInterpreter.isDefaultInterpreter());
registeredInterpreter.isDefaultInterpreter(), registeredInterpreter.getEditor());
Properties properties = new Properties();
Map<String, InterpreterProperty> p = registeredInterpreter.getProperties();
@ -356,10 +373,11 @@ public class InterpreterFactory implements InterpreterGroupFactory {
fis.close();
String json = sb.toString();
InterpreterInfoSaving info = gson.fromJson(json, InterpreterInfoSaving.class);
InterpreterInfoSaving infoSaving = gson.fromJson(json, InterpreterInfoSaving.class);
for (String k : info.interpreterSettings.keySet()) {
InterpreterSetting setting = info.interpreterSettings.get(k);
for (String k : infoSaving.interpreterSettings.keySet()) {
InterpreterSetting setting = infoSaving.interpreterSettings.get(k);
List<InterpreterInfo> infos = setting.getInterpreterInfos();
// Always use separate interpreter process
// While we decided to turn this feature on always (without providing
@ -377,15 +395,23 @@ public class InterpreterFactory implements InterpreterGroupFactory {
depClassPath = interpreterSettingObject.getPath();
setting.setPath(depClassPath);
for (InterpreterInfo info : infos) {
if (info.getEditor() == null) {
Map<String, Object> editor = getEditorFromSettingByClassName(interpreterSettingObject,
info.getClassName());
info.setEditor(editor);
}
}
setting.setInterpreterGroupFactory(this);
loadInterpreterDependencies(setting);
interpreterSettings.put(k, setting);
}
this.interpreterBindings = info.interpreterBindings;
this.interpreterBindings = infoSaving.interpreterBindings;
if (info.interpreterRepositories != null) {
for (RemoteRepository repo : info.interpreterRepositories) {
if (infoSaving.interpreterRepositories != null) {
for (RemoteRepository repo : infoSaving.interpreterRepositories) {
if (!depResolver.getRepos().contains(repo)) {
this.interpreterRepositories.add(repo);
}
@ -393,6 +419,17 @@ public class InterpreterFactory implements InterpreterGroupFactory {
}
}
public Map<String, Object> getEditorFromSettingByClassName(InterpreterSetting intpSetting,
String className) {
List<InterpreterInfo> intpInfos = intpSetting.getInterpreterInfos();
for (InterpreterInfo intpInfo : intpInfos) {
if (className.equals(intpInfo.getClassName())) {
return intpInfo.getEditor();
}
}
return ImmutableMap.of("language", (Object) "text");
}
private void loadInterpreterDependencies(final InterpreterSetting setting) {
setting.setStatus(InterpreterSetting.Status.DOWNLOADING_DEPENDENCIES);
@ -672,7 +709,7 @@ public class InterpreterFactory implements InterpreterGroupFactory {
Interpreter interpreter;
for (InterpreterInfo info : interpreterInfos) {
if (option.isRemote()) {
if (option.isConnectExistingProcess()) {
if (option.isExistingProcess()) {
interpreter =
connectToRemoteRepl(noteId, info.getClassName(), option.getHost(), option.getPort(),
properties);
@ -839,17 +876,21 @@ public class InterpreterFactory implements InterpreterGroupFactory {
synchronized (interpreterSettings) {
InterpreterSetting intpsetting = interpreterSettings.get(id);
if (intpsetting != null) {
try {
stopJobAllInterpreter(intpsetting);
stopJobAllInterpreter(intpsetting);
intpsetting.closeAndRmoveAllInterpreterGroups();
intpsetting.setOption(option);
intpsetting.setProperties(properties);
intpsetting.setDependencies(dependencies);
loadInterpreterDependencies(intpsetting);
intpsetting.closeAndRmoveAllInterpreterGroups();
intpsetting.setOption(option);
intpsetting.setProperties(properties);
intpsetting.setDependencies(dependencies);
loadInterpreterDependencies(intpsetting);
saveToFile();
saveToFile();
} catch (Exception e) {
throw e;
} finally {
loadFromFile();
}
} else {
throw new InterpreterException("Interpreter setting id " + id + " not found");
}
@ -994,7 +1035,7 @@ public class InterpreterFactory implements InterpreterGroupFactory {
new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
remoteInterpreterProcessListener, appEventListener);
remoteInterpreter.setEnv(env);
remoteInterpreter.addEnv(env);
return new LazyOpenInterpreter(remoteInterpreter);
}
@ -1017,15 +1058,16 @@ public class InterpreterFactory implements InterpreterGroupFactory {
public List<InterpreterSetting> getInterpreterSettings(String noteId) {
List<String> interpreterSettingIds = getNoteInterpreterSettingBinding(noteId);
LinkedList<InterpreterSetting> settings = new LinkedList<>();
synchronized (interpreterSettingIds) {
for (String id : interpreterSettingIds) {
InterpreterSetting setting = get(id);
if (setting == null) {
// interpreter setting is removed from factory. remove id from here, too
interpreterSettingIds.remove(id);
} else {
settings.add(setting);
}
Iterator<String> iter = interpreterSettingIds.iterator();
while (iter.hasNext()) {
String id = iter.next();
InterpreterSetting setting = get(id);
if (setting == null) {
// interpreter setting is removed from factory. remove id from here, too
iter.remove();
} else {
settings.add(setting);
}
}
return settings;
@ -1242,6 +1284,35 @@ public class InterpreterFactory implements InterpreterGroupFactory {
this.env = env;
}
public Map<String, Object> getEditorSetting(String noteId, String replName) {
Interpreter intp = getInterpreter(noteId, replName);
Map<String, Object> editor = Maps.newHashMap(
ImmutableMap.<String, Object>builder()
.put("language", "text").build());
String defaultSettingName = getDefaultInterpreterSetting(noteId).getName();
String group = StringUtils.EMPTY;
try {
List<InterpreterSetting> intpSettings = getInterpreterSettings(noteId);
for (InterpreterSetting intpSetting : intpSettings) {
String[] replNameSplit = replName.split("\\.");
if (replNameSplit.length == 2) {
group = replNameSplit[0];
}
// when replName is 'name' of interpreter
if (defaultSettingName.equals(intpSetting.getName())) {
editor = getEditorFromSettingByClassName(intpSetting, intp.getClassName());
}
// when replName is 'alias name' of interpreter or 'group' of interpreter
if (replName.equals(intpSetting.getName()) || group.equals(intpSetting.getName())) {
editor = getEditorFromSettingByClassName(intpSetting, intp.getClassName());
break;
}
}
} catch (NullPointerException e) {
logger.warn("Couldn't get interpreter editor language");
}
return editor;
}
private Interpreter getDevInterpreter() {
if (devInterpreter == null) {

15
zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterInfo.java
View file

@ -19,6 +19,8 @@ package org.apache.zeppelin.interpreter;
import com.google.gson.annotations.SerializedName;
import java.util.Map;
/**
* Information of interpreters in this interpreter setting.
* this will be serialized for conf/interpreter.json and REST api response.
@ -27,11 +29,14 @@ public class InterpreterInfo {
private String name;
@SerializedName("class") private String className;
private boolean defaultInterpreter = false;
private Map<String, Object> editor;
InterpreterInfo(String className, String name, boolean defaultInterpreter) {
InterpreterInfo(String className, String name, boolean defaultInterpreter,
Map<String, Object> editor) {
this.className = className;
this.name = name;
this.defaultInterpreter = defaultInterpreter;
this.editor = editor;
}
public String getName() {
@ -50,6 +55,14 @@ public class InterpreterInfo {
return defaultInterpreter;
}
public Map<String, Object> getEditor() {
return editor;
}
public void setEditor(Map<String, Object> editor) {
this.editor = editor;
}
@Override
public boolean equals(Object obj) {
if (!(obj instanceof InterpreterInfo)) {

4
zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterOption.java
View file

@ -85,10 +85,6 @@ public class InterpreterOption {
this.perNoteSession = perNoteSession;
}
public boolean isConnectExistingProcess() {
return (host != null && port != -1);
}
public String getHost() {
return host;
}

82
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
View file

@ -29,9 +29,9 @@ import java.util.Map;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import com.google.gson.Gson;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -57,11 +57,6 @@ import org.apache.zeppelin.search.SearchService;
import org.apache.zeppelin.user.AuthenticationInfo;
import org.apache.zeppelin.user.Credentials;
import static org.apache.commons.lang.StringUtils.EMPTY;
import static org.apache.commons.lang.StringUtils.isEmpty;
import static org.apache.commons.lang.StringUtils.isNotEmpty;
import static org.apache.commons.lang.StringUtils.isBlank;
/**
* Binded interpreters for a note
*/
@ -82,7 +77,6 @@ public class Note implements Serializable, ParagraphJobListener {
private String name = "";
private String id;
private AtomicReference<String> lastReplName = new AtomicReference<>(EMPTY);
private transient ZeppelinConfiguration conf = ZeppelinConfiguration.create();
private Map<String, List<AngularObject>> angularObjects = new HashMap<>();
@ -128,13 +122,7 @@ public class Note implements Serializable, ParagraphJobListener {
private String getDefaultInterpreterName() {
InterpreterSetting setting = factory.getDefaultInterpreterSetting(getId());
return null != setting ? setting.getName() : EMPTY;
}
void putDefaultReplName() {
String defaultInterpreterName = getDefaultInterpreterName();
logger.info("defaultInterpreterName is '{}'", defaultInterpreterName);
lastReplName.set(defaultInterpreterName);
return null != setting ? setting.getName() : StringUtils.EMPTY;
}
public String getId() {
@ -226,7 +214,7 @@ public class Note implements Serializable, ParagraphJobListener {
*/
public Paragraph addParagraph() {
Paragraph p = new Paragraph(this, this, factory);
addLastReplNameIfEmptyText(p);
setParagraphMagic(p, paragraphs.size());
synchronized (paragraphs) {
paragraphs.add(p);
}
@ -282,7 +270,7 @@ public class Note implements Serializable, ParagraphJobListener {
*/
public Paragraph insertParagraph(int index) {
Paragraph p = new Paragraph(this, this, factory);
addLastReplNameIfEmptyText(p);
setParagraphMagic(p, index);
synchronized (paragraphs) {
paragraphs.add(index, p);
}
@ -292,26 +280,6 @@ public class Note implements Serializable, ParagraphJobListener {
return p;
}
/**
* Add Last Repl name If Paragraph has empty text
*
* @param p Paragraph
*/
private void addLastReplNameIfEmptyText(Paragraph p) {
String replName = lastReplName.get();
if (isEmpty(p.getText()) && isNotEmpty(replName) && isBinding(replName)) {
p.setText(getInterpreterName(replName) + " ");
}
}
public boolean isBinding(String replName) {
return factory.getInterpreter(this.getId(), replName) != null;
}
private String getInterpreterName(String replName) {
return isBlank(replName) ? EMPTY : "%" + replName;
}
/**
* Remove paragraph by id.
*
@ -439,7 +407,7 @@ public class Note implements Serializable, ParagraphJobListener {
List<Map<String, String>> paragraphsInfo = new LinkedList<>();
synchronized (paragraphs) {
for (Paragraph p : paragraphs) {
Map<String, String> info = populatePragraphInfo(p);
Map<String, String> info = populateParagraphInfo(p);
paragraphsInfo.add(info);
}
}
@ -450,14 +418,14 @@ public class Note implements Serializable, ParagraphJobListener {
synchronized (paragraphs) {
for (Paragraph p : paragraphs) {
if (p.getId().equals(paragraphId)) {
return populatePragraphInfo(p);
return populateParagraphInfo(p);
}
}
return new HashMap<>();
}
}
private Map<String, String> populatePragraphInfo(Paragraph p) {
private Map<String, String> populateParagraphInfo(Paragraph p) {
Map<String, String> info = new HashMap<>();
info.put("id", p.getId());
info.put("status", p.getStatus().toString());
@ -473,15 +441,26 @@ public class Note implements Serializable, ParagraphJobListener {
return info;
}
private void setParagraphMagic(Paragraph p, int index) {
if (paragraphs.size() > 0) {
String magic;
if (index == 0) {
magic = paragraphs.get(0).getMagic();
} else {
magic = paragraphs.get(index - 1).getMagic();
}
if (StringUtils.isNotEmpty(magic)) {
p.setText(magic + "\n");
}
}
}
/**
* Run all paragraphs sequentially.
*/
public void runAll() {
String cronExecutingUser = (String) getConfig().get("cronExecutingUser");
synchronized (paragraphs) {
if (!paragraphs.isEmpty()) {
setLastReplName(paragraphs.get(paragraphs.size() - 1));
}
for (Paragraph p : paragraphs) {
if (!p.isEnabled()) {
continue;
@ -607,16 +586,6 @@ public class Note implements Serializable, ParagraphJobListener {
repo.save(this, subject);
}
private void setLastReplName(Paragraph lastParagraphStarted) {
if (isNotEmpty(lastParagraphStarted.getRequiredReplName())) {
lastReplName.set(lastParagraphStarted.getRequiredReplName());
}
}
public void setLastReplName(String paragraphId) {
setLastReplName(getParagraph(paragraphId));
}
/**
* Persist this note with maximum delay.
*/
@ -681,14 +650,6 @@ public class Note implements Serializable, ParagraphJobListener {
this.info = info;
}
String getLastReplName() {
return lastReplName.get();
}
public String getLastInterpreterName() {
return getInterpreterName(getLastReplName());
}
@Override
public void beforeStatusChange(Job job, Status before, Status after) {
if (jobListenerFactory != null) {
@ -723,7 +684,6 @@ public class Note implements Serializable, ParagraphJobListener {
}
}
@Override
public void onOutputAppend(Paragraph paragraph, InterpreterOutput out, String output) {
if (jobListenerFactory != null) {

116
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
View file

@ -24,15 +24,21 @@ import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import com.google.common.base.Predicate;
import com.google.common.collect.FluentIterable;
import com.google.common.collect.Sets;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.stream.JsonReader;
import org.apache.commons.codec.binary.StringUtils;
import org.quartz.CronScheduleBuilder;
import org.quartz.CronTrigger;
import org.quartz.JobBuilder;
@ -155,9 +161,13 @@ public class Notebook implements NoteEventListener {
}
if (interpreterIds != null) {
bindInterpretersToNote(note.getId(), interpreterIds);
note.putDefaultReplName();
}
if (subject != null && !"anonymous".equals(subject.getUser())) {
Set<String> owners = new HashSet<String>();
owners.add(subject.getUser());
notebookAuthorization.setOwners(note.getId(), owners);
}
notebookIndex.addIndexDoc(note);
note.persist(subject);
fireNoteCreateEvent(note);
@ -469,7 +479,7 @@ public class Notebook implements NoteEventListener {
if (notebookRepo instanceof NotebookRepoSync) {
NotebookRepoSync mainRepo = (NotebookRepoSync) notebookRepo;
if (mainRepo.getRepoCount() > 1) {
mainRepo.sync();
mainRepo.sync(subject);
}
}
@ -525,6 +535,35 @@ public class Notebook implements NoteEventListener {
}
}
public List<Note> getAllNotes(AuthenticationInfo subject) {
final Set<String> entities = Sets.newHashSet();
if (subject != null) {
entities.add(subject.getUser());
}
synchronized (notes) {
return FluentIterable.from(notes.values()).filter(new Predicate<Note>() {
@Override
public boolean apply(Note input) {
return input != null && notebookAuthorization.isReader(input.getId(), entities);
}
}).toSortedList(new Comparator<Note>() {
@Override
public int compare(Note note1, Note note2) {
String name1 = note1.getId();
if (note1.getName() != null) {
name1 = note1.getName();
}
String name2 = note2.getId();
if (note2.getName() != null) {
name2 = note2.getName();
}
return name1.compareTo(name2);
}
});
}
}
private Map<String, Object> getParagraphForJobManagerItem(Paragraph paragraph) {
Map<String, Object> paragraphItem = new HashMap<>();
@ -574,7 +613,78 @@ public class Notebook implements NoteEventListener {
return lastRunningUnixTime;
}
public List<Map<String, Object>> getJobListforNotebook(boolean needsReload,
public List<Map<String, Object>> getJobListByParagraphId(String paragraphID) {
String gotNoteId = null;
List<Note> notes = getAllNotes();
for (Note note : notes) {
Paragraph p = note.getParagraph(paragraphID);
if (p != null) {
gotNoteId = note.getId();
}
}
return getJobListBymNotebookId(gotNoteId);
}
public List<Map<String, Object>> getJobListBymNotebookId(String notebookID) {
final String CRON_TYPE_NOTEBOOK_KEYWORD = "cron";
long lastRunningUnixTime = 0;
boolean isNotebookRunning = false;
Note jobNote = getNote(notebookID);
List<Map<String, Object>> notesInfo = new LinkedList<>();
if (jobNote == null) {
return notesInfo;
}
Map<String, Object> info = new HashMap<>();
info.put("notebookId", jobNote.getId());
String notebookName = jobNote.getName();
if (notebookName != null && !notebookName.equals("")) {
info.put("notebookName", jobNote.getName());
} else {
info.put("notebookName", "Note " + jobNote.getId());
}
// set notebook type ( cron or normal )
if (jobNote.getConfig().containsKey(CRON_TYPE_NOTEBOOK_KEYWORD) && !jobNote.getConfig()
.get(CRON_TYPE_NOTEBOOK_KEYWORD).equals("")) {
info.put("notebookType", "cron");
} else {
info.put("notebookType", "normal");
}
// set paragraphs
List<Map<String, Object>> paragraphsInfo = new LinkedList<>();
for (Paragraph paragraph : jobNote.getParagraphs()) {
// check paragraph's status.
if (paragraph.getStatus().isRunning()) {
isNotebookRunning = true;
}
// get data for the job manager.
Map<String, Object> paragraphItem = getParagraphForJobManagerItem(paragraph);
lastRunningUnixTime = getUnixTimeLastRunParagraph(paragraph);
paragraphsInfo.add(paragraphItem);
}
// set interpreter bind type
String interpreterGroupName = null;
if (replFactory.getInterpreterSettings(jobNote.getId()) != null
&& replFactory.getInterpreterSettings(jobNote.getId()).size() >= 1) {
interpreterGroupName = replFactory.getInterpreterSettings(jobNote.getId()).get(0).getName();
}
// notebook json object root information.
info.put("interpreter", interpreterGroupName);
info.put("isRunningJob", isNotebookRunning);
info.put("unixTimeLastRun", lastRunningUnixTime);
info.put("paragraphs", paragraphsInfo);
notesInfo.add(info);
return notesInfo;
};
public List<Map<String, Object>> getJobListByUnixTime(boolean needsReload,
long lastUpdateServerUnixTime, AuthenticationInfo subject) {
final String CRON_TYPE_NOTEBOOK_KEYWORD = "cron";

43
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
View file

@ -17,6 +17,7 @@
package org.apache.zeppelin.notebook;
import org.apache.commons.lang.StringUtils;
import org.apache.zeppelin.display.AngularObject;
import org.apache.zeppelin.display.AngularObjectRegistry;
import org.apache.zeppelin.helium.HeliumPackage;
@ -455,22 +456,22 @@ public class Paragraph extends Job implements Serializable, Cloneable {
Credentials credentials = note.getCredentials();
if (authenticationInfo != null) {
UserCredentials userCredentials = credentials.getUserCredentials(
authenticationInfo.getUser());
authenticationInfo.getUser());
authenticationInfo.setUserCredentials(userCredentials);
}
InterpreterContext interpreterContext = new InterpreterContext(
note.getId(),
getId(),
this.getTitle(),
this.getText(),
this.getAuthenticationInfo(),
this.getConfig(),
this.settings,
registry,
resourcePool,
runners,
output);
note.getId(),
getId(),
this.getTitle(),
this.getText(),
this.getAuthenticationInfo(),
this.getConfig(),
this.settings,
registry,
resourcePool,
runners,
output);
return interpreterContext;
}
@ -565,4 +566,22 @@ public class Paragraph extends Job implements Serializable, Cloneable {
}
return scriptBody;
}
public String getMagic() {
String magic = StringUtils.EMPTY;
String text = getText();
if (text != null && text.startsWith("%")) {
magic = text.split("\\s+")[0];
if (isValidInterpreter(magic.substring(1))) {
return magic;
} else {
return StringUtils.EMPTY;
}
}
return magic;
}
private boolean isValidInterpreter(String replName) {
return factory.getInterpreter(note.getId(), replName) != null;
}
}

28
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepoSync.java
View file

@ -93,7 +93,8 @@ public class NotebookRepoSync implements NotebookRepo {
}
if (getRepoCount() > 1) {
try {
sync(0, 1);
AuthenticationInfo subject = new AuthenticationInfo("anonymous");
sync(0, 1, subject);
} catch (IOException e) {
LOG.warn("Failed to sync with secondary storage on start {}", e);
}
@ -175,12 +176,12 @@ public class NotebookRepoSync implements NotebookRepo {
*
* @throws IOException
*/
void sync(int sourceRepoIndex, int destRepoIndex) throws IOException {
void sync(int sourceRepoIndex, int destRepoIndex, AuthenticationInfo subject) throws IOException {
LOG.info("Sync started");
NotebookRepo srcRepo = getRepo(sourceRepoIndex);
NotebookRepo dstRepo = getRepo(destRepoIndex);
List <NoteInfo> srcNotes = srcRepo.list(null);
List <NoteInfo> dstNotes = dstRepo.list(null);
List <NoteInfo> srcNotes = srcRepo.list(subject);
List <NoteInfo> dstNotes = dstRepo.list(subject);
Map<String, List<String>> noteIDs = notesCheckDiff(srcNotes, srcRepo, dstNotes, dstRepo);
List<String> pushNoteIDs = noteIDs.get(pushKey);
@ -192,7 +193,7 @@ public class NotebookRepoSync implements NotebookRepo {
for (String id : pushNoteIDs) {
LOG.info("ID : " + id);
}
pushNotes(pushNoteIDs, srcRepo, dstRepo);
pushNotes(subject, pushNoteIDs, srcRepo, dstRepo);
} else {
LOG.info("Nothing to push");
}
@ -202,7 +203,7 @@ public class NotebookRepoSync implements NotebookRepo {
for (String id : pullNoteIDs) {
LOG.info("ID : " + id);
}
pushNotes(pullNoteIDs, dstRepo, srcRepo);
pushNotes(subject, pullNoteIDs, dstRepo, srcRepo);
} else {
LOG.info("Nothing to pull");
}
@ -212,7 +213,7 @@ public class NotebookRepoSync implements NotebookRepo {
for (String id : delDstNoteIDs) {
LOG.info("ID : " + id);
}
deleteNotes(delDstNoteIDs, dstRepo);
deleteNotes(subject, delDstNoteIDs, dstRepo);
} else {
LOG.info("Nothing to delete from dest");
}
@ -220,20 +221,21 @@ public class NotebookRepoSync implements NotebookRepo {
LOG.info("Sync ended");
}
public void sync() throws IOException {
sync(0, 1);
public void sync(AuthenticationInfo subject) throws IOException {
sync(0, 1, subject);
}
private void pushNotes(List<String> ids, NotebookRepo localRepo,
private void pushNotes(AuthenticationInfo subject, List<String> ids, NotebookRepo localRepo,
NotebookRepo remoteRepo) throws IOException {
for (String id : ids) {
remoteRepo.save(localRepo.get(id, null), null);
remoteRepo.save(localRepo.get(id, subject), subject);
}
}
private void deleteNotes(List<String> ids, NotebookRepo repo) throws IOException {
private void deleteNotes(AuthenticationInfo subject, List<String> ids, NotebookRepo repo)
throws IOException {
for (String id : ids) {
repo.remove(id, null);
repo.remove(id, subject);
}
}

143
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/socket/Message.java
View file

@ -21,43 +21,43 @@ import java.util.HashMap;
import java.util.Map;
/**
* Zeppelin websocker massage template class.
* Zeppelin websocket massage template class.
*/
public class Message {
/**
* Representation of event type.
*/
public static enum OP {
GET_HOME_NOTE, // [c-s] load note for home screen
GET_HOME_NOTE, // [c-s] load note for home screen
GET_NOTE, // [c-s] client load note
// @param id note id
GET_NOTE, // [c-s] client load note
// @param id note id
NOTE, // [s-c] note info
// @param note serlialized Note object
NOTE, // [s-c] note info
// @param note serlialized Note object
PARAGRAPH, // [s-c] paragraph info
// @param paragraph serialized paragraph object
PARAGRAPH, // [s-c] paragraph info
// @param paragraph serialized paragraph object
PROGRESS, // [s-c] progress update
// @param id paragraph id
// @param progress percentage progress
PROGRESS, // [s-c] progress update
// @param id paragraph id
// @param progress percentage progress
NEW_NOTE, // [c-s] create new notebook
DEL_NOTE, // [c-s] delete notebook
// @param id note id
CLONE_NOTE, // [c-s] clone new notebook
// @param id id of note to clone
// @param name name fpor the cloned note
IMPORT_NOTE, // [c-s] import notebook
// @param object notebook
NEW_NOTE, // [c-s] create new notebook
DEL_NOTE, // [c-s] delete notebook
// @param id note id
CLONE_NOTE, // [c-s] clone new notebook
// @param id id of note to clone
// @param name name fpor the cloned note
IMPORT_NOTE, // [c-s] import notebook
// @param object notebook
NOTE_UPDATE,
RUN_PARAGRAPH, // [c-s] run paragraph
// @param id paragraph id
// @param paragraph paragraph content.ie. script
// @param config paragraph config
// @param params paragraph params
RUN_PARAGRAPH, // [c-s] run paragraph
// @param id paragraph id
// @param paragraph paragraph content.ie. script
// @param config paragraph config
// @param params paragraph params
COMMIT_PARAGRAPH, // [c-s] commit paragraph
// @param id paragraph id
@ -69,72 +69,77 @@ public class Message {
CANCEL_PARAGRAPH, // [c-s] cancel paragraph run
// @param id paragraph id
MOVE_PARAGRAPH, // [c-s] move paragraph order
// @param id paragraph id
// @param index index the paragraph want to go
MOVE_PARAGRAPH, // [c-s] move paragraph order
// @param id paragraph id
// @param index index the paragraph want to go
INSERT_PARAGRAPH, // [c-s] create new paragraph below current paragraph
// @param target index
COMPLETION, // [c-s] ask completion candidates
// @param id
// @param buf current code
// @param cursor cursor position in code
EDITOR_SETTING, // [c-s] ask paragraph editor setting
// @param magic magic keyword written in paragraph
// ex) spark.spark or spark
COMPLETION_LIST, // [s-c] send back completion candidates list
// @param id
// @param completions list of string
COMPLETION, // [c-s] ask completion candidates
// @param id
// @param buf current code
// @param cursor cursor position in code
LIST_NOTES, // [c-s] ask list of note
RELOAD_NOTES_FROM_REPO, // [c-s] reload notes from repo
COMPLETION_LIST, // [s-c] send back completion candidates list
// @param id
// @param completions list of string
NOTES_INFO, // [s-c] list of note infos
// @param notes serialized List<NoteInfo> object
LIST_NOTES, // [c-s] ask list of note
RELOAD_NOTES_FROM_REPO, // [c-s] reload notes from repo
NOTES_INFO, // [s-c] list of note infos
// @param notes serialized List<NoteInfo> object
PARAGRAPH_REMOVE,
PARAGRAPH_CLEAR_OUTPUT,
PARAGRAPH_APPEND_OUTPUT, // [s-c] append output
PARAGRAPH_UPDATE_OUTPUT, // [s-c] update (replace) output
PARAGRAPH_APPEND_OUTPUT, // [s-c] append output
PARAGRAPH_UPDATE_OUTPUT, // [s-c] update (replace) output
PING,
AUTH_INFO,
ANGULAR_OBJECT_UPDATE, // [s-c] add/update angular object
ANGULAR_OBJECT_REMOVE, // [s-c] add angular object del
ANGULAR_OBJECT_UPDATE, // [s-c] add/update angular object
ANGULAR_OBJECT_REMOVE, // [s-c] add angular object del
ANGULAR_OBJECT_UPDATED, // [c-s] angular object value updated,
ANGULAR_OBJECT_UPDATED, // [c-s] angular object value updated,
ANGULAR_OBJECT_CLIENT_BIND, // [c-s] angular object updated from AngularJS z object
ANGULAR_OBJECT_CLIENT_BIND, // [c-s] angular object updated from AngularJS z object
ANGULAR_OBJECT_CLIENT_UNBIND, // [c-s] angular object unbind from AngularJS z object
ANGULAR_OBJECT_CLIENT_UNBIND, // [c-s] angular object unbind from AngularJS z object
LIST_CONFIGURATIONS, // [c-s] ask all key/value pairs of configurations
CONFIGURATIONS_INFO, // [s-c] all key/value pairs of configurations
// @param settings serialized Map<String, String> object
LIST_CONFIGURATIONS, // [c-s] ask all key/value pairs of configurations
CONFIGURATIONS_INFO, // [s-c] all key/value pairs of configurations
// @param settings serialized Map<String, String> object
CHECKPOINT_NOTEBOOK, // [c-s] checkpoint notebook to storage repository
// @param noteId
// @param checkpointName
CHECKPOINT_NOTEBOOK, // [c-s] checkpoint notebook to storage repository
// @param noteId
// @param checkpointName
LIST_REVISION_HISTORY, // [c-s] list revision history of the notebook
// @param noteId
NOTE_REVISION, // [c-s] get certain revision of note
// @param noteId
// @param revisionId
LIST_REVISION_HISTORY, // [c-s] list revision history of the notebook
// @param noteId
NOTE_REVISION, // [c-s] get certain revision of note
// @param noteId
// @param revisionId
APP_APPEND_OUTPUT, // [s-c] append output
APP_UPDATE_OUTPUT, // [s-c] update (replace) output
APP_LOAD, // [s-c] on app load
APP_STATUS_CHANGE, // [s-c] on app status change
APP_APPEND_OUTPUT, // [s-c] append output
APP_UPDATE_OUTPUT, // [s-c] update (replace) output
APP_LOAD, // [s-c] on app load
APP_STATUS_CHANGE, // [s-c] on app status change
LIST_NOTEBOOK_JOBS, // [c-s] get notebook job management infomations
LIST_UPDATE_NOTEBOOK_JOBS, // [c-s] get job management informations for until unixtime
// @param unixTime
GET_INTERPRETER_BINDINGS, // [c-s] get interpreter bindings
// @param noteID
SAVE_INTERPRETER_BINDINGS, // [c-s] save interpreter bindings
// @param noteID
// @param selectedSettingIds
INTERPRETER_BINDINGS // [s-c] interpreter bindings
LIST_NOTEBOOK_JOBS, // [c-s] get notebook job management infomations
LIST_UPDATE_NOTEBOOK_JOBS, // [c-s] get job management informations for until unixtime
UNSUBSCRIBE_UPDATE_NOTEBOOK_JOBS, // [c-s] unsubscribe job information for job management
// @param unixTime
GET_INTERPRETER_BINDINGS, // [c-s] get interpreter bindings
// @param noteID
SAVE_INTERPRETER_BINDINGS, // [c-s] save interpreter bindings
// @param noteID
// @param selectedSettingIds
INTERPRETER_BINDINGS // [s-c] interpreter bindings
}
public OP op;

91
zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterFactoryTest.java
View file

@ -21,6 +21,8 @@ import java.io.*;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.HashMap;
import java.util.Properties;
import org.apache.commons.io.FileUtils;
@ -31,12 +33,23 @@ import org.apache.zeppelin.dep.Dependency;
import org.apache.zeppelin.dep.DependencyResolver;
import org.apache.zeppelin.interpreter.mock.MockInterpreter1;
import org.apache.zeppelin.interpreter.mock.MockInterpreter2;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreter;
import org.apache.zeppelin.notebook.JobListenerFactory;
import org.apache.zeppelin.notebook.Note;
import org.apache.zeppelin.notebook.Notebook;
import org.apache.zeppelin.notebook.repo.NotebookRepo;
import org.apache.zeppelin.notebook.repo.VFSNotebookRepo;
import org.apache.zeppelin.scheduler.SchedulerFactory;
import org.apache.zeppelin.search.SearchService;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.quartz.SchedulerException;
import org.sonatype.aether.RepositoryException;
import static org.junit.Assert.*;
import static org.mockito.Mockito.mock;
import org.mockito.Mock;
public class InterpreterFactoryTest {
@ -44,23 +57,43 @@ public class InterpreterFactoryTest {
private File tmpDir;
private ZeppelinConfiguration conf;
private InterpreterContext context;
private Notebook notebook;
private NotebookRepo notebookRepo;
private DependencyResolver depResolver;
private SchedulerFactory schedulerFactory;
@Mock
private JobListenerFactory jobListenerFactory;
@Before
public void setUp() throws Exception {
tmpDir = new File(System.getProperty("java.io.tmpdir")+"/ZeppelinLTest_"+System.currentTimeMillis());
tmpDir.mkdirs();
new File(tmpDir, "conf").mkdirs();
FileUtils.copyDirectory(new File("src/test/resources/interpreter"), new File(tmpDir, "interpreter"));
MockInterpreter1.register("mock1", "org.apache.zeppelin.interpreter.mock.MockInterpreter1");
Map<String, InterpreterProperty> propertiesMockInterpreter1 = new HashMap<String, InterpreterProperty>();
propertiesMockInterpreter1.put("PROPERTY_1", new InterpreterProperty("PROPERTY_1", "", "VALUE_1", "desc"));
propertiesMockInterpreter1.put("property_2", new InterpreterProperty("", "property_2", "value_2", "desc"));
MockInterpreter1.register("mock1", "mock1", "org.apache.zeppelin.interpreter.mock.MockInterpreter1", propertiesMockInterpreter1);
MockInterpreter2.register("mock2", "org.apache.zeppelin.interpreter.mock.MockInterpreter2");
System.setProperty(ConfVars.ZEPPELIN_HOME.getVarName(), tmpDir.getAbsolutePath());
System.setProperty(ConfVars.ZEPPELIN_INTERPRETERS.getVarName(), "org.apache.zeppelin.interpreter.mock.MockInterpreter1,org.apache.zeppelin.interpreter.mock.MockInterpreter2");
System.setProperty(ConfVars.ZEPPELIN_INTERPRETERS.getVarName(),
"org.apache.zeppelin.interpreter.mock.MockInterpreter1," +
"org.apache.zeppelin.interpreter.mock.MockInterpreter2," +
"org.apache.zeppelin.interpreter.mock.MockInterpreter11");
System.setProperty(ConfVars.ZEPPELIN_INTERPRETER_GROUP_ORDER.getVarName(),
"mock1,mock2,mock11,dev");
conf = new ZeppelinConfiguration();
schedulerFactory = new SchedulerFactory();
depResolver = new DependencyResolver(tmpDir.getAbsolutePath() + "/local-repo");
factory = new InterpreterFactory(conf, new InterpreterOption(false), null, null, null, depResolver);
context = new InterpreterContext("note", "id", "title", "text", null, null, null, null, null, null, null);
SearchService search = mock(SearchService.class);
notebookRepo = new VFSNotebookRepo(conf);
notebook = new Notebook(conf, notebookRepo, schedulerFactory, factory, jobListenerFactory, search,
null, null);
}
@After
@ -95,11 +128,34 @@ public class InterpreterFactoryTest {
assertNull(mock1Setting.getInterpreterGroup("sharedProcess").get("session"));
}
@Test
public void testRemoteRepl() throws Exception {
factory = new InterpreterFactory(conf, new InterpreterOption(true), null, null, null, depResolver);
List<InterpreterSetting> all = factory.get();
InterpreterSetting mock1Setting = null;
for (InterpreterSetting setting : all) {
if (setting.getName().equals("mock1")) {
mock1Setting = setting;
break;
}
}
InterpreterGroup interpreterGroup = mock1Setting.getInterpreterGroup("sharedProcess");
factory.createInterpretersForNote(mock1Setting, "sharedProcess", "session");
// get interpreter
assertNotNull("get Interpreter", interpreterGroup.get("session").get(0));
assertTrue(interpreterGroup.get("session").get(0) instanceof LazyOpenInterpreter);
LazyOpenInterpreter lazyInterpreter = (LazyOpenInterpreter)(interpreterGroup.get("session").get(0));
assertTrue(lazyInterpreter.getInnerInterpreter() instanceof RemoteInterpreter);
RemoteInterpreter remoteInterpreter = (RemoteInterpreter) lazyInterpreter.getInnerInterpreter();
assertEquals("VALUE_1", remoteInterpreter.getEnv().get("PROPERTY_1"));
assertEquals("value_2", remoteInterpreter.getProperty("property_2"));
}
@Test
public void testFactoryDefaultList() throws IOException, RepositoryException {
// get default settings
List<String> all = factory.getDefaultInterpreterSettingList();
assertTrue(factory.getRegisteredInterpreterList().size() >= all.size());
assertTrue(factory.get().size() >= all.size());
}
@Test
@ -137,8 +193,8 @@ public class InterpreterFactoryTest {
@Test
public void testInterpreterAliases() throws IOException, RepositoryException {
factory = new InterpreterFactory(conf, null, null, null, depResolver);
final InterpreterInfo info1 = new InterpreterInfo("className1", "name1", true);
final InterpreterInfo info2 = new InterpreterInfo("className2", "name1", true);
final InterpreterInfo info1 = new InterpreterInfo("className1", "name1", true, null);
final InterpreterInfo info2 = new InterpreterInfo("className2", "name1", true, null);
factory.add("group1", new ArrayList<InterpreterInfo>(){{
add(info1);
}}, new ArrayList<Dependency>(), new InterpreterOption(true), new Properties(), "/path1");
@ -167,4 +223,29 @@ public class InterpreterFactoryTest {
assertEquals("'.' is invalid for InterpreterSetting name.", e.getMessage());
}
}
@Test
public void getEditorSetting() throws IOException, RepositoryException, SchedulerException {
List<String> intpIds = new ArrayList<>();
for(InterpreterSetting intpSetting: factory.get()) {
if (intpSetting.getName().startsWith("mock1")) {
intpIds.add(intpSetting.getId());
}
}
Note note = notebook.createNote(intpIds, null);
// get editor setting from interpreter-setting.json
Map<String, Object> editor = factory.getEditorSetting(note.getId(), "mock11");
assertEquals("java", editor.get("language"));
// when interpreter is not loaded via interpreter-setting.json
// or editor setting doesn't exit
editor = factory.getEditorSetting(note.getId(), "mock1");
assertEquals(null, editor.get("language"));
// when interpreter is not bound to note
editor = factory.getEditorSetting(note.getId(), "mock2");
assertEquals("text", editor.get("language"));
}
}

94
zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NoteTest.java
View file

@ -17,13 +17,8 @@
package org.apache.zeppelin.notebook;
import com.google.common.base.Optional;
import org.apache.commons.lang.StringUtils;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterFactory;
import org.apache.zeppelin.interpreter.InterpreterSetting;
import org.apache.zeppelin.interpreter.mock.MockInterpreter2;
import org.apache.zeppelin.notebook.repo.NotebookRepo;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.search.SearchService;
@ -32,7 +27,6 @@ import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.runners.MockitoJUnitRunner;
import static org.junit.Assert.*;
@ -84,89 +78,47 @@ public class NoteTest {
}
@Test
public void putDefaultReplNameIfInterpreterSettingAbsent() {
when(interpreterFactory.getDefaultInterpreterSetting(anyString()))
.thenReturn(null);
public void addParagraphWithEmptyReplNameTest() {
Note note = new Note(repo, interpreterFactory, jobListenerFactory, index, credentials, noteEventListener);
note.putDefaultReplName();
assertEquals(StringUtils.EMPTY, note.getLastReplName());
assertEquals(StringUtils.EMPTY, note.getLastInterpreterName());
}
@Test
public void putDefaultReplNameIfInterpreterSettingPresent() {
InterpreterSetting interpreterSetting = Mockito.mock(InterpreterSetting.class);
when(interpreterSetting.getName()).thenReturn("spark");
when(interpreterFactory.getDefaultInterpreterSetting(anyString()))
.thenReturn(interpreterSetting);
Note note = new Note(repo, interpreterFactory, jobListenerFactory, index, credentials, noteEventListener);
note.putDefaultReplName();
assertEquals("spark", note.getLastReplName());
assertEquals("%spark", note.getLastInterpreterName());
}
@Test
public void addParagraphWithLastReplName() {
InterpreterSetting interpreterSetting = Mockito.mock(InterpreterSetting.class);
when(interpreterSetting.getName()).thenReturn("spark");
when(interpreterFactory.getDefaultInterpreterSetting(anyString()))
.thenReturn(interpreterSetting);
Note note = new Note(repo, interpreterFactory, jobListenerFactory, index, credentials, noteEventListener);
note.putDefaultReplName(); //set lastReplName
when(interpreterFactory.getInterpreter(note.getId(), "spark")).thenReturn(new MockInterpreter2(null));
Paragraph p = note.addParagraph();
assertEquals("%spark ", p.getText());
assertNull(p.getText());
}
@Test
public void insertParagraphWithLastReplName() {
InterpreterSetting interpreterSetting = Mockito.mock(InterpreterSetting.class);
when(interpreterSetting.getName()).thenReturn("spark");
when(interpreterFactory.getDefaultInterpreterSetting(anyString()))
.thenReturn(interpreterSetting);
public void addParagraphWithLastReplNameTest() {
when(interpreterFactory.getInterpreter(anyString(), eq("spark"))).thenReturn(interpreter);
Note note = new Note(repo, interpreterFactory, jobListenerFactory, index, credentials, noteEventListener);
note.putDefaultReplName(); //set lastReplName
when(interpreterFactory.getInterpreter(note.getId(), "spark")).thenReturn(new MockInterpreter2(null));
Paragraph p1 = note.addParagraph();
p1.setText("%spark ");
Paragraph p2 = note.addParagraph();
Paragraph p = note.insertParagraph(note.getParagraphs().size());
assertEquals("%spark ", p.getText());
assertEquals("%spark\n", p2.getText());
}
@Test
public void setLastReplName() {
String paragraphId = "HelloWorld";
Note note = Mockito.spy(new Note(repo, interpreterFactory, jobListenerFactory, index, credentials, noteEventListener));
Paragraph mockParagraph = Mockito.mock(Paragraph.class);
when(note.getParagraph(paragraphId)).thenReturn(mockParagraph);
when(mockParagraph.getRequiredReplName()).thenReturn("spark");
public void insertParagraphWithLastReplNameTest() {
when(interpreterFactory.getInterpreter(anyString(), eq("spark"))).thenReturn(interpreter);
note.setLastReplName(paragraphId);
Note note = new Note(repo, interpreterFactory, jobListenerFactory, index, credentials, noteEventListener);
Paragraph p1 = note.addParagraph();
p1.setText("%spark ");
Paragraph p2 = note.insertParagraph(note.getParagraphs().size());
assertEquals("spark", note.getLastReplName());
assertEquals("%spark\n", p2.getText());
}
@Test
public void isBindingTest() {
Note note = spy(new Note());
when(note.getId()).thenReturn("test1");
InterpreterFactory mockInterpreterFactory = mock(InterpreterFactory.class);
note.setInterpreterFactory(mockInterpreterFactory);
public void insertParagraphWithInvalidReplNameTest() {
when(interpreterFactory.getInterpreter(anyString(), eq("invalid"))).thenReturn(null);
//when is not binding
assertFalse(note.isBinding("spark"));
Note note = new Note(repo, interpreterFactory, jobListenerFactory, index, credentials, noteEventListener);
Paragraph p1 = note.addParagraph();
p1.setText("%invalid ");
Paragraph p2 = note.insertParagraph(note.getParagraphs().size());
//when is binding
when(mockInterpreterFactory.getInterpreter("test1", "spark")).
thenReturn(new MockInterpreter2(null));
assertTrue(note.isBinding("spark"));
assertNull(p2.getText());
}
}

89
zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
View file

@ -42,6 +42,7 @@ import org.apache.zeppelin.scheduler.Job;
import org.apache.zeppelin.scheduler.Job.Status;
import org.apache.zeppelin.scheduler.SchedulerFactory;
import org.apache.zeppelin.search.SearchService;
import org.apache.zeppelin.user.AuthenticationInfo;
import org.apache.zeppelin.user.Credentials;
import org.junit.After;
import org.junit.Before;
@ -74,6 +75,7 @@ public class NotebookTest implements JobListenerFactory{
notebookDir = new File(tmpDir + "/notebook");
notebookDir.mkdirs();
System.setProperty(ConfVars.ZEPPELIN_CONF_DIR.getVarName(), tmpDir.toString() + "/conf");
System.setProperty(ConfVars.ZEPPELIN_HOME.getVarName(), tmpDir.getAbsolutePath());
System.setProperty(ConfVars.ZEPPELIN_NOTEBOOK_DIR.getVarName(), notebookDir.getAbsolutePath());
System.setProperty(ConfVars.ZEPPELIN_INTERPRETERS.getVarName(), "org.apache.zeppelin.interpreter.mock.MockInterpreter1,org.apache.zeppelin.interpreter.mock.MockInterpreter2");
@ -94,8 +96,7 @@ public class NotebookTest implements JobListenerFactory{
credentials = new Credentials(conf.credentialsPersist(), conf.getCredentialsPath());
notebook = new Notebook(conf, notebookRepo, schedulerFactory, factory, this, search,
notebookAuthorization, credentials);
notebookAuthorization, credentials);
}
@After
@ -108,7 +109,7 @@ public class NotebookTest implements JobListenerFactory{
Note note = notebook.createNote(null);
factory.setInterpreters(note.getId(), factory.getDefaultInterpreterSettingList());
// run with defatul repl
// run with default repl
Paragraph p1 = note.addParagraph();
Map config = p1.getConfig();
config.put("enabled", true);
@ -209,6 +210,18 @@ public class NotebookTest implements JobListenerFactory{
assertEquals(1, notebook2.getAllNotes().size());
}
@Test
public void testCreateNoteWithSubject() throws IOException, SchedulerException, RepositoryException {
AuthenticationInfo subject = new AuthenticationInfo("user1");
Note note = notebook.createNote(subject);
assertNotNull(notebook.getNotebookAuthorization().getOwners(note.getId()));
assertEquals(1, notebook.getNotebookAuthorization().getOwners(note.getId()).size());
Set<String> owners = new HashSet<>();
owners.add("user1");
assertEquals(owners, notebook.getNotebookAuthorization().getOwners(note.getId()));
}
@Test
public void testClearParagraphOutput() throws IOException, SchedulerException{
Note note = notebook.createNote(null);
@ -219,7 +232,7 @@ public class NotebookTest implements JobListenerFactory{
p1.setText("hello world");
note.run(p1.getId());
while(p1.isTerminated()==false || p1.getResult()==null) Thread.yield();
while(p1.isTerminated() == false || p1.getResult() == null) Thread.yield();
assertEquals("repl1: hello world", p1.getResult().message());
// clear paragraph output/result
@ -254,7 +267,7 @@ public class NotebookTest implements JobListenerFactory{
note.runAll();
// wait for finish
while(p3.isTerminated()==false) {
while(p3.isTerminated() == false) {
Thread.yield();
}
@ -266,7 +279,7 @@ public class NotebookTest implements JobListenerFactory{
}
@Test
public void testSchedule() throws InterruptedException, IOException{
public void testSchedule() throws InterruptedException, IOException {
// create a note and a paragraph
Note note = notebook.createNote(null);
factory.setInterpreters(note.getId(), factory.getDefaultInterpreterSettingList());
@ -284,7 +297,7 @@ public class NotebookTest implements JobListenerFactory{
config.put("cron", "* * * * * ?");
note.setConfig(config);
notebook.refreshCron(note.getId());
Thread.sleep(1*1000);
Thread.sleep(1 * 1000);
// remove cron scheduler.
config.put("cron", null);
@ -293,7 +306,7 @@ public class NotebookTest implements JobListenerFactory{
Thread.sleep(1000);
dateFinished = p.getDateFinished();
assertNotNull(dateFinished);
Thread.sleep(1*1000);
Thread.sleep(1 * 1000);
assertEquals(dateFinished, p.getDateFinished());
}
@ -351,7 +364,7 @@ public class NotebookTest implements JobListenerFactory{
@Test
public void testExportAndImportNote() throws IOException, CloneNotSupportedException,
InterruptedException {
InterruptedException, InterpreterException, SchedulerException, RepositoryException {
Note note = notebook.createNote(null);
factory.setInterpreters(note.getId(), factory.getDefaultInterpreterSettingList());
@ -374,11 +387,20 @@ public class NotebookTest implements JobListenerFactory{
assertEquals(p.getId(), p2.getId());
assertEquals(p.text, p2.text);
assertEquals(p.getResult().message(), p2.getResult().message());
// Verify import note with subject
AuthenticationInfo subject = new AuthenticationInfo("user1");
Note importedNote2 = notebook.importNote(exportedNoteJson, "Title2", subject);
assertNotNull(notebook.getNotebookAuthorization().getOwners(importedNote2.getId()));
assertEquals(1, notebook.getNotebookAuthorization().getOwners(importedNote2.getId()).size());
Set<String> owners = new HashSet<>();
owners.add("user1");
assertEquals(owners, notebook.getNotebookAuthorization().getOwners(importedNote2.getId()));
}
@Test
public void testCloneNote() throws IOException, CloneNotSupportedException,
InterruptedException {
InterruptedException, InterpreterException, SchedulerException, RepositoryException {
Note note = notebook.createNote(null);
factory.setInterpreters(note.getId(), factory.getDefaultInterpreterSettingList());
@ -396,6 +418,15 @@ public class NotebookTest implements JobListenerFactory{
assertEquals(cp.getId(), p.getId());
assertEquals(cp.text, p.text);
assertEquals(cp.getResult().message(), p.getResult().message());
// Verify clone note with subject
AuthenticationInfo subject = new AuthenticationInfo("user1");
Note cloneNote2 = notebook.cloneNote(note.getId(), "clone note2", subject);
assertNotNull(notebook.getNotebookAuthorization().getOwners(cloneNote2.getId()));
assertEquals(1, notebook.getNotebookAuthorization().getOwners(cloneNote2.getId()).size());
Set<String> owners = new HashSet<>();
owners.add("user1");
assertEquals(owners, notebook.getNotebookAuthorization().getOwners(cloneNote2.getId()));
}
@Test
@ -445,8 +476,8 @@ public class NotebookTest implements JobListenerFactory{
p2.setText("%mock2 world");
note.runAll();
while(p1.isTerminated()==false || p1.getResult()==null) Thread.yield();
while(p2.isTerminated()==false || p2.getResult()==null) Thread.yield();
while (p1.isTerminated() == false || p1.getResult() == null) Thread.yield();
while (p2.isTerminated() == false || p2.getResult() == null) Thread.yield();
assertEquals(2, ResourcePoolUtils.getAllResources().size());
@ -573,26 +604,26 @@ public class NotebookTest implements JobListenerFactory{
new HashSet<String>(Arrays.asList("user1")));
assertEquals(notebookAuthorization.isOwner(note.getId(),
new HashSet<String>(Arrays.asList("user2"))), false);
new HashSet<String>(Arrays.asList("user2"))), false);
assertEquals(notebookAuthorization.isOwner(note.getId(),
new HashSet<String>(Arrays.asList("user1"))), true);
assertEquals(notebookAuthorization.isReader(note.getId(),
new HashSet<String>(Arrays.asList("user3"))), false);
new HashSet<String>(Arrays.asList("user3"))), false);
assertEquals(notebookAuthorization.isReader(note.getId(),
new HashSet<String>(Arrays.asList("user2"))), true);
new HashSet<String>(Arrays.asList("user2"))), true);
assertEquals(notebookAuthorization.isWriter(note.getId(),
new HashSet<String>(Arrays.asList("user2"))), false);
new HashSet<String>(Arrays.asList("user2"))), false);
assertEquals(notebookAuthorization.isWriter(note.getId(),
new HashSet<String>(Arrays.asList("user1"))), true);
new HashSet<String>(Arrays.asList("user1"))), true);
// Test clearing of permssions
notebookAuthorization.setReaders(note.getId(), Sets.<String>newHashSet());
assertEquals(notebookAuthorization.isReader(note.getId(),
new HashSet<String>(Arrays.asList("user2"))), true);
new HashSet<String>(Arrays.asList("user2"))), true);
assertEquals(notebookAuthorization.isReader(note.getId(),
new HashSet<String>(Arrays.asList("user3"))), true);
new HashSet<String>(Arrays.asList("user3"))), true);
notebook.removeNote(note.getId(), null);
}
@ -822,6 +853,26 @@ public class NotebookTest implements JobListenerFactory{
notebook.removeNote(note1.getId(), null);
}
@Test
public void testGetAllNotes() throws Exception {
Note note1 = notebook.createNote(null);
Note note2 = notebook.createNote(null);
assertEquals(2, notebook.getAllNotes(new AuthenticationInfo("anonymous")).size());
notebook.getNotebookAuthorization().setOwners(note1.getId(), Sets.newHashSet("user1"));
notebook.getNotebookAuthorization().setWriters(note1.getId(), Sets.newHashSet("user1"));
notebook.getNotebookAuthorization().setReaders(note1.getId(), Sets.newHashSet("user1"));
assertEquals(1, notebook.getAllNotes(new AuthenticationInfo("anonymous")).size());
assertEquals(2, notebook.getAllNotes(new AuthenticationInfo("user1")).size());
notebook.getNotebookAuthorization().setOwners(note2.getId(), Sets.newHashSet("user2"));
notebook.getNotebookAuthorization().setWriters(note2.getId(), Sets.newHashSet("user2"));
notebook.getNotebookAuthorization().setReaders(note2.getId(), Sets.newHashSet("user2"));
assertEquals(0, notebook.getAllNotes(new AuthenticationInfo("anonymous")).size());
assertEquals(1, notebook.getAllNotes(new AuthenticationInfo("user1")).size());
assertEquals(1, notebook.getAllNotes(new AuthenticationInfo("user2")).size());
}
private void delete(File file){
if(file.isFile()) file.delete();
else if(file.isDirectory()){

6
zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitNotebookRepoTest.java
View file

@ -224,7 +224,7 @@ public class GitNotebookRepoTest {
Map<String, Object> config = p1.getConfig();
config.put("enabled", true);
p1.setConfig(config);
p1.setText("%md checkpoint test text");
p1.setText("checkpoint test text");
notebookRepo.save(note, null);
// second checkpoint
@ -245,7 +245,7 @@ public class GitNotebookRepoTest {
Paragraph p2 = note.addParagraph();
config.put("enabled", false);
p2.setConfig(config);
p2.setText("%md get revision when modified note test text");
p2.setText("get revision when modified note test text");
notebookRepo.save(note, null);
note = notebookRepo.get(TEST_NOTE_ID, null);
int paragraphCount_3 = note.getParagraphs().size();
@ -282,7 +282,7 @@ public class GitNotebookRepoTest {
Map<String, Object> config = p1.getConfig();
config.put("enabled", true);
p1.setConfig(config);
p1.setText("%md get revision when modified note test text");
p1.setText("get revision when modified note test text");
notebookRepo.save(note, null);
int paragraphCount_2 = note.getParagraphs().size();

2
zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/NotebookRepoSyncTest.java
View file

@ -185,7 +185,7 @@ public class NotebookRepoSyncTest implements JobListenerFactory {
assertEquals(0, notebookRepoSync.get(1,
notebookRepoSync.list(1, null).get(0).getId(), null).getParagraphs().size());
/* apply sync */
notebookRepoSync.sync();
notebookRepoSync.sync(null);
/* check whether added to second storage */
assertEquals(1, notebookRepoSync.get(1,
notebookRepoSync.list(1, null).get(0).getId(), null).getParagraphs().size());

12
zeppelin-zengine/src/test/resources/interpreter/mock/interpreter-setting.json Normal file
View file

@ -0,0 +1,12 @@
[
{
"group": "mock11",
"name": "mock11",
"className": "org.apache.zeppelin.interpreter.mock.MockInterpreter11",
"properties": {
},
"editor": {
"language": "java"
}
}
]