[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
---
layout: page
2016-08-06 05:50:25 +00:00
title: "Writing a new Application(Experimental)"
description: "Apache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. Make your own Application in Apache Zeppelin is quite easy."
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
group: development
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
{% include JB/setup %}
2016-07-05 08:36:32 +00:00
# Writing a new Application (Experimental)
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
< div id = "toc" > < / div >
## What is Apache Zeppelin Application
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
Apache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. While application runs on Interpreter process, it's able to access resources provided by Interpreter through ResourcePool. Output is always rendered by AngularDisplaySystem. Therefore application provides all the possiblities of making interactive graphical application that uses data and processing power of any Interpreter.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
## Make your own Application
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
Writing Application means extending `org.apache.zeppelin.helium.Application` . You can use your favorite IDE and language while Java class files are packaged into jar. `Application` class looks like
```java
/**
* Constructor. Invoked when application is loaded
*/
public Application(ApplicationContext context);
/**
* Invoked when there're (possible) updates in required resource set.
* i.e. invoked after application load and after paragraph finishes.
*/
public abstract void run(ResourceSet args);
/**
* Invoked before application unload.
* Application is automatically unloaded with paragraph/notebook removal
*/
public abstract void unload();
```
You can check example applications under [./zeppelin-examples ](https://github.com/apache/incubator-zeppelin/tree/master/zeppelin-examples ) directory.
## Development mode
In the development mode, you can run your Application in your IDE as a normal java application and see the result inside of Zeppelin notebook.
2016-07-05 08:36:32 +00:00
`org.apache.zeppelin.interpreter.dev.ZeppelinApplicationDevServer` can run Zeppelin Application in development mode.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```java
// entry point for development mode
public static void main(String[] args) throws Exception {
// add resources for development mode
LocalResourcePool pool = new LocalResourcePool("dev");
pool.put("date", new Date());
2016-07-05 08:36:32 +00:00
// run application in devlopment mode with given resource
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
// in this case, Clock.class.getName() will be the application class name
ZeppelinApplicationDevServer devServer = new ZeppelinApplicationDevServer(
2016-07-05 08:36:32 +00:00
Clock.class.getName(), pool.getAll());
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
// start development mode
devServer.start();
devServer.join();
}
```
In the Zeppelin notebook, run `%dev run` will connect to application running in development mode.
## Package file
Package file is a json file that provides information about the application.
2016-07-05 08:36:32 +00:00
Json file contains the following information
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```
{
name : "[organization].[name]",
description : "Description",
artifact : "groupId:artifactId:version",
className : "your.package.name.YourApplicationClass",
resources : [
["resource.name", ":resource.class.name"],
["alternative.resource.name", ":alternative.class.name"]
],
icon : "< i class = "icon" > < / i > "
}
```
#### name
2016-07-05 08:36:32 +00:00
Name is a string in `[group].[name]` format.
`[group]` and `[name]` allow only `[A-Za-z0-9_]` .
Group is normally the name of an organization who creates this application.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
#### description
2016-07-05 08:36:32 +00:00
A short description about the application
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
#### artifact
Location of the jar artifact.
2016-07-05 08:36:32 +00:00
`"groupId:artifactId:version"` will load artifact from maven repository.
If jar exists in the local filesystem, absolute/relative can be used.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
e.g.
When artifact exists in Maven repository
2016-07-05 08:36:32 +00:00
```
artifact: "org.apache.zeppelin:zeppelin-examples:0.6.0"
```
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
When artifact exists in the local filesystem
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
```
artifact: "zeppelin-example/target/zeppelin-example-0.6.0.jar"
```
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
#### className
Entry point. Class that extends `org.apache.zeppelin.helium.Application`
#### resources
2016-07-05 08:36:32 +00:00
Two dimensional array that defines required resources by name or by className. Helium Application launcher will compare resources in the ResourcePool with the information in this field and suggest application only when all required resources are available in the ResourcePool.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
Resouce name is a string which will be compared with the name of objects in the ResourcePool. className is a string with ":" prepended, which will be compared with className of the objects in the ResourcePool.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
Application may require two or more resources. Required resources can be listed inside of the json array. For example, if the application requires object "name1", "name2" and "className1" type of object to run, resources field can be
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```
resources: [
[ "name1", "name2", ":className1", ...]
]
```
2016-07-05 08:36:32 +00:00
If Application can handle alternative combination of required resources, alternative set can be listed as below.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```
resources: [
[ "name", ":className"],
[ "altName", ":altClassName1"],
...
]
```
2016-07-05 08:36:32 +00:00
Easier way to understand this scheme is
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```
resources: [
[ 'resource' AND 'resource' AND ... ] OR
[ 'resource' AND 'resource' AND ... ] OR
...
]
```
#### icon
2016-07-05 08:36:32 +00:00
Icon to be used on the application button. String in this field will be rendered as a HTML tag.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
e.g.
```
icon: "< i class = 'fa fa-clock-o' > < / i > "
```