[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
---
layout: page
[ZEPPELIN-2217] AdvancedTransformation for Visualization
### What is this PR for?
`AdvancedTransformation` has more detailed options while providing existing features of `PivotTransformation` and `ColumnselectorTransformation` which Zeppelin already has

Here are some features which advanced-transformation can provide.
1. **(screenshot)** multiple sub charts
2. **(screenshot)** parameter widgets: `input`, `checkbox`, `option`, `textarea`
3. **(screenshot)** expand/fold axis and parameter panels
4. **(screenshot)** clear axis and parameter panels
5. **(screenshot)** remove duplicated columns in an axis
6. **(screenshot)** limit column count in an axis
7. configurable char axes: `valueType`, `axisType`, `description`, ...
8. configurable chart parameters
9. lazy transformation
10. parsing parameters automatically based on their type: `int`, `float`, `string`, `JSON`
11. multiple transformation methods
12. re-initialize whole configuration based on spec hash.
13. **(screenshot)** shared axis
#### API Details: Spec
`AdvancedTransformation` requires `spec` which includes axis and parameter details for charts.
- Let's create 2 sub-charts called `simple-line` and `step-line`.
- Each sub chart can have different `axis` and `parameter` depending on their requirements.
```js
constructor(targetEl, config) {
super(targetEl, config)
const spec = {
charts: {
'simple-line': {
sharedAxis: true, /** set if you want to share axes between sub charts, default is `false` */
axis: {
'xAxis': { dimension: 'multiple', axisType: 'key', },
'yAxis': { dimension: 'multiple', axisType: 'aggregator'},
'category': { dimension: 'multiple', axisType: 'group', },
},
parameter: {
'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },
'yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', },
'dashLength': { valueType: 'int', defaultValue: 0, description: 'the length of dash', },
},
},
'step-line': {
axis: {
'xAxis': { dimension: 'single', axisType: 'unique', },
'yAxis': { dimension: 'multiple', axisType: 'value', },
},
parameter: {
'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },
'yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', },
'noStepRisers': { valueType: 'boolean', defaultValue: false, description: 'no risers in step line', widget: 'checkbox', },
},
},
}
this.transformation = new AdvancedTransformation(config, spec)
}
```
#### API Details: Axis Spec
| Field Name | Available Values (type) | Description |
| --- | --- | --- |
|`dimension` | `single` | Axis can contains only 1 column |
|`dimension` | `multiple` | Axis can contains multiple columns |
|`axisType` | `key` | Column(s) in this axis will be used as `key` like in `PivotTransformation`. These columns will be served in `column.key` |
|`axisType` | `aggregator` | Column(s) in this axis will be used as `value` like in `PivotTransformation`. These columns will be served in `column.aggregator` |
|`axisType` | `group` | Column(s) in this axis will be used as `group` like in `PivotTransformation`. These columns will be served in `column.group` |
|`axisType` | (string) | Any string value can be used here. These columns will be served in `column.custom` |
|`maxAxisCount` | (int) | The maximum column count that this axis can contains. (unlimited if `undefined`) |
|`valueType` | (string) | Describe the value type just for annotation |
Here is an example.
```js
axis: {
'xAxis': { dimension: 'multiple', axisType: 'key', },
'yAxis': { dimension: 'multiple', axisType: 'aggregator'},
'category': { dimension: 'multiple', axisType: 'group', maxAxisCount: 2, valueType: 'string', },
},
```
#### API Details: Parameter Spec
| Field Name | Available Values (type) | Description |
| --- | --- | --- |
|`valueType` | `string` | Parameter which has string value |
|`valueType` | `int` | Parameter which has int value |
|`valueType` | `float` | Parameter which has float value |
|`valueType` | `boolean` | Parameter which has boolean value used with `checkbox` widget usually |
|`valueType` | `JSON` | Parameter which has JSON value used with `textarea` widget usually. `defaultValue` should be `""` (empty string). This ||`defaultValue` | (any) | Default value of this parameter. `JSON` type should have `""` (empty string) |
|`description` | (string) | Description of this parameter. This value will be parsed as HTML for pretty output |
|`widget` | `input` | Use [input](https://developer.mozilla.org/en/docs/Web/HTML/Element/input) widget. This is the default widget (if `widget` is undefined)|
|`widget` | `checkbox` | Use [checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) widget. |
|`widget` | `textarea` | Use [textarea](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/textarea) widget. |
|`widget` | `option` | Use [select + option](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) widget. This parameter should have `optionValues` field as well. |
|`optionValues` | (Array<string>) | Available option values used with the `option` widget |
Here is an example.
```js
parameter: {
// string type, input widget
'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },
// boolean type, checkbox widget
'inverted': { widget: 'checkbox', valueType: 'boolean', defaultValue: false, description: 'invert x and y axes', },
// string type, option widget with `optionValues`
'graphType': { widget: 'option', valueType: 'string', defaultValue: 'line', description: 'graph type', optionValues: [ 'line', 'smoothedLine', 'step', ], },
// HTML in `description`
'dateFormat': { valueType: 'string', defaultValue: '', description: 'format of date (<a href="https://docs.amcharts.com/3/javascriptcharts/AmGraph#dateFormat">doc</a>) (e.g YYYY-MM-DD)', },
// JSON type, textarea widget
'yAxisGuides': { widget: 'textarea', valueType: 'JSON', defaultValue: '', description: 'guides of yAxis ', },
```
#### API Details: Transformer Spec
`AdvancedTransformation` supports 3 transformation methods. The return value will depend on the transformation method type.
```js
const spec = {
charts: {
'simple': {
/** default value of `transform.method` is the flatten cube. */
axis: { ... },
parameter: { ... }
},
'cube-group': {
transform: { method: 'cube', },
axis: { ... },
parameter: { ... },
}
'no-group': {
transform: { method: 'raw', },
axis: { ... },
parameter: { ... },
}
```
| Field Name | Available Values (type) | Description |
| --- | --- | --- |
|`method` | `object` | designed for [amcharts: serial](https://www.amcharts.com/demos/date-based-data/) |
|`method` | `array` | designed for [highcharts: column](http://jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/highcharts/demo/column-basic/) |
|`method` | `drill-down` | designed for [highcharts: drill-down](http://jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/highcharts/demo/column-drilldown/) |
|`method` | `raw` | will return the original `tableData.rows` |
Whatever you specified as `transform.method`, the `transformer` value will be always function for lazy computation.
```js
// advanced-transformation.util#getTransformer
if (transformSpec.method === 'raw') {
transformer = () => { return rows; }
} else if (transformSpec.method === 'array') {
transformer = () => {
...
return { ... }
}
}
```
#### Feature Details: Automatic parameter parsing
Advanced transformation will parse parameter values automatically based on their type: `int`, `float`, `string`, `JSON`
- See also `advanced-transformation-util.js#parseParameter`
#### Feature Details: re-initialize the whole configuration based on spec hash
```js
// advanced-transformation-util#initializeConfig
const currentVersion = JSON.stringify(spec)
if (!config.spec || !config.spec.version || config.spec.version !== currentVersion) {
spec.version = currentVersion
// reset config...
}
```
#### Feature Details: Shared Axes
If you set `sharedAxis` to `true` in chart specification, then these charts will share their axes. (default is `false`)
```js
const spec = {
charts: {
'column': {
transform: { method: 'array', },
sharedAxis: true,
axis: { ... },
parameter: { ... },
},
'stacked': {
transform: { method: 'array', },
sharedAxis: true,
axis: { ... }
parameter: { ... },
},
```

#### API Details: Usage in Visualization#render()
Let's assume that we want to create 2 sub-charts called `basic` and `no-group`.
- https://github.com/1ambda/zeppelin-ultimate-line-chart (an practical example)
```js
drawBasicChart(parameter, column, transformer) {
const { ... } = transformer()
}
drawNoGroupChart(parameter, column, transformer) {
const { ... } = transformer()
}
render(data) {
const { chart, parameter, column, transformer, } = data
if (chart === 'basic') {
this.drawBasicChart(parameter, column, transformer)
} else if (chart === 'no-group') {
this.drawNoGroupChart(parameter, column, transformer)
}
}
```
### What type of PR is it?
[Feature]
### Todos
NONE
### What is the Jira issue?
[ZEPPELIN-2217](https://issues.apache.org/jira/browse/ZEPPELIN-2217)
### How should this be tested?
1. Clone https://github.com/1ambda/zeppelin-ultimate-line-chart
2. Create a symbolic link `ultimate-line-chart.json` into `$ZEPPELIN_HOME/helium`
3. Modify the `artifact` value to proper absolute path considering your local machine.
4. Install the above visualization in `localhost:9000/#helium`
5. Test it
### Screenshots (if appropriate)
#### 1. *(screenshot)* multiple sub charts

#### 2. *(screenshot)* parameter widgets: `input`, `checkbox`, `option`, `textarea`

#### 3. *(screenshot)* expand/fold axis and parameter panels

#### 4. *(screenshot)* clear axis and parameter panels

#### 5. *(screenshot)* remove duplicated columns in an axis

#### 6. *(screenshot)* limit column count in an axis

### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes #2098 from 1ambda/ZEPPELIN-2217/advanced-transformation and squashes the following commits:
6cde7c9 [1ambda] fix reset params when spec change
c75a3f2 [1ambda] fix: Reset persisted axis
6a2130a [1ambda] fix: clear config only when axis changed
5464e84 [1ambda] fix: Optimize array 2 key method
9beb1e7 [1ambda] fix: Type error
2408225 [1ambda] test: Add test for array 2key
bf56761 [1ambda] feat: Add array:2-key transform method
7c6768f [1ambda] feat: Use axisSpec.desc as tooltip
f98d4c9 [1ambda] fix: Remove invalid key prop
5cf2ece [1ambda] feat: Add minAxisCount
4887800 [1ambda] fix: Remove local module yarn caches
3e29572 [1ambda] refactor: copyModule func
c91a033 [1ambda] fix: Set yarn cache dir in helium-bundles
04b5140 [1ambda] fix: Import a-tr
0a876cf [1ambda] docs: Update index.md
380b1af [1ambda] docs: Fix typo and add desc for existing trs
908214b [1ambda] docs: Move experimental tags
a009627 [1ambda] feat: Allow dup aggr axis
3b44e92 [1ambda] fix: Remove unuse const
ab6c22e [1ambda] test: Add test for drill-down method
756107a [1ambda] test: Add array transformation method
d819c73 [1ambda] test: Add object method
bf00fba [1ambda] test: Add MockTableData
39fe5ae [1ambda] test: Add test for getColumnsFromAxis
4c393b4 [1ambda] fix: Add polyfill for es6 funcs in test
e92c787 [1ambda] test: Add test for rmDup, aplMaxAxisCount
843f45d [1ambda] test: Add test for getCurrent* funcs
ae5277c [1ambda] test: Add test for initializeConfig
c14a9dc7 [1ambda] test: Add tests for widget, params
c510af1 [1ambda] docs: Add doc for Transformation
52db37b [1ambda] feat: Show panel menus only when opened
17ad4a4 [1ambda] feat: Support chartChanged, parameterChanged
c0d33d3 [1ambda] fix: Sort selectors in drilldown method
cfd6fef [1ambda] feat: sharedAxis
9af80ce [1ambda] style: Indent
79b5654 [1ambda] fix: return the same info in transform
7bee464 [1ambda] fix: Keynames
ee8788e [1ambda] feat: Support drill-down
666025a [1ambda] fix: DON'T reset current chart
ae1891f [1ambda] add array:key transform
4167a2e [1ambda] fix: Sort keyNames
912b5b7 [1ambda] fix: Persist initialized config
f1f6b0c [1ambda] feat: Support ARRAY transform.method
812f9a2 [1ambda] fix: Set proper aggr value when 0 group
20f9437 [1ambda] fix: getCube func
25d51a9 [1ambda] DON'T display aggr.name when aggrColumns.length == 1
f37e13d [1ambda] fix: Add 'object' transform.method
da2370c [1ambda] fix: Add resetAxis, Param funcs
2370682 [1ambda] fix: average is not caculated correctly
dd08e38 [1ambda] fix: Set param panel height to 400
881695a [1ambda] feat: clear chart, param separately
4d0d62b [1ambda] fix: DON'T clean panel config
92676d1 [1ambda] fix: limit parameter panel height to 370
cc29060 [1ambda] feat: parse param description as HTML
9a2d227 [1ambda] fix: Stop event propagation in widgets
fcc625c [1ambda] feat: Automatic param parsing
b4d774c [1ambda] fix: Dont close param panel when enter
088705b [1ambda] refactor: Remove util and add Widget funcs
bf88b4f [1ambda] feat: textare widget and update hook
4e73012 [1ambda] feat: widget checkbox
11b7eaa [1ambda] feat: option widget
5d3efc9 [1ambda] fix: Change panel header
b1d9d31 [1ambda] feat: Save and close with enter key
53f508c [1ambda] feat: custom axisSpec
0dbc431 [1ambda] feat: Support transformer
94d837a [1ambda] feat: Automatic spec versioning
74b8b4e [1ambda] fix: Duplicated radio btn id, name
5b88f08 [1ambda] fix: Modify margin of subchart radio btns
019892c [1ambda] feat: Support transform: flatten
0484e1e [1ambda] feat: Support maxAxisCount in axisSpec
936901b [1ambda] feat: Support undefined valueType in axisSpec
7a454ff [1ambda] feat: Cube Transformation
f0ed02f [1ambda] feat: Support same axis types
49985c6 [1ambda] refactor: Refine axis, param spec
d89e223 [1ambda] feat: advanced-transformation-api
75569ce [1ambda] feat: Support multiple charts in UI
e1fcc2e [1ambda] feat: Support multiple charts
97be629 [1ambda] fix: Add singleDimensionAggregatorChanged
676bd7e [1ambda] refactor: Refine transform API
9fb398e [1ambda] feat: Add clearConfig
a8a4fb1 [1ambda] refactor: Add getAxisInSingleDimension func
9768ecf [1ambda] feat: Add groupBase axis option
91ae54d [1ambda] fix: Overflow issue in single aggr
10c80fc [1ambda] feat: AdvancedTransformation
2017-04-11 14:14:46 +00:00
title: "Writing a new Application"
2016-08-06 05:50:25 +00:00
description: "Apache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. Make your own Application in Apache Zeppelin is quite easy."
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
group: development
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
{% include JB/setup %}
[ZEPPELIN-2217] AdvancedTransformation for Visualization
### What is this PR for?
`AdvancedTransformation` has more detailed options while providing existing features of `PivotTransformation` and `ColumnselectorTransformation` which Zeppelin already has

Here are some features which advanced-transformation can provide.
1. **(screenshot)** multiple sub charts
2. **(screenshot)** parameter widgets: `input`, `checkbox`, `option`, `textarea`
3. **(screenshot)** expand/fold axis and parameter panels
4. **(screenshot)** clear axis and parameter panels
5. **(screenshot)** remove duplicated columns in an axis
6. **(screenshot)** limit column count in an axis
7. configurable char axes: `valueType`, `axisType`, `description`, ...
8. configurable chart parameters
9. lazy transformation
10. parsing parameters automatically based on their type: `int`, `float`, `string`, `JSON`
11. multiple transformation methods
12. re-initialize whole configuration based on spec hash.
13. **(screenshot)** shared axis
#### API Details: Spec
`AdvancedTransformation` requires `spec` which includes axis and parameter details for charts.
- Let's create 2 sub-charts called `simple-line` and `step-line`.
- Each sub chart can have different `axis` and `parameter` depending on their requirements.
```js
constructor(targetEl, config) {
super(targetEl, config)
const spec = {
charts: {
'simple-line': {
sharedAxis: true, /** set if you want to share axes between sub charts, default is `false` */
axis: {
'xAxis': { dimension: 'multiple', axisType: 'key', },
'yAxis': { dimension: 'multiple', axisType: 'aggregator'},
'category': { dimension: 'multiple', axisType: 'group', },
},
parameter: {
'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },
'yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', },
'dashLength': { valueType: 'int', defaultValue: 0, description: 'the length of dash', },
},
},
'step-line': {
axis: {
'xAxis': { dimension: 'single', axisType: 'unique', },
'yAxis': { dimension: 'multiple', axisType: 'value', },
},
parameter: {
'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },
'yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', },
'noStepRisers': { valueType: 'boolean', defaultValue: false, description: 'no risers in step line', widget: 'checkbox', },
},
},
}
this.transformation = new AdvancedTransformation(config, spec)
}
```
#### API Details: Axis Spec
| Field Name | Available Values (type) | Description |
| --- | --- | --- |
|`dimension` | `single` | Axis can contains only 1 column |
|`dimension` | `multiple` | Axis can contains multiple columns |
|`axisType` | `key` | Column(s) in this axis will be used as `key` like in `PivotTransformation`. These columns will be served in `column.key` |
|`axisType` | `aggregator` | Column(s) in this axis will be used as `value` like in `PivotTransformation`. These columns will be served in `column.aggregator` |
|`axisType` | `group` | Column(s) in this axis will be used as `group` like in `PivotTransformation`. These columns will be served in `column.group` |
|`axisType` | (string) | Any string value can be used here. These columns will be served in `column.custom` |
|`maxAxisCount` | (int) | The maximum column count that this axis can contains. (unlimited if `undefined`) |
|`valueType` | (string) | Describe the value type just for annotation |
Here is an example.
```js
axis: {
'xAxis': { dimension: 'multiple', axisType: 'key', },
'yAxis': { dimension: 'multiple', axisType: 'aggregator'},
'category': { dimension: 'multiple', axisType: 'group', maxAxisCount: 2, valueType: 'string', },
},
```
#### API Details: Parameter Spec
| Field Name | Available Values (type) | Description |
| --- | --- | --- |
|`valueType` | `string` | Parameter which has string value |
|`valueType` | `int` | Parameter which has int value |
|`valueType` | `float` | Parameter which has float value |
|`valueType` | `boolean` | Parameter which has boolean value used with `checkbox` widget usually |
|`valueType` | `JSON` | Parameter which has JSON value used with `textarea` widget usually. `defaultValue` should be `""` (empty string). This ||`defaultValue` | (any) | Default value of this parameter. `JSON` type should have `""` (empty string) |
|`description` | (string) | Description of this parameter. This value will be parsed as HTML for pretty output |
|`widget` | `input` | Use [input](https://developer.mozilla.org/en/docs/Web/HTML/Element/input) widget. This is the default widget (if `widget` is undefined)|
|`widget` | `checkbox` | Use [checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) widget. |
|`widget` | `textarea` | Use [textarea](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/textarea) widget. |
|`widget` | `option` | Use [select + option](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) widget. This parameter should have `optionValues` field as well. |
|`optionValues` | (Array<string>) | Available option values used with the `option` widget |
Here is an example.
```js
parameter: {
// string type, input widget
'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },
// boolean type, checkbox widget
'inverted': { widget: 'checkbox', valueType: 'boolean', defaultValue: false, description: 'invert x and y axes', },
// string type, option widget with `optionValues`
'graphType': { widget: 'option', valueType: 'string', defaultValue: 'line', description: 'graph type', optionValues: [ 'line', 'smoothedLine', 'step', ], },
// HTML in `description`
'dateFormat': { valueType: 'string', defaultValue: '', description: 'format of date (<a href="https://docs.amcharts.com/3/javascriptcharts/AmGraph#dateFormat">doc</a>) (e.g YYYY-MM-DD)', },
// JSON type, textarea widget
'yAxisGuides': { widget: 'textarea', valueType: 'JSON', defaultValue: '', description: 'guides of yAxis ', },
```
#### API Details: Transformer Spec
`AdvancedTransformation` supports 3 transformation methods. The return value will depend on the transformation method type.
```js
const spec = {
charts: {
'simple': {
/** default value of `transform.method` is the flatten cube. */
axis: { ... },
parameter: { ... }
},
'cube-group': {
transform: { method: 'cube', },
axis: { ... },
parameter: { ... },
}
'no-group': {
transform: { method: 'raw', },
axis: { ... },
parameter: { ... },
}
```
| Field Name | Available Values (type) | Description |
| --- | --- | --- |
|`method` | `object` | designed for [amcharts: serial](https://www.amcharts.com/demos/date-based-data/) |
|`method` | `array` | designed for [highcharts: column](http://jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/highcharts/demo/column-basic/) |
|`method` | `drill-down` | designed for [highcharts: drill-down](http://jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/highcharts/demo/column-drilldown/) |
|`method` | `raw` | will return the original `tableData.rows` |
Whatever you specified as `transform.method`, the `transformer` value will be always function for lazy computation.
```js
// advanced-transformation.util#getTransformer
if (transformSpec.method === 'raw') {
transformer = () => { return rows; }
} else if (transformSpec.method === 'array') {
transformer = () => {
...
return { ... }
}
}
```
#### Feature Details: Automatic parameter parsing
Advanced transformation will parse parameter values automatically based on their type: `int`, `float`, `string`, `JSON`
- See also `advanced-transformation-util.js#parseParameter`
#### Feature Details: re-initialize the whole configuration based on spec hash
```js
// advanced-transformation-util#initializeConfig
const currentVersion = JSON.stringify(spec)
if (!config.spec || !config.spec.version || config.spec.version !== currentVersion) {
spec.version = currentVersion
// reset config...
}
```
#### Feature Details: Shared Axes
If you set `sharedAxis` to `true` in chart specification, then these charts will share their axes. (default is `false`)
```js
const spec = {
charts: {
'column': {
transform: { method: 'array', },
sharedAxis: true,
axis: { ... },
parameter: { ... },
},
'stacked': {
transform: { method: 'array', },
sharedAxis: true,
axis: { ... }
parameter: { ... },
},
```

#### API Details: Usage in Visualization#render()
Let's assume that we want to create 2 sub-charts called `basic` and `no-group`.
- https://github.com/1ambda/zeppelin-ultimate-line-chart (an practical example)
```js
drawBasicChart(parameter, column, transformer) {
const { ... } = transformer()
}
drawNoGroupChart(parameter, column, transformer) {
const { ... } = transformer()
}
render(data) {
const { chart, parameter, column, transformer, } = data
if (chart === 'basic') {
this.drawBasicChart(parameter, column, transformer)
} else if (chart === 'no-group') {
this.drawNoGroupChart(parameter, column, transformer)
}
}
```
### What type of PR is it?
[Feature]
### Todos
NONE
### What is the Jira issue?
[ZEPPELIN-2217](https://issues.apache.org/jira/browse/ZEPPELIN-2217)
### How should this be tested?
1. Clone https://github.com/1ambda/zeppelin-ultimate-line-chart
2. Create a symbolic link `ultimate-line-chart.json` into `$ZEPPELIN_HOME/helium`
3. Modify the `artifact` value to proper absolute path considering your local machine.
4. Install the above visualization in `localhost:9000/#helium`
5. Test it
### Screenshots (if appropriate)
#### 1. *(screenshot)* multiple sub charts

#### 2. *(screenshot)* parameter widgets: `input`, `checkbox`, `option`, `textarea`

#### 3. *(screenshot)* expand/fold axis and parameter panels

#### 4. *(screenshot)* clear axis and parameter panels

#### 5. *(screenshot)* remove duplicated columns in an axis

#### 6. *(screenshot)* limit column count in an axis

### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes #2098 from 1ambda/ZEPPELIN-2217/advanced-transformation and squashes the following commits:
6cde7c9 [1ambda] fix reset params when spec change
c75a3f2 [1ambda] fix: Reset persisted axis
6a2130a [1ambda] fix: clear config only when axis changed
5464e84 [1ambda] fix: Optimize array 2 key method
9beb1e7 [1ambda] fix: Type error
2408225 [1ambda] test: Add test for array 2key
bf56761 [1ambda] feat: Add array:2-key transform method
7c6768f [1ambda] feat: Use axisSpec.desc as tooltip
f98d4c9 [1ambda] fix: Remove invalid key prop
5cf2ece [1ambda] feat: Add minAxisCount
4887800 [1ambda] fix: Remove local module yarn caches
3e29572 [1ambda] refactor: copyModule func
c91a033 [1ambda] fix: Set yarn cache dir in helium-bundles
04b5140 [1ambda] fix: Import a-tr
0a876cf [1ambda] docs: Update index.md
380b1af [1ambda] docs: Fix typo and add desc for existing trs
908214b [1ambda] docs: Move experimental tags
a009627 [1ambda] feat: Allow dup aggr axis
3b44e92 [1ambda] fix: Remove unuse const
ab6c22e [1ambda] test: Add test for drill-down method
756107a [1ambda] test: Add array transformation method
d819c73 [1ambda] test: Add object method
bf00fba [1ambda] test: Add MockTableData
39fe5ae [1ambda] test: Add test for getColumnsFromAxis
4c393b4 [1ambda] fix: Add polyfill for es6 funcs in test
e92c787 [1ambda] test: Add test for rmDup, aplMaxAxisCount
843f45d [1ambda] test: Add test for getCurrent* funcs
ae5277c [1ambda] test: Add test for initializeConfig
c14a9dc7 [1ambda] test: Add tests for widget, params
c510af1 [1ambda] docs: Add doc for Transformation
52db37b [1ambda] feat: Show panel menus only when opened
17ad4a4 [1ambda] feat: Support chartChanged, parameterChanged
c0d33d3 [1ambda] fix: Sort selectors in drilldown method
cfd6fef [1ambda] feat: sharedAxis
9af80ce [1ambda] style: Indent
79b5654 [1ambda] fix: return the same info in transform
7bee464 [1ambda] fix: Keynames
ee8788e [1ambda] feat: Support drill-down
666025a [1ambda] fix: DON'T reset current chart
ae1891f [1ambda] add array:key transform
4167a2e [1ambda] fix: Sort keyNames
912b5b7 [1ambda] fix: Persist initialized config
f1f6b0c [1ambda] feat: Support ARRAY transform.method
812f9a2 [1ambda] fix: Set proper aggr value when 0 group
20f9437 [1ambda] fix: getCube func
25d51a9 [1ambda] DON'T display aggr.name when aggrColumns.length == 1
f37e13d [1ambda] fix: Add 'object' transform.method
da2370c [1ambda] fix: Add resetAxis, Param funcs
2370682 [1ambda] fix: average is not caculated correctly
dd08e38 [1ambda] fix: Set param panel height to 400
881695a [1ambda] feat: clear chart, param separately
4d0d62b [1ambda] fix: DON'T clean panel config
92676d1 [1ambda] fix: limit parameter panel height to 370
cc29060 [1ambda] feat: parse param description as HTML
9a2d227 [1ambda] fix: Stop event propagation in widgets
fcc625c [1ambda] feat: Automatic param parsing
b4d774c [1ambda] fix: Dont close param panel when enter
088705b [1ambda] refactor: Remove util and add Widget funcs
bf88b4f [1ambda] feat: textare widget and update hook
4e73012 [1ambda] feat: widget checkbox
11b7eaa [1ambda] feat: option widget
5d3efc9 [1ambda] fix: Change panel header
b1d9d31 [1ambda] feat: Save and close with enter key
53f508c [1ambda] feat: custom axisSpec
0dbc431 [1ambda] feat: Support transformer
94d837a [1ambda] feat: Automatic spec versioning
74b8b4e [1ambda] fix: Duplicated radio btn id, name
5b88f08 [1ambda] fix: Modify margin of subchart radio btns
019892c [1ambda] feat: Support transform: flatten
0484e1e [1ambda] feat: Support maxAxisCount in axisSpec
936901b [1ambda] feat: Support undefined valueType in axisSpec
7a454ff [1ambda] feat: Cube Transformation
f0ed02f [1ambda] feat: Support same axis types
49985c6 [1ambda] refactor: Refine axis, param spec
d89e223 [1ambda] feat: advanced-transformation-api
75569ce [1ambda] feat: Support multiple charts in UI
e1fcc2e [1ambda] feat: Support multiple charts
97be629 [1ambda] fix: Add singleDimensionAggregatorChanged
676bd7e [1ambda] refactor: Refine transform API
9fb398e [1ambda] feat: Add clearConfig
a8a4fb1 [1ambda] refactor: Add getAxisInSingleDimension func
9768ecf [1ambda] feat: Add groupBase axis option
91ae54d [1ambda] fix: Overflow issue in single aggr
10c80fc [1ambda] feat: AdvancedTransformation
2017-04-11 14:14:46 +00:00
# Writing a new Application
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
< div id = "toc" > < / div >
## What is Apache Zeppelin Application
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
Apache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. While application runs on Interpreter process, it's able to access resources provided by Interpreter through ResourcePool. Output is always rendered by AngularDisplaySystem. Therefore application provides all the possiblities of making interactive graphical application that uses data and processing power of any Interpreter.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
## Make your own Application
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
Writing Application means extending `org.apache.zeppelin.helium.Application` . You can use your favorite IDE and language while Java class files are packaged into jar. `Application` class looks like
```java
/**
* Constructor. Invoked when application is loaded
*/
public Application(ApplicationContext context);
/**
* Invoked when there're (possible) updates in required resource set.
* i.e. invoked after application load and after paragraph finishes.
*/
public abstract void run(ResourceSet args);
/**
* Invoked before application unload.
* Application is automatically unloaded with paragraph/notebook removal
*/
public abstract void unload();
```
You can check example applications under [./zeppelin-examples ](https://github.com/apache/incubator-zeppelin/tree/master/zeppelin-examples ) directory.
## Development mode
In the development mode, you can run your Application in your IDE as a normal java application and see the result inside of Zeppelin notebook.
2017-01-08 15:44:56 +00:00
`org.apache.zeppelin.helium.ZeppelinApplicationDevServer` can run Zeppelin Application in development mode.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```java
// entry point for development mode
public static void main(String[] args) throws Exception {
// add resources for development mode
LocalResourcePool pool = new LocalResourcePool("dev");
pool.put("date", new Date());
2016-07-05 08:36:32 +00:00
// run application in devlopment mode with given resource
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
// in this case, Clock.class.getName() will be the application class name
2017-01-08 15:44:56 +00:00
org.apache.zeppelin.helium.ZeppelinApplicationDevServer devServer = new org.apache.zeppelin.helium.ZeppelinApplicationDevServer(
2016-07-05 08:36:32 +00:00
Clock.class.getName(), pool.getAll());
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
// start development mode
devServer.start();
devServer.join();
}
```
In the Zeppelin notebook, run `%dev run` will connect to application running in development mode.
## Package file
Package file is a json file that provides information about the application.
2016-07-05 08:36:32 +00:00
Json file contains the following information
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2017-04-10 10:07:40 +00:00
```json
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
{
2017-04-10 10:07:40 +00:00
"name" : "[organization].[name]",
"description" : "Description",
"artifact" : "groupId:artifactId:version",
"className" : "your.package.name.YourApplicationClass",
"resources" : [
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
["resource.name", ":resource.class.name"],
["alternative.resource.name", ":alternative.class.name"]
],
2017-04-10 10:07:40 +00:00
"icon" : "< i class = 'icon' > < / i > "
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
}
```
#### name
2016-07-05 08:36:32 +00:00
Name is a string in `[group].[name]` format.
`[group]` and `[name]` allow only `[A-Za-z0-9_]` .
Group is normally the name of an organization who creates this application.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
#### description
2016-07-05 08:36:32 +00:00
A short description about the application
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
#### artifact
Location of the jar artifact.
2016-07-05 08:36:32 +00:00
`"groupId:artifactId:version"` will load artifact from maven repository.
If jar exists in the local filesystem, absolute/relative can be used.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
e.g.
When artifact exists in Maven repository
2016-07-05 08:36:32 +00:00
```
artifact: "org.apache.zeppelin:zeppelin-examples:0.6.0"
```
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
When artifact exists in the local filesystem
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
```
artifact: "zeppelin-example/target/zeppelin-example-0.6.0.jar"
```
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
#### className
Entry point. Class that extends `org.apache.zeppelin.helium.Application`
#### resources
2016-07-05 08:36:32 +00:00
Two dimensional array that defines required resources by name or by className. Helium Application launcher will compare resources in the ResourcePool with the information in this field and suggest application only when all required resources are available in the ResourcePool.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
Resouce name is a string which will be compared with the name of objects in the ResourcePool. className is a string with ":" prepended, which will be compared with className of the objects in the ResourcePool.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
2016-07-05 08:36:32 +00:00
Application may require two or more resources. Required resources can be listed inside of the json array. For example, if the application requires object "name1", "name2" and "className1" type of object to run, resources field can be
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```
resources: [
[ "name1", "name2", ":className1", ...]
]
```
2016-07-05 08:36:32 +00:00
If Application can handle alternative combination of required resources, alternative set can be listed as below.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```
resources: [
[ "name", ":className"],
[ "altName", ":altClassName1"],
...
]
```
2016-07-05 08:36:32 +00:00
Easier way to understand this scheme is
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
```
resources: [
[ 'resource' AND 'resource' AND ... ] OR
[ 'resource' AND 'resource' AND ... ] OR
...
]
```
#### icon
2016-07-05 08:36:32 +00:00
Icon to be used on the application button. String in this field will be rendered as a HTML tag.
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
3577777 [Lee moon soo] Address Ahyoung's comment
0665380 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
83b529c [Lee moon soo] Update unittest
e1b0e79 [Lee moon soo] Update unittest
6785441 [Lee moon soo] interpreter order changed
3249ddb [Lee moon soo] Remove incubating
4030fff [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
4e98cfe [Lee moon soo] Update DependencyResolver
c714a19 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
d558591 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
0467077 [Lee moon soo] update test
d43ab57 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f999f71 [Lee moon soo] Update getParagraphXpath
eaa68c6 [Lee moon soo] print processes
5b5f6c8 [Lee moon soo] trigger ci
1bdea70 [Lee moon soo] Print error message in the test
562f083 [Lee moon soo] CLose interpreter after test
385dab3 [Lee moon soo] Fix xpath in the SparkParagraphIt test
ac963ff [Lee moon soo] Remove unnecessary stacktrace and provide more meaningful message
644add4 [Lee moon soo] Takecare csv/tsv download button position after merge
6a5a19e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
5bef173 [Lee moon soo] Update NoteTest
e54d633 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
88dbc71 [Lee moon soo] change style
6f1343f [Lee moon soo] add license header
4571781 [Lee moon soo] double quote the keys in json
710487b [Lee moon soo] fix syntax error
6c5019e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
3619afd [Lee moon soo] Exclude test data file from rat check
b6811a6 [Lee moon soo] set SPARK_HOME to find bin/interpreter.sh
9110897 [Lee moon soo] arrange buttons
14efb6c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
76de11c [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
bff2217 [Lee moon soo] Fix SparkParagraphIT
e60cd8e [Lee moon soo] Add docs
7f174e3 [Lee moon soo] Add more comments
0671dfd [Lee moon soo] Fix compile error
bfcdf34 [Lee moon soo] Fix style
1dc4409 [Lee moon soo] Remove incubating from example pom.xml
e36d2fb [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e0a2046 [Lee moon soo] Add annotation to api
93fa766 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
99dd0b0 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
e89177e [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
f782fb0 [Lee moon soo] clean pom.xml file
b974eb1 [Lee moon soo] Add horizontal bar visualization example
5225551 [Lee moon soo] let instead of var
b068065 [Lee moon soo] Replace . to _
b6e4141 [Lee moon soo] helper for printing javascript and inject $z
8186daf [Lee moon soo] Update Application Status from RemoteInterpreterProcess event
f07ada1 [Lee moon soo] Construct classpath correctly
7451479 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
ab1de03 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
51fc113 [Lee moon soo] Include examples in CI build
83eba98 [Lee moon soo] UnloadApp on interpreter restart
3b891de [Lee moon soo] Provide resource pool for testing
0730ece [Lee moon soo] Add license into package json
620f79f [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
01a1646 [Lee moon soo] Add example
1f1a3b5 [Lee moon soo] Fix test
864bea1 [Lee moon soo] Merge branch 'master' into ZEPPELIN-732-up
c30f53c [Lee moon soo] null check
03be3a1 [Lee moon soo] Pass required resource to run() method
fade3c1 [Lee moon soo] Handle output update in angular mode
0e4d81c [Lee moon soo] Remove unnecessary log
ec2fdea [Lee moon soo] Match classname correctly
b4ff52f [Lee moon soo] Put last value of scala repl into resource pool
f2ab95d [Lee moon soo] Prevent unnecessary output update
71f814d [Lee moon soo] Better way to find resource dir for InterpreterOutput watcher
024d7fc [Lee moon soo] Dev mode
98f3872 [Lee moon soo] Managed interpreter process and Running interpreter process
b47ca74 [Lee moon soo] Fix tests
5503f9c [Lee moon soo] Improved
be3a1fa [Lee moon soo] Add license header
412480a [Lee moon soo] Fix style
16f6887 [Lee moon soo] Angular object update for helium app
6223cd4 [Lee moon soo] App output display
bd0f467 [Lee moon soo] Style
94b490d [Lee moon soo] initial rest api impl
134bbe6 [Lee moon soo] Change HeliumRegistry constructor argument type
7aeb64a [Lee moon soo] Unload app on paragraph / note removal as well as interpreter unbind
4eaeea7 [Lee moon soo] sync -> async api
9f5c493 [Lee moon soo] Application output
b239f1b [Lee moon soo] Helium application factory
b891b98 [Lee moon soo] HeliumRegistry
568ee54 [Lee moon soo] ApplicationLoader
7424af2 [Lee moon soo] Remove resource on note / paragraph removal
2016-07-03 14:58:25 +00:00
e.g.
```
icon: "< i class = 'fa fa-clock-o' > < / i > "
```