mirror of
https://github.com/apache/zeppelin
synced 2026-05-24 09:38:26 +00:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
1ambda
|
f35d5de7d6 |
[ZEPPELIN-2069] Helium Package Configuration
### What is this PR for?
Supporting helium package configurations. I attached screenshots.
#### Implementation details.
In case of spell, spell developer can create config spec in their `package.json` and it will be part of `helium.json` which is consumed by Zeppelin.
```
"config": {
"repeat": {
"type": "number",
"description": "How many times to repeat",
"defaultValue": 1
}
},
```
1. Persists conf per `package namepackage version` since each version can require different configs even if they are the same package.
2. Saves key-value config only. Since config spec (e.g `type`, `desc`, `defaultValue`) can be provided. So it's not efficient save both of them.
3. Extracts config related functions to `helium.service.js` since it can be used not only in `helium.controller.js` for view but also should be used in `paragraph.controller.js`, `result.controller.js` for executing spell.
### What type of PR is it?
[Feature]
### Todos
* [x] - create config view in `/helium`
* [x] - persist config per `packageversion`
* [x] - pass config to spell
### What is the Jira issue?
[ZEPPELIN-2069](https://issues.apache.org/jira/browse/ZEPPELIN-2069)
### How should this be tested?
- Build with examples `mvn clean package -Phelium-dev -Pexamples -DskipTests;`
- Open `/helium` page
- Update the `echo-spell` config
- Execute the spell like the screenshot below. (you don't need to refresh the page, since executing spell will fetch config from server)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes #1982 from 1ambda/ZEPPELIN-2069/helium-package-configuration and squashes the following commits:
|
||
|
1ambda
|
0589e27e7b |
[ZEPPELIN-2008] Introduce Spell
### What is this PR for? Implemented **Spell** as one of Helium categories. *Technically, it's the frontend interpreter* runs on browser not backend. Spell can provide many benefits. 1. Anyone can install, remove easily using helium package registry by #1936 2. Implementing spell is extremely easier rather than adding backend interpreter 3. Can use existing javsacript libraries. (e.g [flowchart.js](http://flowchart.js.org/), [sequence diagram js](https://github.com/bramp/js-sequence-diagrams), ...). This enable us to add many visualization tools. Imagine that you can implement some custom interpreters with few lines of code like [flowchart-spell-example](https://github.com/apache/zeppelin/compare/master...1ambda:ZEPPELIN-2008/introduce-spell?expand=1#diff-364845b20d68e4d94688e44fef03da98) 4. The most important thing is, spell is not only interpreter but also display system. Because it runs on browser. So we can use spell display system with another spell **Display System with Spell** (see the screenshot section below) **In future**, we will be able to combine existing backend interpreters with spell like (**not supported in this PR cause we need to modify backend code a lot**) ``` // if we have markdown spell, we can use `%markdown` display in the spark interpreter %spark val calculated = doSomething() println(s"%markdown _${calculated}) ``` I added some examples. Checkout `echo`, `markdown`, `translator`, `flowchart` spells. ### What type of PR is it? [Feature] ### Todos * [x] - Add `SPELL` as one of Helium categories. * [x] - Implement framework code (`zeppelin-spell`) * [x] - Make some examples (flowchart, google translator, markdown, echo) * [x] - Support custom display system * [x] - Fix some bugs in `HeliumBundleFactory` * [x] - Save spell rendering result into `note.json` while broadcasting to other websocket clients * [x] - Fix `renderText` for stream output ### What is the Jira issue? [ZEPPELIN-2008](https://issues.apache.org/jira/browse/ZEPPELIN-2008) ### How should this be tested? - Build `mvn clean package -Phelium-dev -Pexamples -DskipTests;` - Go to helium page `http://localhost:8080/#/helium` - Enable all spells - Go to a notebook and refresh - Follow actions in the screenshots below. ### Screenshots (if appropriate) #### Flowchart Spell (Sample)  #### Google Translator Spell (Sample)  #### Display System with Spell  ### Questions: * Does the licenses files need update - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - YES, but framework can be enhanced so i would like to defer to write document right now. Author: 1ambda <1amb4a@gmail.com> Closes #1940 from 1ambda/ZEPPELIN-2008/introduce-spell and squashes the following commits: |
||
|
Mina Lee
|
61aeeaf10f |
Bump up version to 0.8.0-SNAPSHOT
### What is this PR for? Bump up version to 0.8.0-SNAPSHOT Author: Mina Lee <minalee@apache.org> Closes #1883 from minahlee/0.8.0-SNAPSHOT and squashes the following commits: |
||
|
Lee moon soo
|
300f753234 |
[ZEPPELIN-1619] Load js package as a plugin visualization
### What is this PR for?
Current helium plugin application api (experimental) requires create library in java class, and need to create both backend / frontend code in the package. Which is good if your plugin requires both frontend and backend code running.
However, when user just want to make new visualization which completely runs on front-end side in javascript, creating helium application in java project and taking care of backend code can be bit of overhead and barrier for javascript developers.
This PR adds capability to load pure javascript package as a visualization.
### how it works
1. create (copy, download) 'helium package json' file into `ZEPPELIN_HOME/helium` directory.
The json file point visualization js package in npm repository or local file system in `artifact` field.
`type` field in the json file need to be `VISUALIZATION`
Here's an example (zeppelin-examples/zeppelin-example-horizontalbar/zeppelin-example-horizontalbar.json)
```
{
"type" : "VISUALIZATION",
"name" : "zeppelin_horizontalbar",
"description" : "Horizontal Bar chart (example)",
"artifact" : "./zeppelin-examples/zeppelin-example-horizontalbar",
"icon" : "<i class='fa fa-bar-chart rotate90flipX'></i>"
}
```
2. Go to helium GUI menu. (e.g. http://localhost:8080/#/helium).
The menu will list all available packages.
<img width="796" alt="writing_visualization_helium_menu" src="https://cloud.githubusercontent.com/assets/1540981/21749660/0f401c10-d558-11e6-9961-b6d0a9c023d8.png">
3. click 'enable' in any package want to use.
Once a visualization package is enabled, `HeliumVisualizationFactory` will collect all enabled visualizations and create js bundle on the fly.
4. js bundle will be loaded on notebook and additional visualization becomes available
### Programming API to create new plugin visualization.
Simply extends [visualization.js](https://github.com/apache/zeppelin/blob/master/zeppelin-web/src/app/visualization/visualization.js) and overrides some methods, such as
```
/**
* get transformation
*/
getTransformation() {
// override this
};
/**
* Method will be invoked when data or configuration changed
*/
render(tableData) {
// override this
};
/**
* Refresh visualization.
*/
refresh() {
// override this
};
/**
* method will be invoked when visualization need to be destroyed.
* Don't need to destroy this.targetEl.
*/
destroy() {
// override this
};
/**
* return {
* template : angular template string or url (url should end with .html),
* scope : an object to bind to template scope
* }
*/
getSetting() {
// override this
};
```
This is exactly the same api that built-in visualization uses.
an example implementation included `zeppelin-examples/zeppelin-example-horizontalbar/horizontalbar.js`.
Actually [all built-in visualizations](https://github.com/apache/zeppelin/tree/master/zeppelin-web/src/app/visualization/builtins) are example
### Packaging and publishing visualization
Each visualization will need `package.json` file (e.g. `zeppelin-examples/zeppelin-example-horizontalbar/package.json`) to be packaged.
Package can be published in npm repository or package can be deployed to the local filesystem.
`zeppelin-examples/zeppelin-example-horizontalbar/` is an example package that is deployed in the local filesystem
### Development mode
First, locally install and enable your development package by setting `artifact` field to the development directory.
Then run zeppelin-web in visualization development mode with following command
```
cd zeppelin-web
npm run visdev
```
When you have change in your local development package, just reload your notebook. Then Zeppelin will automatically rebuild / reload the package.
Any feedback would be appreciated!
### What type of PR is it?
Feature
### Todos
* [x] - Load plugin visualization js package on runtime
* [x] - Make the feature works in zeppelin Binary package
* [x] - Show loading indicator while 'enable' / 'disable' package
* [x] - Add document
* [x] - Add license of new dependency
* [x] - Development mode
* [x] - Propagate error to front-end
* [x] - Display multiple versions of a package.
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1619
### How should this be tested?
Build Zeppelin with `-Pexamples` flag. That'll install example visualization package `horizontalbar`.
You'll able to select `horizontalbar` along with other built-in visualizations
To test npm online package install capability, Place [zeppelin-bubble.json](https://github.com/Leemoonsoo/zeppelin-bubble/blob/master/zeppelin-bubble.json) in hour local registry (`ZEPPELIN_HOME/helium`) and enable it in Helium gui menu.
And then zeppelin will download package from npm repository and load.
### Questions:
* Does the licenses files need update? yes
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #1842 from Leemoonsoo/ZEPPELIN-1619-rebased and squashes the following commits:
|
||
|
Jongyoul Lee
|
e32ccf35d7 |
ZEPPELIN-1862 Move packages related Helium into a new module named helium-dev
### What is this PR for? Reducing zeppelin-interpreter by moving helium files into a new module ### What type of PR is it? [Improvement] ### Todos * [x] - Move files into a new module ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-1862 ### How should this be tested? If you develop a new helium application, you have to depend helium-dev into your package ### Screenshots (if appropriate) N/A ### Questions: * Does the licenses files need update? N/A * Is there breaking changes for older versions? N/A, AFAIK, there's no helium application for now * Does this needs documentation? Yes, I'll make a new PR for handling it Author: Jongyoul Lee <jongyoul@gmail.com> Closes #1871 from jongyoul/ZEPPELIN-1862 and squashes the following commits: |
||
|
1ambda
|
ad3586450a |
[ZEPPELIN-1695] Centralize libs, plugin versions in all pom.xml
### What is this PR for? **First of all, this PR doesn't affect on runtime application behaivor and existing build processes. Just abstracting variables in pom.xml** The main goal of this PR is bringing consistency and maintainability in all pom.xml. I referred these 2 projects. - [apache/spark: pom.xml](https://github.com/apache/spark/blob/master/pom.xml) - [apache/storm: pom.xml](https://github.com/apache/storm/blob/master/pom.xml) Currently, all libraries and plugins have duplicated version fields since we are not using `dependencyManagement`, `pluginManagement` efficiently in pom.xml This results in - unmanaged plugin, library versions (someone might update only a child's library version. it happend in our pom.xml files) - so many duplicated code blocks (bad) - fragmented plugin, library versions (some project uses junit 4.11 while another uses 4.12) These are some examples ```xml // hard coded, non consistent version management <dependency> <groupId>org.apache.thrift</groupId> <artifactId>libthrift</artifactId> <version>${libthrift.version}</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpcore</artifactId> <version>4.3.3</version> </dependency> ``` ```xml // a/pom.xml <plugin> <artifactId>maven-enforcer-plugin</artifactId> <version>1.3.1</version> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>2.8</version> // b/pom.xml <plugin> <artifactId>maven-enforcer-plugin</artifactId> <version>1.3.1</version> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>2.8</version> // c/pom.xml <plugin> <artifactId>maven-enforcer-plugin</artifactId> <version>1.3.1</version> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>2.8</version> ``` ### What type of PR is it? [Refactoring] ### Todos - [x] refactor all pom.xml ### What is the Jira issue? [ZEPPELIN-1695](https://issues.apache.org/jira/browse/ZEPPELIN-1695) ### How should this be tested? Since this PR doesn't affect runtime behavior at all, CI test would be enough. ### Questions: * Does the licenses files need update - NO * Is there breaking changes for older versions - NO * Does this needs documentation - NO Author: 1ambda <1amb4a@gmail.com> Closes #1668 from 1ambda/chore/centralize-maven-pom-versions and squashes the following commits: |
||
|
Luciano Resende
|
f9bc7a9fcc |
[ZEPPELIN-1409] Refactor RAT build on Travis.CI configuration
### What is this PR for? Create a specific build for checking license compliance with RAT and avoid running these checks on every build that compose the PR build. As for normal development builds, this follows the same pattern used for maven tests, RAT is enabled to run by default, but now there is support disabling it with -DskipRat. Travis CI will run RAT once, on the RAT build, and disable RAT checks on all other build profiles. ### What type of PR is it? [Enhancement] ### What is the Jira issue? * [ZEPPELIN-1409](https://issues.apache.org/jira/browse/ZEPPELIN-1409) Author: Luciano Resende <lresende@apache.org> Closes #1401 from lresende/build and squashes the following commits: |
||
|
Mina Lee
|
04da56403b |
[MINOR] Change url in pom.xml files
### What is this PR for? Set project url to `http://zeppelin.apache.org` in pom.xml files ### What type of PR is it? Refactoring ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <minalee@apache.org> Closes #1221 from minahlee/pom_url and squashes the following commits: |
||
|
Mina Lee
|
e0f77d68e8 |
Bump up version to 0.7.0-SNAPSHOT
### What is this PR for? Bump up version to 0.7.0-SNAPSHOT Author: Mina Lee <minalee@apache.org> Closes #1016 from minahlee/0.7.0-SNAPSHOT and squashes the following commits: |
||
|
Lee moon soo
|
9463fb8547 |
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.
User can see and select available application for this paragraph.
**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.
if built-in visualization is available, application icon will be displayed next to built-in visualizations
**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
|
