mirror of
https://github.com/apache/zeppelin
synced 2026-05-24 09:38:26 +00:00
90 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Lee moon soo
|
b13651cedf |
[ZEPPELIN-3840] Zeppelin on Kubernetes
### What type of PR is it? This PR adds ability to run Zeppelin on Kubernetes. It aims - Zero configuration to start Zeppelin on Kubernetes. (and Spark on Kubernetes) - Run everything on Kubernetes: Zeppelin, Interpreters, Spark. - Highly customizable to adopt various user configurations and extensions. Key features are - Provides zeppelin-server.yaml file for `kubectl` to run Zeppelin server - All interpreters are automatically running as a Pod. - Spark interpreter automatically configured to use [Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html) - Reverse proxy is configured to access Spark UI To do - [x] Document how reverse proxy for Spark UI works and how to configure custom domain. - [x] Document how to customize zeppelin-server and interpreter yaml. - [x] Document new configurations - [x] Document how to mount volume for notebook and configurations ### How it works #### Run Zeppelin Server on Kubernetes `k8s/zeppelin-server.yaml` is provided to run Zeppelin Server with few sidecars and configurations. This file is easy to publish (user can easily consume it using `curl`), highly customizable while it includes all the necessary things. #### K8s Interpreter launcher This PR adds new module, `launcher-k8s-standard` under `zeppelin/zeppelin-plugins/launcher/k8s-standard/` directory. This launcher is [automatically being selected](https://github.com/apache/zeppelin/pull/3240/files#diff-82fddd2ffb77aaffc4b9cf7b5b1eaa79) when Zeppelin is running on Kubernetes. The launcher both handles Spark interpreter and All other interpreters. The launcher launches interpreter as a Pod using template [k8s/interpreter/100-interpreter-pod.yaml](https://github.com/apache/zeppelin/pull/3240/files#diff-d9ce62e2c992d32f0184d7edb862f3c4). Reason filename has `100-` in prefix is because all files in the directory is consumed in alphabetical order by launcher on interpreter start/stop. User can drop more files here to extend/customize interpreter, and filename can be used to control order. The template is rendered by [jinjava](https://github.com/HubSpot/jinjava). #### Spark interpreter When interpreter group is `spark`, K8sRemoteInterpreterProcess [sets necessary spark configuration](https://github.com/apache/zeppelin/pull/3240/files#diff-6d1d3084f55bdd519e39ede4a619e73dR297) automatically to use [Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html). User doesn't have to configure anything. It uses client mode. #### Spark UI We may make user manually configure port-forward or do something to access Spark UI, but that's not optimal. It is the best when Spark UI is automatically accessible when user have access to Zeppelin UI, without any extra configuration. To enable this, Zeppelin server Pod has a reverse proxy as a sidecar, and it split traffic to Zeppelin server and Spark UI running in the other Pod. It assume both `service.domain.com` and `*.service.domain.com` point the nginx proxy address. `service.domain.com` is directed to ZeppelinServer, `*.service.domain.com` is directed to interpreter Pod. `<port>-<interpreter pod svc name>.service.domain.com` is convention to access any application running in interpreter Pod. If Spark interpreter Pod is running with a name `spark-axefeg` and Spark UI is running on port 4040, ``` 4040-spark-axefeg.service.domain.com ``` is the address to access Spark UI. Default service domain is [local.zeppelin-project.org:8080](https://github.com/apache/zeppelin/pull/3240/files#diff-56ccb2e2c2617b27dbaae866d9431e51R22), while `local.zeppelin-project.org` and `*.local.zeppelin-project.org` point `127.0.0.1`, and it works with `kubectl port-forward`. ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3840 ### How should this be tested? Prepare a Kubernetes cluster with enough resources (cpus > 5, mem > 6g). If you're using [minikube](https://github.com/kubernetes/minikube), check your capacity using `kubectl describe node` command before start. You'll need to build Zeppelin docker image and Spark docker image to test. Please follow guide docs/quickstart/kubernetes.md. To quickly try without building docker images, I have uploaded pre-built image on docker hub `moon/zeppelin:0.9.0-SNAPSHOT`, `moon/spark:2.4.0`. Try following command ``` ZEPPELIN_SERVER_YAML="curl -s https://raw.githubusercontent.com/Leemoonsoo/zeppelin/kubernetes/k8s/zeppelin-server.yaml" $ZEPPELIN_SERVER_YAML | sed 's/apache\/zeppelin:0.9.0-SNAPSHOT/moon\/zeppelin:0.9.0-SNAPSHOT/' | sed 's/spark:2.4.0/moon\/spark:2.4.0/' | kubectl apply -f - ``` And port forward ``` kubectl port-forward zeppelin-server 8080:80 ``` And browse http://localhost:8080 To clean up ``` $ZEPPELIN_SERVER_YAML | sed 's/apache\/zeppelin:0.9.0-SNAPSHOT/moon\/zeppelin:0.9.0-SNAPSHOT/' | sed 's/spark:2.4.0/moon\/spark:2.4.0/' | kubectl delete -f - ``` ### Screenshots (if appropriate) See this video https://youtu.be/7E4ZGn4pnTo ### Future work - Per interpreter docker image - Blocking communication between interpreter Pod. - Spark Interpreter Pod has Role CRUD for any pod/service in the same namespace. Which should be restricted to only Spark executors Pod. - Per note interpreter mode by default when Zeppelin is running on Kubernetes ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Lee moon soo <leemoonsoo@gmail.com> Author: Lee moon soo <moon@apache.org> Closes #3240 from Leemoonsoo/kubernetes and squashes the following commits: |
||
|
Vincenzo Selvaggio
|
794e189718 |
[ZEPPELIN-3654] - New Hazelcast Jet interpreter
### What is this PR for? **New Hazelcast Jet interpreter** Zeppelin has interpreters for different data processing systems like Flink, Spark, Kylin, Ignite, Geode, Beam, etc. Hazelcast Jet is a general purpose distributed data processing engine, built on top of Hazelcast for stream/batch processing, comparable if not better in terms of performance to the engine supported by Zeppelin therefore a perfect candidate for a Zeppelin interpreter. Part of the interpreter is to have a set of utility methods that print out Hazelcast data structures (IMap and ICache) and leverage Zeppelin's built in visualization (%table). What's more, a nice addition is to have the Hazelcast Jet DAG of the pipeline displayed as a network graph using %network display system. ### What type of PR is it? * Feature ### Todos * Utility method to display Hazelcast Jet DAG using %network * Review zeppelin-distribution/src/bin_license/LICENSE file * Any feedback from reviewers ### What is the Jira issue? * [ZEPPELIN-3654] ### How should this be tested? Manually * Start the Zeppelin server * Create a new note with the hazelcastjet interpreter binding * Write an Hazelcast Jet job as per documentation (docs/interpreter/hazelcastjet.md) Unit tests * Run unit tests (HazelcastJetInterpreterTest.java) ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? Yes, added Apache License for Hazelcast Jet. * Is there breaking changes for older versions? No. * Does this needs documentation? Yes, it has been added to the PR, see docs/interpreter/hazelcastjet.md. Author: Vincenzo Selvaggio <vselvaggio@hotmail.it> Closes #3157 from selvinsource/hazelcast-jet-interpreter and squashes the following commits: |
||
|
Vincenzo Selvaggio
|
ac2e957e2d |
[ZEPPELIN-3653] - New Java interpreter
### What is this PR for? **New Java interpreter** There are several Java libraries that could be used to leverage the Data Visualization & Collaboration features of Zeppelin hence the need of a Java interpreter to run any java code / library with no further dependencies. Dependencies to any java library can be added by the end users in the Java Zeppelin interpreter Dependencies settings section. ### What type of PR is it? * Feature ### Todos * Any feedback from reviewers ### What is the Jira issue? * [ZEPPELIN-3653] ### How should this be tested? Manually * Start the Zeppelin server * Create a new note with the java interpreter binding * Write some java code as per documentation (docs/interpreter/java.md) Unit tests * Run unit tests (JavaInterpreterTest.java and JavaInterpreterUtilsTest.java) ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No, the dependency on com.thoughtworks.qdox was already added as part of the Beam Interpreter. * Is there breaking changes for older versions? No. * Does this needs documentation? Yes, it has been added to the PR, see docs/interpreter/java.md. Author: Vincenzo Selvaggio <vselvaggio@hotmail.it> Closes #3092 from selvinsource/java-interpreter and squashes the following commits: |
||
|
Lee moon soo
|
c4da31b2bb |
[DOC] Fix broken link to writing visualizations doc
### What is this PR for?
This PR fixes broken link to doc. Need to be merged branch-0.8.
### What type of PR is it?
Bug Fix
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes #2956 from Leemoonsoo/fix/doc_link_to_helium and squashes the following commits:
|
||
|
Sanjay Dasgupta
|
0c3260e91f |
[Zeppelin 3388] Refactor documentation for ZeppelinContext
### What is this PR for? This PR is a refactoring of the `ZepplinContext` documentation. The description of `ZepplinContext` is now almost entirely contained within the _Spark_ interpreter's documentation ([spark.md](https://github.com/apache/zeppelin/blob/master/docs/interpreter/spark.md)). But `ZepplinContext` has many generic features that are available to all interpreters, and it is important for `ZepplinContext` to have a more visible and independent presence in the Zeppelin documentation. ### What type of PR is it? [Documentation] ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3388 ### How should this be tested? Only documentation. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No (This _is_ documentation) Author: Sanjay Dasgupta <sanjay.dasgupta@gmail.com> Closes #2953 from sanjaydasgupta/zeppelin-3388 and squashes the following commits: |
||
|
Savalek
|
468cea2907 |
[ZEPPELIN-3431] - Fix links, images in docs
### What is this PR for?
Page http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/other_features/notebook_actions.html
contains errors.
Some images are missing and three links does not work.
### What type of PR is it?
[Bug Fix | Documentation]
### What is the Jira issue?
[ZEPPELIN-3431](https://issues.apache.org/jira/browse/ZEPPELIN-3431)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Savalek <def113@mail.ru>
Closes #2947 from Savalek/ZEPPELIN-3431 and squashes the following commits:
|
||
|
Keiji Yoshida
|
888a05d1e4 |
[ZEPPELIN-3077] Cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck
### What is this PR for?
The cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck.
I sometimes come across the issue that the cron scheduler stops working suddenly. According to the thread dump of ZeppelinServer, all of the DefaultQuartzScheduler_Worker threads were waiting for the job's completion and there was no thread to launch a new job.
Here is the contents of the thread dump:
```
"DefaultQuartzScheduler_Worker-10" #76 prio=5 os_prio=0 tid=0x00007fb41d3b4000 nid=0x1b521 sleeping[0x00007fb3daef1000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000c0a7dbf0> (a java.lang.Object)
Locked ownable synchronizers:
- None
"DefaultQuartzScheduler_Worker-9" #75 prio=5 os_prio=0 tid=0x00007fb41d3b2000 nid=0x1b520 waiting on condition [0x00007fb3daff2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000c0a7a470> (a java.lang.Object)
Locked ownable synchronizers:
- None
...
"DefaultQuartzScheduler_Worker-2" #68 prio=5 os_prio=0 tid=0x00007fb41d3c8800 nid=0x1b519 waiting on condition [0x00007fb3da473000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000c0a7a7b0> (a java.lang.Object)
Locked ownable synchronizers:
- None
"DefaultQuartzScheduler_Worker-1" #67 prio=5 os_prio=0 tid=0x00007fb41d3cc800 nid=0x1b518 waiting on condition [0x00007fb3da372000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000c0a7dd90> (a java.lang.Object)
Locked ownable synchronizers:
- None
```
The above thread dump says that all of the worker threads get stuck at https://github.com/apache/zeppelin/blob/v0.7.3/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java#L889.
One way to reproduce this kind of issue is creating a paragraph whose status is "READY" and "disable run". That makes the paragraph status "READY" permanently and `note.isTerminated()` never turns to `true`.
To fix this issue, the following two improvements has been made at this PR:
1. Remove the unnecessary `while (!note.isTerminated()) { ... }` block because the execution of all of the paragraphs is finished after `note.runAll()`.
2. Skip the cron execution if there is a running or pending paragraph. That prevents the Zeppelin cron scheduler from getting stuck by the long running paragraph whose execution duration is greater than the cron execution cycle.
### What type of PR is it?
[Bug]
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3077
### How should this be tested?
* Tested manually.
1. The cron scheduler does not get stuck if there is a paragraph whose status is "READY" and "disable run".
2. The following message is printed on the log file when the cron job is launched while the previous cron job still has been running.
* `execution of the cron job is skipped because there is a running or pending paragraph (note id: XXXXXXXXX)`
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No.
* Is there breaking changes for older versions? No.
* Does this needs documentation? Yes. The behavior of the cron job was changed not to run if there is a running or pending paragraph by this PR. Thus, the documentation `docs/usage/other_features/cron_scheduler.md` was also added by this PR. Its layout is as follow:
<img width="711" alt="screen shot 2017-11-28 at 18 30 54" src="https://user-images.githubusercontent.com/31149688/33312407-20664e02-d46b-11e7-9715-9e2562d5e064.png">
Author: Keiji Yoshida <kjmrknsn@gmail.com>
Closes #2687 from kjmrknsn/ZEPPELIN-3077 and squashes the following commits:
|
||
|
Andrea Santurbano
|
246f696719 |
[ZEPPELIN-3101] updated network label, added link to network display in index.md
### What is this PR for?
The docs index must show the reference to the network visualization as for the the other types
### What type of PR is it?
[Improvement]
### Todos
* [x] - Add missing link
### What is the Jira issue?
[ZEPPELIN-3101](https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-3101)
### How should this be tested?
1. cd `docs/`
2. build: `bundle exec jekyll build --safe`
3. check the link is present
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Andrea Santurbano <santand@gmail.com>
Closes #2702 from conker84/graph-docs and squashes the following commits:
|
||
|
tinkoff-dwh
|
7241348cff |
[ZEPPELIN-2813] revisions comparator
### What is this PR for? Sometimes need to see the difference between versions and to switch to another version and look for changes are not convenient (the page reloaded). This feature allows you to compare any two versions of the notebook. ### What type of PR is it? Feature ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2813 ### How should this be tested? 1 make some commits. Сchange the contents of paragraphs (delete, add, edit) 2 open Revisions comparator 3 compare revisions and check diff ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? yes (updated) * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <tinkoff.dwh@gmail.com> Author: Tinkoff DWH <tinkoff.dwh@gmail.com> Closes #2506 from tinkoff-dwh/ZEPPELIN-2813 and squashes the following commits: |
||
|
conker84
|
e0fafac75c |
[ZEPPELIN-2761] - Neo4j Interpreter
### What is this PR for? This contribution would to introduce Neo4j Cypher intepreter and at the same time provides base APIs that enable other graph databases (or graph framworks such as GraphX or Giraph). ### What type of PR is it? [Feature] ### Todos * [x] - Provide base APIs to manage graph results (under: org.apache.zeppelin.interpreter.graph.*) * [x] - Create the Neo4j intepreter ### What is the Jira issue? [[ZEPPELIN-2761]](https://issues.apache.org/jira/browse/ZEPPELIN-2761) ### How should this be tested? Donwload and execute [Neo4j](https://neo4j.com/download/?ref=home) v3.x, you can also pull a [Docker image](https://neo4j.com/developer/docker/). In order to execute test cases, if you are running Java 7, you need to also provide an environment variable telling the tests where to find Java 8, because Neo4j-the-database needs it to run. ```bash export NEO4J_JAVA=<path/to/java/home> ``` Use this statement to create a dummy dataset ```bash %neo4j UNWIND range(1,100) as id CREATE (p:Person {id:id, name: "Name " + id, age: id % 3}) WITH collect(p) as people UNWIND people as p1 UNWIND range(1,10) as friend WITH p1, people[(p1.id + friend) % size(people)] as p2 CREATE (p1)-[:KNOWS {years: abs(p2.id - p2.id)}]->(p2) ``` Then you can write some simple queries like: ```bash %neo4j MATCH (p:Person)-[r:KNOWS]-(p1:Person) RETURN p, r, p1 LIMIT 10; ``` ```bash %neo4j MATCH (p:Person)-[r:KNOWS]-(p1:Person) RETURN p.id AS ID_A, p.name AS NAME_A, r.years AS YEARS, p1.id AS ID_B, p1.name AS NAME_B LIMIT 20; ``` ### Video  ### Questions: * Does the licenses files need update? **Yes** * Is there breaking changes for older versions? **No** * Does this needs documentation? **Yes** Author: conker84 <santand@gmail.com> Closes #2478 from conker84/master and squashes the following commits: |
||
|
krishna-pandey
|
780f0ebb43 |
[ZEPPELIN-2775] Strict-Transport-Security and X-XSS-Protection Headers
### What is this PR for? The HTTP Strict-Transport-Security response header (often abbreviated as HSTS) is a security feature that lets a web site tell browsers that it should only be communicated with using HTTPS, instead of using HTTP. Note: The Strict-Transport-Security header is ignored by the browser when your site is accessed using HTTP; this is because an attacker may intercept HTTP connections and inject the header or remove it. When your site is accessed over HTTPS with no certificate errors, the browser knows your site is HTTPS capable and will honor the Strict-Transport-Security header. The HTTP X-XSS-Protection response header is a feature of Internet Explorer, Chrome and Safari that stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. ### What type of PR is it? [Bug Fix | Improvement ] ### What is the Jira issue? * [ZEPPELIN-2775](https://issues.apache.org/jira/browse/ZEPPELIN-2775) ### How should this be tested? Make a curl call to Zeppelin? Go to Chrome Browser and select "More Tools" -> "Developer Tools" from the right-side menu. Under Network Section, select any request and check for "Response Headers". You should see below headers along with existing ones. > strict-transport-security:max-age=631138519 > x-xss-protection:1; mode=block <img width="1436" alt="screen shot 2017-07-14 at 8 19 14 pm" src="https://user-images.githubusercontent.com/6433184/28217231-16ce6cee-68d2-11e7-91aa-77ad083612c7.png"> ### Questions: * Does this needs documentation? Author: krishna-pandey <krish.pandey21@gmail.com> Closes #2492 from krishna-pandey/ZEPPELIN-2775 and squashes the following commits: |
||
|
1ambda
|
b5e039fe1b |
[ZEPPELIN-2708][DOCS] feat: Add v scroll to setup, usage menus in navbar
### What is this PR for?
Added v scroll to setup, usage menus in the navbar.
Because some users are experiencing trimmed menus (not sure exact OS, browser versions, See the JIRA issue)
### What type of PR is it?
[Bug Fix | Improvement]
### Todos
DONE
### What is the Jira issue?
[ZEPPELIN-2708](https://issues.apache.org/jira/browse/ZEPPELIN-2708)
### How should this be tested?
1. cd `docs/`
2. run: `bundle exec jekyll serve --watch`
### Screenshots (if appropriate)
#### Before (no scrollbar)
#### After
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes #2451 from 1ambda/ZEPPELIN-2708/provide-V-scrolls-for-navbar-menus and squashes the following commits:
|
||
|
1ambda
|
1c23f21388 |
[ZEPPELIN-2707][DOCS][HOTFIX] fix: broken image URLs in 0.8.0-SNAPSHOT doc
### What is this PR for?
fix: broken image URLs in 0.8.0-SNAPSHOT doc
using the path `/asset` (the absolute path) for image URLs is actually invalid. That's because each version has its own image directory. So they should use the relative path. `{{BASE_PATH}}`
```
➜ asf-zeppelin tree site | grep asset
├── assets # root asset, we shouldn't use it in versioned doc.
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ ├── assets
```
### What type of PR is it?
[Bug Fix]
### Todos
DONE
### What is the Jira issue?
[ZEPPELIN-2707](https://issues.apache.org/jira/browse/ZEPPELIN-2707)
### How should this be tested?
1. cd `docs/`
2. build: `bundle exec jekyll build --safe`
3. check whether links in `_site` include `/docs/0.8.0-SNAPSHOT` as prefix or not
### Screenshots (if appropriate)
#### Current
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/interpreter/overview.html
#### After
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes #2450 from 1ambda/ZEPPELIN-2707/should-use-its-own-asset-directory and squashes the following commits:
|
||
|
1ambda
|
4b6d3e5574 |
[ZEPPELIN-2596] Improving documentation page
### What is this PR for? Improving documentation page. Please check *TODO* and *Screenshots* sections for detail. The motivation is described in [the JIRA ticket](https://issues.apache.org/jira/browse/ZEPPELIN-2583) and discussion is ongoing on the mailing list. ### What type of PR is it? [Improvement | Documentation] ### Todos * [x] - improved the navbar style * [x] - improved the main page * [x] - re-organized content structure * [x] - added tutorial pages: `spark_with_zeppelin.md`, `python_with_zeppelin.md`, `sql_with_zeppelin.md` for overview * [x] - added `multi_user_support.md` page to provide overview * [x] - added the empty `interpreter_binding_mode` page. This will be handed in the different issue: [ZEPPELIN-2582](https://issues.apache.org/jira/browse/ZEPPELIN-2582) * [x] - added the empty `trouble_shooting` page. This can be filled in the following PRs. * [x] - added the empty `useful_developer_tools` page. This can be filled in the following PRs. ### What is the Jira issue? [ZEPPELIN-2596](https://issues.apache.org/jira/browse/ZEPPELIN-2596) ### How should this be tested? 1. checkout 2. `cd docs` 3. `bundle install` (make sure that you have ruby 2.1.0+ and bundle) 4. `bundle exec jekyll serve --watch` 5. open `localhost:4000` ### Screenshots (if appropriate) #### better navbar: before  #### better navbar: after  #### improved main page: before  #### improved main page: after  #### organized content structure: before  #### organized content structure: after  ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - related with docs Author: 1ambda <1amb4a@gmail.com> Closes #2371 from 1ambda/updating-version-doc and squashes the following commits: |
||
|
conker84
|
43926485cd |
[ZEPPELIN-2222] Add Network Visualization
### What is this PR for? This issue is about a new network visualization that can leverage the Property Graph Model (https://github.com/tinkerpop/gremlin/wiki/Defining-a-Property-Graph), but also simple graphs in order to provide a set of base apis that can be used by graph dbs (like Neo4j) or graph processing frameworks (like GraphX or Giraph). ### What type of PR is it? [Feature] Is related to the #1582 ### Todos * [x] - Added the intepreter apis to manage graphs (under the pakage **org.apache.zeppelin.interpreter.graph**) * [x] - Added the frontend apis to manage graphs (via d3js) ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2222 ### How should this be tested? You can download [this notebook](https://gist.github.com/conker84/9574127c2389d08164423894aa93b67f) to test the PR ### Screenshots (if appropriate)  ### Video  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: conker84 <santand@gmail.com> Closes #2125 from conker84/master and squashes the following commits: |
||
|
Nelson Costa
|
87480056a0 |
[ZEPPELIN-2152] Added proxy config support in Helium bundle factory
### What is this PR for? Proxy configuration implementation in Helium bundle factory. Used by Node, NPM and YarnPkg installers. ### What type of PR is it? [Improvement] ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2152 ### How should this be tested? Trigger Helium plugins behind proxy ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Nelson Costa <nelson.costa85@gmail.com> Closes #2363 from necosta/zeppelin2152 and squashes the following commits: |
||
|
Karup
|
be9c1e84d7 |
ZEPPELIN-2543: Add ability to change the log level at runtime
### What is this PR for? Add ability to change the log level at runtime ### What type of PR is it? Improvement ### Todos * [x] - Update/Add the interpreter rest API documentation(edited: added a new Zeppelin server API doc page) ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2543 ### How should this be tested? >➜ ~ curl -X PUT http://localhost:8080/api/log/level/debug {"status":"OK"} >➜ ~ curl -X PUT http://localhost:8080/api/log/level/adx {"status":"NOT_ACCEPTABLE","message":"Not a valid LOG level"} ### Screenshots (if appropriate) NA ### Questions: * Does the licenses files need update? NA * Is there breaking changes for older versions? No * Does this needs documentation? Yes(Will need to the interpreter Rest api doc, if the approach looks fine) Author: Karup <karuppayyar@qubole.com> Closes #2341 from karuppayya/ZEPPELIN-2543 and squashes the following commits: |
||
|
1ambda
|
45cc8a9e8a |
[ZEPPELIN-2217] AdvancedTransformation for Visualization
### What is this PR for? `AdvancedTransformation` has more detailed options while providing existing features of `PivotTransformation` and `ColumnselectorTransformation` which Zeppelin already has  Here are some features which advanced-transformation can provide. 1. **(screenshot)** multiple sub charts 2. **(screenshot)** parameter widgets: `input`, `checkbox`, `option`, `textarea` 3. **(screenshot)** expand/fold axis and parameter panels 4. **(screenshot)** clear axis and parameter panels 5. **(screenshot)** remove duplicated columns in an axis 6. **(screenshot)** limit column count in an axis 7. configurable char axes: `valueType`, `axisType`, `description`, ... 8. configurable chart parameters 9. lazy transformation 10. parsing parameters automatically based on their type: `int`, `float`, `string`, `JSON` 11. multiple transformation methods 12. re-initialize whole configuration based on spec hash. 13. **(screenshot)** shared axis #### API Details: Spec `AdvancedTransformation` requires `spec` which includes axis and parameter details for charts. - Let's create 2 sub-charts called `simple-line` and `step-line`. - Each sub chart can have different `axis` and `parameter` depending on their requirements. ```js constructor(targetEl, config) { super(targetEl, config) const spec = { charts: { 'simple-line': { sharedAxis: true, /** set if you want to share axes between sub charts, default is `false` */ axis: { 'xAxis': { dimension: 'multiple', axisType: 'key', }, 'yAxis': { dimension: 'multiple', axisType: 'aggregator'}, 'category': { dimension: 'multiple', axisType: 'group', }, }, parameter: { 'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', }, 'yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', }, 'dashLength': { valueType: 'int', defaultValue: 0, description: 'the length of dash', }, }, }, 'step-line': { axis: { 'xAxis': { dimension: 'single', axisType: 'unique', }, 'yAxis': { dimension: 'multiple', axisType: 'value', }, }, parameter: { 'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', }, 'yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', }, 'noStepRisers': { valueType: 'boolean', defaultValue: false, description: 'no risers in step line', widget: 'checkbox', }, }, }, } this.transformation = new AdvancedTransformation(config, spec) } ``` #### API Details: Axis Spec | Field Name | Available Values (type) | Description | | --- | --- | --- | |`dimension` | `single` | Axis can contains only 1 column | |`dimension` | `multiple` | Axis can contains multiple columns | |`axisType` | `key` | Column(s) in this axis will be used as `key` like in `PivotTransformation`. These columns will be served in `column.key` | |`axisType` | `aggregator` | Column(s) in this axis will be used as `value` like in `PivotTransformation`. These columns will be served in `column.aggregator` | |`axisType` | `group` | Column(s) in this axis will be used as `group` like in `PivotTransformation`. These columns will be served in `column.group` | |`axisType` | (string) | Any string value can be used here. These columns will be served in `column.custom` | |`maxAxisCount` | (int) | The maximum column count that this axis can contains. (unlimited if `undefined`) | |`valueType` | (string) | Describe the value type just for annotation | Here is an example. ```js axis: { 'xAxis': { dimension: 'multiple', axisType: 'key', }, 'yAxis': { dimension: 'multiple', axisType: 'aggregator'}, 'category': { dimension: 'multiple', axisType: 'group', maxAxisCount: 2, valueType: 'string', }, }, ``` #### API Details: Parameter Spec | Field Name | Available Values (type) | Description | | --- | --- | --- | |`valueType` | `string` | Parameter which has string value | |`valueType` | `int` | Parameter which has int value | |`valueType` | `float` | Parameter which has float value | |`valueType` | `boolean` | Parameter which has boolean value used with `checkbox` widget usually | |`valueType` | `JSON` | Parameter which has JSON value used with `textarea` widget usually. `defaultValue` should be `""` (empty string). This ||`defaultValue` | (any) | Default value of this parameter. `JSON` type should have `""` (empty string) | |`description` | (string) | Description of this parameter. This value will be parsed as HTML for pretty output | |`widget` | `input` | Use [input](https://developer.mozilla.org/en/docs/Web/HTML/Element/input) widget. This is the default widget (if `widget` is undefined)| |`widget` | `checkbox` | Use [checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) widget. | |`widget` | `textarea` | Use [textarea](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/textarea) widget. | |`widget` | `option` | Use [select + option](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) widget. This parameter should have `optionValues` field as well. | |`optionValues` | (Array<string>) | Available option values used with the `option` widget | Here is an example. ```js parameter: { // string type, input widget 'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', }, // boolean type, checkbox widget 'inverted': { widget: 'checkbox', valueType: 'boolean', defaultValue: false, description: 'invert x and y axes', }, // string type, option widget with `optionValues` 'graphType': { widget: 'option', valueType: 'string', defaultValue: 'line', description: 'graph type', optionValues: [ 'line', 'smoothedLine', 'step', ], }, // HTML in `description` 'dateFormat': { valueType: 'string', defaultValue: '', description: 'format of date (<a href="https://docs.amcharts.com/3/javascriptcharts/AmGraph#dateFormat">doc</a>) (e.g YYYY-MM-DD)', }, // JSON type, textarea widget 'yAxisGuides': { widget: 'textarea', valueType: 'JSON', defaultValue: '', description: 'guides of yAxis ', }, ``` #### API Details: Transformer Spec `AdvancedTransformation` supports 3 transformation methods. The return value will depend on the transformation method type. ```js const spec = { charts: { 'simple': { /** default value of `transform.method` is the flatten cube. */ axis: { ... }, parameter: { ... } }, 'cube-group': { transform: { method: 'cube', }, axis: { ... }, parameter: { ... }, } 'no-group': { transform: { method: 'raw', }, axis: { ... }, parameter: { ... }, } ``` | Field Name | Available Values (type) | Description | | --- | --- | --- | |`method` | `object` | designed for [amcharts: serial](https://www.amcharts.com/demos/date-based-data/) | |`method` | `array` | designed for [highcharts: column](http://jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/highcharts/demo/column-basic/) | |`method` | `drill-down` | designed for [highcharts: drill-down](http://jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/highcharts/demo/column-drilldown/) | |`method` | `raw` | will return the original `tableData.rows` | Whatever you specified as `transform.method`, the `transformer` value will be always function for lazy computation. ```js // advanced-transformation.util#getTransformer if (transformSpec.method === 'raw') { transformer = () => { return rows; } } else if (transformSpec.method === 'array') { transformer = () => { ... return { ... } } } ``` #### Feature Details: Automatic parameter parsing Advanced transformation will parse parameter values automatically based on their type: `int`, `float`, `string`, `JSON` - See also `advanced-transformation-util.js#parseParameter` #### Feature Details: re-initialize the whole configuration based on spec hash ```js // advanced-transformation-util#initializeConfig const currentVersion = JSON.stringify(spec) if (!config.spec || !config.spec.version || config.spec.version !== currentVersion) { spec.version = currentVersion // reset config... } ``` #### Feature Details: Shared Axes If you set `sharedAxis` to `true` in chart specification, then these charts will share their axes. (default is `false`) ```js const spec = { charts: { 'column': { transform: { method: 'array', }, sharedAxis: true, axis: { ... }, parameter: { ... }, }, 'stacked': { transform: { method: 'array', }, sharedAxis: true, axis: { ... } parameter: { ... }, }, ```  #### API Details: Usage in Visualization#render() Let's assume that we want to create 2 sub-charts called `basic` and `no-group`. - https://github.com/1ambda/zeppelin-ultimate-line-chart (an practical example) ```js drawBasicChart(parameter, column, transformer) { const { ... } = transformer() } drawNoGroupChart(parameter, column, transformer) { const { ... } = transformer() } render(data) { const { chart, parameter, column, transformer, } = data if (chart === 'basic') { this.drawBasicChart(parameter, column, transformer) } else if (chart === 'no-group') { this.drawNoGroupChart(parameter, column, transformer) } } ``` ### What type of PR is it? [Feature] ### Todos NONE ### What is the Jira issue? [ZEPPELIN-2217](https://issues.apache.org/jira/browse/ZEPPELIN-2217) ### How should this be tested? 1. Clone https://github.com/1ambda/zeppelin-ultimate-line-chart 2. Create a symbolic link `ultimate-line-chart.json` into `$ZEPPELIN_HOME/helium` 3. Modify the `artifact` value to proper absolute path considering your local machine. 4. Install the above visualization in `localhost:9000/#helium` 5. Test it ### Screenshots (if appropriate) #### 1. *(screenshot)* multiple sub charts  #### 2. *(screenshot)* parameter widgets: `input`, `checkbox`, `option`, `textarea`  #### 3. *(screenshot)* expand/fold axis and parameter panels  #### 4. *(screenshot)* clear axis and parameter panels  #### 5. *(screenshot)* remove duplicated columns in an axis  #### 6. *(screenshot)* limit column count in an axis  ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO Author: 1ambda <1amb4a@gmail.com> Closes #2098 from 1ambda/ZEPPELIN-2217/advanced-transformation and squashes the following commits: |
||
|
dlukyanov
|
53a28a3a91 |
Groovy Interpreter for Apazhe Zeppelin [ZEPPELIN-2176]
### What is this PR for? Groovy Interpreter ### What type of PR is it? Feature ### Todos * [ Tests ] - Task * [ Documentation ] - Task ### What is the Jira issue? [ZEPPELIN-2176] ### How should this be tested? Follow the groovy interpreter documentation samples ### Questions: * Does the licenses files need update? YES * Is there breaking changes for older versions? NO * Does this needs documentation? YES Author: dlukyanov <dlukyanov@ukr.net> Author: dm <dm> Closes #2135 from dlukyanov/master and squashes the following commits: |
||
|
AhyoungRyu
|
9e8d7eb90c |
[Minor][ZEPPELIN-2328] Separate Helium related docs from 'Contribution' section
### What is this PR for?
Actually writing "Visualization", "Spell" or "Application" type of Helium package is not a direct contribution to Zeppelin. So i created "Helium Framework" section and moved "Writing Zeppelin Visualization", "Writing Zeppelin Spell", and "Writing Zeppelin Application" under "Helium Framework" from "Contribution".
### What type of PR is it?
Documentation
### What is the Jira issue?
[ZEPPELIN-2328](https://issues.apache.org/jira/browse/ZEPPELIN-2328)
### Screenshots (if appropriate)
- Before
<img src="https://cloud.githubusercontent.com/assets/10060731/24444171/743d8c7e-14a0-11e7-8a10-ec02596d2a19.png" width="300px">
- After
<img src="https://cloud.githubusercontent.com/assets/10060731/24444245/aecfe274-14a0-11e7-8488-99086b1db415.png" width="300px">
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Closes #2202 from AhyoungRyu/ZEPPELIN-2328 and squashes the following commits:
|
||
|
AhyoungRyu
|
8bb37c2c2e |
[DOCS][ZEPPELIN-2140] Add docs for notebookRepo REST API
### What is this PR for? Added a docs page for notebookRepo REST API based on [NotebookRepoRestApi.java](https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRepoRestApi.java). And this PR will cover "notebook repository reload endpoint" : #2043 as well. ### What type of PR is it? Documentation ### What is the Jira issue? [ZEPPELIN-2140](https://issues.apache.org/jira/browse/ZEPPELIN-2140) ### How should this be tested? Just checking screenshots will be faster! ### Screenshots (if appropriate) - In navbar <img width="300" alt="screen shot 2017-02-25 at 11 02 02 pm" src="https://cloud.githubusercontent.com/assets/10060731/23331707/76551c8a-fbae-11e6-99b8-cc686e208c39.png"> - API description list <img width="751" alt="screen shot 2017-02-25 at 10 54 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/23331704/57dd7d4c-fbae-11e6-8cc2-189fc9e68ece.png"> <img width="715" alt="screen shot 2017-02-25 at 10 54 29 pm" src="https://cloud.githubusercontent.com/assets/10060731/23331708/85f722a0-fbae-11e6-9b40-7f4c98761dd6.png"> <img width="730" alt="screen shot 2017-02-25 at 10 54 38 pm" src="https://cloud.githubusercontent.com/assets/10060731/23331709/8882b93a-fbae-11e6-9fe2-95bc2620e2c9.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #2067 from AhyoungRyu/ZEPPELIN-2140 and squashes the following commits: |
||
|
Jun Kim
|
5e75145ac8 |
[ZEPPELIN-1859] Add MongoNotebookRepo
### What is this PR for? This PR adds Mongo notebook storage. The reason that I made this feature is for HA(High Availability). S3 and Git storage are the only available method for HA as far as I know. I'm managing Ambari cluster in my lab, but Zeppelin is the most vulnerable part of it. Because one server contains all Zeppelin notes. Therefore, by deploying MongoDB's [replica set](https://docs.mongodb.com/manual/replication/) and using it as Zeppelin notebook storage, I would like to achieve HA. #### The way to use Mongo DB as notebook storage ```sh export ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.MongoNotebookRepo ``` or at `zeppelin-site.xml`: ```xml <property> <name>zeppelin.notebook.storage</name> <value>org.apache.zeppelin.notebook.repo.MongoNotebookRepo</value> <description>notebook persistence layer implementation</description> </property> ``` #### Configurable environment variables * `ZEPPELIN_NOTEBOOK_MONGO_URI` MongoDB connection URI * `ZEPPELIN_NOTEBOOK_MONGO_DATABASE` Database name * `ZEPPELIN_NOTEBOOK_MONGO_COLLECTION` Collection name * `ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT` If `true`, automatically import your local notes. Default `false` They can be configured at `zeppelin-site.xml` as well: * `zeppelin.notebook.mongo.uri` * `zeppelin.notebook.mongo.database` * `zeppelin.notebook.mongo.collection` * `zeppelin.notebook.mongo.autoimport` #### Future work If we use Mongo DB's [oplog tailing](https://docs.mongodb.com/manual/core/replica-set-oplog/), maybe multi-server architecture is possible. ### What type of PR is it? [Feature] ### Todos * [ ] - Write a documentation for Mongo storage ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1859 ### How should this be tested? #### Install MongoDB (if you don't have) ```sh brew update brew install mongodb ``` #### Build Zepppelin ```sh mvn clean package -DskipTests ``` #### Run Zeppelin wih Mongo storage ```sh export ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.MongoNotebookRepo export ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT=true bin/zeppelin-daemon.sh restart ``` The default database and collection names are `zeppelin`, `notes` respectively. And `ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT` option will automatically import your `local notes` that don't exist in MongoDB. #### Check whether a document in MongoDB updated Create, update, remove a note and open mongo shell: ```sh mongo zeppelin ``` And check state of the note is the same as you think: ```sh db.notes.findOne({_id: '<NOTE_ID_THAT_YOU_WANT_TO_SEE>'}) ``` #### Confirm that configurations works ```sh export ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.MongoNotebookRepo export ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT=true export ZEPPELIN_NOTEBOOK_MONGO_DATABASE=otherdb export ZEPPELIN_NOTEBOOK_MONGO_COLLECTION=mynotes export ZEPPELIN_NOTEBOOK_MONGO_URI=mongodb://localhost:27017 bin/zeppelin-daemon.sh restart ``` The collection `mynotes` should be created in db `otherdb`. Let's check it! ```sh mongo otherdb db.mynotes.count() ``` The result should not be zero. #### Confirm that configurations from `zeppelin-site.xml` works Open your `conf/zeppelin-site.xml` file (copy from `zeppelin-site.xml.template` if you don't have one), and comment lines below: ```xml <!-- <property> <name>zeppelin.notebook.storage</name> <value>org.apache.zeppelin.notebook.repo.VFSNotebookRepo</value> <description>notebook persistence layer implementation</description> </property> --> ``` And add lines below: ```xml <property> <name>zeppelin.notebook.storage</name> <value>org.apache.zeppelin.notebook.repo.MongoNotebookRepo</value> <description>notebook persistence layer implementation</description> </property> <property> <name>zeppelin.notebook.mongo.uri</name> <value>mongodb://localhost</value> <description>MongoDB connection URI used to connect to a MongoDB database server</description> </property> <property> <name>zeppelin.notebook.mongo.database</name> <value>zepl</value> <description>database name for notebook storage</description> </property> <property> <name>zeppelin.notebook.mongo.collection</name> <value>notes</value> <description>collection name for notebook storage</description> </property> <property> <name>zeppelin.notebook.mongo.autoimport</name> <value>false</value> <description>import local notes into MongoDB automatically on startup</description> </property> ``` This time we will import a note via `mongoimport`. I made it possible to import a note from JSON just in case. ```sh cd $ZEPPELIN_HOME/notebook/<NOTE_ID_YOU_WANT_TO_IMPORT> mongoimport --db zepl --collection notes --file note.json ``` Ensure that your environment variables are clean(just reopen your terminal if you are not), and restart zeppelin: ```sh bin/zeppelin-daemon.sh restart ``` Open browser and go to `localhost:8080`. The note that you imported should be shown. ### Questions: * Does the licenses files need update? Maybe...? I used [java-mongodb-driver](https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver/3.4.1) which has *The Apache Software License, Version 2.0* * Is there breaking changes for older versions? NO * Does this needs documentation? YES Author: Jun Kim <i2r.jun@gmail.com> Closes #1826 from tae-jun/ZEPPELIN-1859 and squashes the following commits: |
||
|
1ambda
|
90e8b80b56 |
[ZEPPELIN-2070][DOCS] Create Docs for Spell
### What is this PR for? Docs for Spell which was added by #1940 ### What type of PR is it? [Documentation] ### Todos * [x] - Add docs for spell * [x] - Fix typo in vis docs ### What is the Jira issue? [ZEPPELIN-2070](https://issues.apache.org/jira/browse/ZEPPELIN-2070) ### How should this be tested? - `cd docs` - `/usr/local/bin/bundle exec jekyll serve --watch` - Open `http://localhost:4000/development/writingzeppelinspell.html` ### Screenshots (if appropriate)   ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - YES Author: 1ambda <1amb4a@gmail.com> Closes #1998 from 1ambda/ZEPPELIN-2070/docs-for-spell and squashes the following commits: |
||
|
AhyoungRyu
|
d053e5b333 |
[MINOR][ZEPPELIN-2100] Enable to go back to zeppelin.apache.org in docs site
### What is this PR for? Currently there is no link to go back to [zeppelin.apache.org](https://zeppelin.apache.org) in each docs site. It's a bit inconvenient. e.g. In [https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/](https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/), if I click Zeppelin main logo, it keeps me staying in docs main page not the root: [zeppelin.apache.org](https://zeppelin.apache.org). So I separated main logo in navbar to "Zeppelin" and "version". And linked [production_url](https://github.com/apache/zeppelin/blob/master/docs/_config.yml#L34) and [BASE_PATH](https://github.com/apache/zeppelin/blob/master/docs/_config.yml#L62) to each of them. Please see the below screenshot img. ### What type of PR is it? Improvement ### What is the Jira issue? [ZEPPELIN-2100](https://issues.apache.org/jira/browse/ZEPPELIN-2100) ### How should this be tested? Run docs site locally under `ZEPPELIN_HOME/docs` as described in here: [Run website locally](https://github.com/apache/zeppelin/blob/master/docs/README.md#run-website-locally) ### Screenshots (if appropriate) In [https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/](https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/), - Before <img width="400px" src="https://cloud.githubusercontent.com/assets/10060731/22860466/6064c964-f142-11e6-9bc1-bbd34fa42c18.png"> - After <img width="500px" src="https://cloud.githubusercontent.com/assets/10060731/22860469/69364568-f142-11e6-963d-7b6ab33330c3.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #2008 from AhyoungRyu/change/mainLogoUrlToOfficialSite and squashes the following commits: |
||
|
Lee moon soo
|
300f753234 |
[ZEPPELIN-1619] Load js package as a plugin visualization
### What is this PR for?
Current helium plugin application api (experimental) requires create library in java class, and need to create both backend / frontend code in the package. Which is good if your plugin requires both frontend and backend code running.
However, when user just want to make new visualization which completely runs on front-end side in javascript, creating helium application in java project and taking care of backend code can be bit of overhead and barrier for javascript developers.
This PR adds capability to load pure javascript package as a visualization.
### how it works
1. create (copy, download) 'helium package json' file into `ZEPPELIN_HOME/helium` directory.
The json file point visualization js package in npm repository or local file system in `artifact` field.
`type` field in the json file need to be `VISUALIZATION`
Here's an example (zeppelin-examples/zeppelin-example-horizontalbar/zeppelin-example-horizontalbar.json)
```
{
"type" : "VISUALIZATION",
"name" : "zeppelin_horizontalbar",
"description" : "Horizontal Bar chart (example)",
"artifact" : "./zeppelin-examples/zeppelin-example-horizontalbar",
"icon" : "<i class='fa fa-bar-chart rotate90flipX'></i>"
}
```
2. Go to helium GUI menu. (e.g. http://localhost:8080/#/helium).
The menu will list all available packages.
<img width="796" alt="writing_visualization_helium_menu" src="https://cloud.githubusercontent.com/assets/1540981/21749660/0f401c10-d558-11e6-9961-b6d0a9c023d8.png">
3. click 'enable' in any package want to use.
Once a visualization package is enabled, `HeliumVisualizationFactory` will collect all enabled visualizations and create js bundle on the fly.
4. js bundle will be loaded on notebook and additional visualization becomes available
### Programming API to create new plugin visualization.
Simply extends [visualization.js](https://github.com/apache/zeppelin/blob/master/zeppelin-web/src/app/visualization/visualization.js) and overrides some methods, such as
```
/**
* get transformation
*/
getTransformation() {
// override this
};
/**
* Method will be invoked when data or configuration changed
*/
render(tableData) {
// override this
};
/**
* Refresh visualization.
*/
refresh() {
// override this
};
/**
* method will be invoked when visualization need to be destroyed.
* Don't need to destroy this.targetEl.
*/
destroy() {
// override this
};
/**
* return {
* template : angular template string or url (url should end with .html),
* scope : an object to bind to template scope
* }
*/
getSetting() {
// override this
};
```
This is exactly the same api that built-in visualization uses.
an example implementation included `zeppelin-examples/zeppelin-example-horizontalbar/horizontalbar.js`.
Actually [all built-in visualizations](https://github.com/apache/zeppelin/tree/master/zeppelin-web/src/app/visualization/builtins) are example
### Packaging and publishing visualization
Each visualization will need `package.json` file (e.g. `zeppelin-examples/zeppelin-example-horizontalbar/package.json`) to be packaged.
Package can be published in npm repository or package can be deployed to the local filesystem.
`zeppelin-examples/zeppelin-example-horizontalbar/` is an example package that is deployed in the local filesystem
### Development mode
First, locally install and enable your development package by setting `artifact` field to the development directory.
Then run zeppelin-web in visualization development mode with following command
```
cd zeppelin-web
npm run visdev
```
When you have change in your local development package, just reload your notebook. Then Zeppelin will automatically rebuild / reload the package.
Any feedback would be appreciated!
### What type of PR is it?
Feature
### Todos
* [x] - Load plugin visualization js package on runtime
* [x] - Make the feature works in zeppelin Binary package
* [x] - Show loading indicator while 'enable' / 'disable' package
* [x] - Add document
* [x] - Add license of new dependency
* [x] - Development mode
* [x] - Propagate error to front-end
* [x] - Display multiple versions of a package.
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1619
### How should this be tested?
Build Zeppelin with `-Pexamples` flag. That'll install example visualization package `horizontalbar`.
You'll able to select `horizontalbar` along with other built-in visualizations
To test npm online package install capability, Place [zeppelin-bubble.json](https://github.com/Leemoonsoo/zeppelin-bubble/blob/master/zeppelin-bubble.json) in hour local registry (`ZEPPELIN_HOME/helium`) and enable it in Helium gui menu.
And then zeppelin will download package from npm repository and load.
### Questions:
* Does the licenses files need update? yes
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #1842 from Leemoonsoo/ZEPPELIN-1619-rebased and squashes the following commits:
|
||
|
AhyoungRyu
|
1697275717 |
[MINOR][ZEPPELIN-1913] Separate "Apache Zeppelin Configuration" from quickstart page
### What is this PR for? I think [Quick Start](https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/install.html) page should contain minimum contents to get start Zeppelin for beginners. If not, this can be making the entrance barrier high. But after "SSL configuration" contents were added, configuration section came so long. So I created new page `configuration.md` and separated the all configuration related contents from Quick Start page. ### What type of PR is it? Documentation ### What is the Jira issue? [ZEPPELIN-1913](https://issues.apache.org/jira/browse/ZEPPELIN-1913) ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) - before (quickstart guide + Zeppelin configuration in one page)  - after (separated two page)  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1855 from AhyoungRyu/separate/confContents and squashes the following commits: |
||
|
AhyoungRyu
|
c854fda936 |
[DOCS] Separate "interpreter exec hooks (experimental)" from interpreter overview page
### What is this PR for?
After #1470 merged, "(Experimental) Interpreter Execution Hooks" is added under https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/manual/interpreters.html page. But i think we need to keep this page as simple as possible since it's explaining the basic concept of Zeppelin interpreters. So I separated "(Experimental) Interpreter Execution Hooks" section from `interpreters.md` and created new page `interpreterexechooks.md`.
And also fixed some markdown rendering issues.
### What type of PR is it?
Documentation
### What is the Jira issue?
N/A
### How should this be tested?
Please see below screenshots :)
### Screenshots (if appropriate)
- Before (under [manual/interpreter.md](https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/manual/interpreters.html#experimental-interpreter-execution-hooks)
<img width="437" alt="screen shot 2016-12-15 at 5 05 49 pm]" src="https://cloud.githubusercontent.com/assets/10060731/21216362/a3bb89f0-c2e9-11e6-9678-8e6d8749229b.png">
- After
<img width="300" alt="dropdown" src="https://cloud.githubusercontent.com/assets/10060731/21216570/dcca63f0-c2ea-11e6-90a9-969d363b423a.png">
<img width="437" alt="screen shot 2016-12-15 at 5 05 58 pm" src="https://cloud.githubusercontent.com/assets/10060731/21216363/a6c2d82e-c2e9-11e6-920c-a603e25e1699.png">
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Closes #1768 from AhyoungRyu/separate-eventhook-section and squashes the following commits:
|
||
|
Prabhjyot Singh
|
31f584cfee |
[ZEPPELIN-1320] Run zeppelin interpreter process as web front end user
Have recreated this from https://github.com/apache/zeppelin/pull/1322 ### What is this PR for? While running a Notebook using shell, spark, python uses same user as which zeppelin server is running. Which means these interprets have same permission on file system as zeppelin server. IMO users should be able to impersonate themselves as a complete security system. ### What type of PR is it? [Improvement] ### Todos - [x] - Update doc - [x] - FIX NPEs - [x] - FIX CI ### What is the Jira issue? - [ZEPPELIN-1320](https://issues.apache.org/jira/browse/ZEPPELIN-1320) ### How should this be tested? - Enable shiro auth in shiro.ini - Add ssh key for the same user you want to try and impersonate (say user1). ``` adduser user1 ssh-keygen ssh user1localhost mkdir -p .ssh cat ~/.ssh/id_rsa.pub | ssh user1localhost 'cat >> .ssh/authorized_keys' ``` - Start zeppelin server, try and run following in paragraph in a notebook - Go to interpreter setting page, and enable "User Impersonate" in any of the interpreter (in my example its shell interpreter) ``` %sh whoami ``` Check that it should run as new user, i.e. "user1" ### Screenshots (if appropriate)  ### Questions: - Does the licenses files need update? no - Is there breaking changes for older versions? no - Does this needs documentation? yes Author: Prabhjyot Singh <prabhjyotsingh@gmail.org> Closes #1554 from prabhjyotsingh/ZEPPELIN-1320-2 and squashes the following commits: |
||
|
Rafal Wojdyla
|
f127237fb1 |
Closes [ZEPPELIN-1505] Add Scio interpreter
### What is this PR for? Closes #ZEPPELIN-1505. Adds Scio interpreter. Scio is a Scala DSL on top of Dataflow/Beam. ### What type of PR is it? Improvement - ZEPPELIN-1505 ### Todos - [x] - test integration with zeppelin context (zeppelin context is too tightly coupled withs spark) - [x] - what to do about code completion? - [x] - add more tests? - [x] - add helpers to display data - [x] - add doc in `docs/interpreter/scio` ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1505 ### How should this be tested? ``` mvn -pl scio,zeppelin-display,zeppelin-interpreter -Dtest='org.apache.zeppelin.scio.*' -DfailIfNoTests=false test ``` ### Screenshots  ### Questions: - Does the licenses files need update? ~~no~~ yes - Is there breaking changes for older versions? no - Does this needs documentation? yes (included in the PR) Author: Rafal Wojdyla <rav@spotify.com> Closes #1471 from ravwojdyla/scio and squashes the following commits: |
||
|
zhongjian
|
c9adf7161f |
ZEPPELIN-415 document for Apache Kylin Interpreter
### What is this PR for? This is document for Apache Kylin Interpreter for Apache Zeppelin ### What type of PR is it? Documentation ### Todos Done ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-415 ### How should this be tested? ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? This is document Author: zhongjian <jiatuer@163.com> Author: Jason <jiatuer@163.com> Closes #1576 from janzhongi/master and squashes the following commits: |
||
|
Lee moon soo
|
5b1b811540 |
[ZEPPELIN-1644] make document easier to follow key instructions
### What is this PR for? Doc should deliver key features and recommended usage more simple and easy way. - docs/install/install.md has lots of duplicated section with README.md. - docs/install/install.md includes install from binary as well as build from source. I've seen that makes some beginners try download binary and then source build it again. - recommended and key usage need to be highlighted. - Be less verbose in key instructions. Move optional, additional info from in the middle of key instruction to end of the each page. ### What type of PR is it? Improvement ### Todos * [x] - improve doc ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1644 ### How should this be tested? Run doc locally ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <moon@apache.org> Closes #1615 from Leemoonsoo/ZEPPELIN-1644 and squashes the following commits: |
||
|
hyonzin
|
201d601224 |
[ZEPPELIN-1552] Search button goes to next line when display's width shortens.
### What is this PR for? In document page(http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/), the search button goes to next line when display's width shortens. ### What type of PR is it? [Improvement] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1552 ### Screenshots (if appropriate) <img width="927" alt="2016-10-15 2 04 00" src="https://cloud.githubusercontent.com/assets/6567102/19398321/0d76a1c4-9287-11e6-86e0-9f120c00b143.png"> when (768px ≤ width < 992px) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyonzin <hyeonjin507@gmail.com> Author: AhyoungRyu <ahyoungryu@apache.org> Author: 정현진 <hyeonjin507@gmail.com> Closes #1525 from hyonzin/ZEPPELIN-1552 and squashes the following commits: |
||
|
Jeff Zhang
|
465c51a419 |
ZEPPELIN-335. Pig Interpreter
### What is this PR for? Based on #338 , I refactor most of pig interpreter. As I don't think the approach in #338 is the best approach. In #338, we use script `bin/pig` to launch pig script, it is different to control that job (hard to kill and get progress and stats info). In this PR, I use pig api to launch pig script. Besides that I implement another interpreter type `%pig.query` to leverage the display system of zeppelin. For the details you can check `pig.md` ### What type of PR is it? [Feature] ### Todos * Syntax Highlight * new interpreter type `%pig.udf`, so that user can write pig udf in zeppelin directly and don't need to build udf jar manually. ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-335 ### How should this be tested? Unit test is added and also manual test is done ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Author: Ali Bajwa <abajwa@hortonworks.com> Author: AhyoungRyu <ahyoungryu@apache.org> Author: Jeff Zhang <zjffdu@gmail.com> Closes #1476 from zjffdu/ZEPPELIN-335 and squashes the following commits: |
||
|
rawkintrevo
|
523868ed8d |
[DOCS] Zeppelin Flink Spark tutorial
Moon recommended I migrate a How To tutorial I wrote into the website.
http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-users/201511.mbox/browser
I didn't create a JIRA issue, it doesn't look like that is nessicary for website updates? If it is, I'm sorry I'll go make one. Sorry, I'm new.
Author: rawkintrevo <trevor.d.grant@gmail.com>
Closes #418 from rawkintrevo/zeppelin-flink-spark-tutorial and squashes the following commits:
|
||
|
astroshim
|
c7ce709f35 |
[ZEPPELIN-1279] Zeppelin with CDH5.x docker document.
### What is this PR for? This PR is for the documentation of running zeppelin with CDH docker environment. and This PR is the part of https://issues.apache.org/jira/browse/ZEPPELIN-1198. Tested CDH5.7 on ubuntu. ### What type of PR is it? Documentation ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1281 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Author: AhyoungRyu <ahyoungryu@apache.org> Author: HyungSung <hsshim@nflabs.com> Closes #1451 from astroshim/ZEPPELIN-1281 and squashes the following commits: |
||
|
mahmoudelgamal
|
403c8c4e79 |
[ZEPPELIN-682] New interpreter for Apache Beam (incubating)/DataFlow
## What is this PR for? The PR is a interpreter for [Apache Beam](http://beam.incubator.apache.org) which is an open source unified platform for data processing pipelines. A pipeline can be build using one of the Beam SDKs. The execution of the pipeline is done by different Runners . Currently, Beam supports Apache Flink Runner, Apache Spark Runner, and Google Dataflow Runner. ### What type of PR is it? - Feature ### Todos * Test case * Review Comments * Documentation ### What is the Jira issue? * [ZEPPELIN-682] ### How should this be tested? - Start the Zeppelin server - The prefix of interpreter is `%beam` and then write your code with required imports and the runner ### Screenshots (if appropriate)   ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: mahmoudelgamal <mahmoudf.elgamal@gmail.com> Author: mfelgamal <mahmoudf.elgamal@gmail.com> Author: Fouad <fuad.assayadi@gmail.com> Closes #1334 from mfelgamal/beam-interpreter-static-repl-7 and squashes the following commits: |
||
|
astroshim
|
cee58aa038 |
[ZEPPELIN-1279] Spark on Mesos Docker.
### What is this PR for? This PR is for the documentation of running zeppelin on production environments especially spark on mesos via Docker. Related issue is https://github.com/apache/zeppelin/pull/1227 and https://github.com/apache/zeppelin/pull/1318 and I got a lot of hints from https://github.com/sequenceiq/hadoop-docker. Tested on ubuntu. ### What type of PR is it? Documentation ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1279 ### How should this be tested? You can refer to https://github.com/apache/zeppelin/blob/master/docs/README.md#build-documentation. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Author: AhyoungRyu <fbdkdud93@hanmail.net> Author: HyungSung <hsshim@nflabs.com> Closes #1389 from astroshim/ZEPPELIN-1279 and squashes the following commits: |
||
|
astroshim
|
eccfe0076b |
[ZEPPELIN-1280][Spark on Yarn] Documents for running zeppelin on production environments using docker.
### What is this PR for? This PR is for the documentation of running zeppelin on production environments especially spark on yarn. Related issue is https://github.com/apache/zeppelin/pull/1227 and I got a lot of hints from https://github.com/sequenceiq/hadoop-docker. Tested on ubuntu. ### What type of PR is it? Documentation ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1280 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Author: AhyoungRyu <fbdkdud93@hanmail.net> Author: HyungSung <hsshim@nflabs.com> Closes #1318 from astroshim/ZEPPELIN-1280 and squashes the following commits: |
||
|
AhyoungRyu
|
85d4df4f0c |
[ZEPPELIN-1219] Add searching feature to Zeppelin docs site
### What is this PR for? As more and more document pages are added, it's really hard to find specific pages. So I added searching feature to Zeppelin documentation site([jekyll](https://jekyllrb.com/) based site) using [lunr.js](http://lunrjs.com/). - **How does it work?** I created [`search_data.json`]( |
||
|
astroshim
|
b965503291 |
[ZEPPELIN-1198][Spark Standalone] Documents for running zeppelin on production environments.
### What is this PR for? This PR is for documentation for running zeppelin on production environments. ### What type of PR is it? Documentation ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1198 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Closes #1227 from astroshim/ZEPPELIN-1198/standalone and squashes the following commits: |
||
|
Babu Prasad Elumalai
|
57c264da2d |
BigQuery Interpreter for Apazhe Zeppelin[ZEPPELIN-1153]
### What is this PR for? Google BigQuery is a popular no-ops datawarehouse. This commit will enable Apache Zeppelin users to perform BI and Analytics on their datasets in BigQuery. ### What type of PR is it? Feature ### Todos * Make bigquery interpreter appear in the interpreters section in the UI * Build SQL completion * Authorization of non-gcp ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1153 ### How should this be tested? copy conf/zeppelin-site.xml.template to conf/zeppelin-site.xml Add org.apache.zeppelin.bigquery.bigQueryInterpreter to property zeppelin.interpreters in zeppelin-site.xml Start Zeppelin Add BigQuery Interpreter with your project ID Create new note with %bsql.sql and run your SQL against public datasets in bigquery. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Babu Prasad Elumalai <babupe@google.com> Author: babupe <babupe@google.com> Author: Alexander Bezzubov <bzz@apache.org> Closes #1170 from babupe/babupe-bigquery and squashes the following commits: |
||
|
AhyoungRyu
|
6bd4ede7e5 |
[DOC][MINOR] Add shell interpreter docs to _navigation.html
### What is this PR for? After #1087 merged, a new docs `shell.md` was added. But in the docs website, still Shell interpreter link points to `pleasecontribute.html`. So I changed this link, applied TOC and added more descriptions. ### What type of PR is it? Documentation ### Todos * [x] - Change `pleasecontribute.html` -> `shell.html` * [x] - Apply TOC(table of contents) * [x] - Add more description to `shell.md` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1138 from AhyoungRyu/improve/shell-docs and squashes the following commits: |
||
|
AhyoungRyu
|
d87f2e5dfb |
[ZEPPELIN-1054] Improve "Credentials" UI
### What is this PR for? Currently, users can add new their credential info for data source authentication in Zeppelin "Credentials" menu. Even though it was saved successfully, they can't see the whole list of credentials in Zeppelin UI. This PR enables to `get` all credential list, `edit` and `remove` via UI. *NOTE : Since this patch was implemented based on #1030 API, should be tested after #1030 merged.* ### What type of PR is it? Improvement & Documentation ### Todos * [x] - rename `interpreter_authorization.md` -> `datasource_authorization.md` * [x] - remove `Interpreter Authorization` section (since we don't have this feature yet : [ZEPPELIN-945](https://issues.apache.org/jira/browse/ZEPPELIN-945)) * [x] - rebase after #1030 & #1064 merged * [ ] - address reviews ### What is the Jira issue? [ZEPPELIN-1054](https://issues.apache.org/jira/browse/ZEPPELIN-1054) ### How should this be tested? 1. Apply this patch and build `zeppelin-web` as described in [here](https://github.com/apache/zeppelin/tree/master/zeppelin-web#configured-environment). 2. Go to `Credentials` menu. 3. Add new credentials -> you can see the credential info in the credential list table. 4. You can edit & delete them. -> Compare with `conf/credentials.json` ### Screenshots (if appropriate) - Before <img width="952" alt="screen shot 2016-06-28 at 12 37 10 am" src="https://cloud.githubusercontent.com/assets/10060731/16407604/69b0c4d8-3cc9-11e6-8284-9abe2969cdc1.png"> - After  If there is no credential <img width="957" alt="screen shot 2016-06-28 at 12 19 46 am" src="https://cloud.githubusercontent.com/assets/10060731/16407620/7838995e-3cc9-11e6-90ba-1bd0173a1b49.png"> - `datasource_authorization.md` <img width="845" alt="screen shot 2016-06-28 at 7 58 24 pm" src="https://cloud.githubusercontent.com/assets/10060731/16439169/d4026034-3d6a-11e6-930f-86de12e5fc49.png"> <img width="851" alt="screen shot 2016-06-28 at 7 58 44 pm" src="https://cloud.githubusercontent.com/assets/10060731/16439170/d62f2842-3d6a-11e6-9d3f-ecc5cda29c77.png"> <img width="846" alt="screen shot 2016-06-28 at 8 00 20 pm" src="https://cloud.githubusercontent.com/assets/10060731/16439200/fed58390-3d6a-11e6-9aa2-8cff5a1b7b66.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1100 from AhyoungRyu/ZEPPELIN-1054 and squashes the following commits: |
||
|
AhyoungRyu
|
3ad809f6b5 |
[DOC][ZEPPELIN-732] Helium Application
### What is this PR for? After #836 and #1031 merged into master branch, I also applied TOC(table of contents) to newly added `writingzeppelinapplication.md`. And also added this docs link under `index.md`'s docs list. ### What type of PR is it? Documentation ### Todos * [x] - Apply TOC(table of contents) to `writingzeppelinapplication.md` and add this docs to `index.md` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1133 from AhyoungRyu/docs/ZEPPELIN-732 and squashes the following commits: |
||
|
Lee moon soo
|
9463fb8547 |
[ZEPPELIN-732] Helium Application
### What is this PR for?
This PR implements pluggable Application Framework.
Here's summary of the change.
**Applicaiton**
Base class of user application. User application should extends `org.apache.zeppelin.helium.Application`.
It looks like
```java
public abstract class Application {
/**
* When application loaded
* params context Application context
*/
public Application(ApplicationContext context) {
this.context = context;
}
/**
* This method can be invoked multiple times before unload(),
* Invoked just after application loaded or when paragraph output is updated
* param args Required resources from paragraph
*/
public abstract void run(ResourceSet args)
throws ApplicationException, IOException;
/**
* this method is invoked just before application is removed
*/
public abstract void unload() throws ApplicationException;
}
```
**HeliumPackage**
`org.apache.zeppelin.helium.HeliumPackage` keeps information of application package.
Package informations includes
```
{
type : "APPLICATION", // type of this package
name : "zeppelin.app", // application name. [organizationName].[appName] format
description : "App description", // description
artifact : "groupId:artifactId:version", // artifact name for maven repository
className : "org.apache.zeppelin.helium.MyApp", // application class that extends org.apache.zeppelin.helium.Application
resources : [[]], // required resource
icon : '<i class="fa fa-clock-o"></i>' // icon
}
```
`resources` field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.
For example,
```
resources : [
[ "name1", ":java.util.Date"],
[ "name1", "name2"]
]
```
Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".
Once resources are found, they'll be passed to `Application.run()` method
**Package Registry**
`org.apache.zeppelin.helium.HeliumRegistry` provides list of available HeliumPackages.
Currently `HeliumLocalRegistry` is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.
**Development mode**
`org.apache.zeppelin.interpreter.dev.*` package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.
**Application Suggestion**
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.
User can see and select available application for this paragraph.
**Application selection**
After application is loaded, application icon will be displayed. So user can select and switch between apps.
if built-in visualization is available, application icon will be displayed next to built-in visualizations
**Experimental**
While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.
### What type of PR is it?
Feature
### Todos
* [x] - Helium Application framework
* [x] - App launcher
* [x] - App display and selectors
* [x] - Package registry
* [x] - Development mode
* [x] - Make CI green
* [x] - Improve comment in source codes
* [x] - Documentation
* [x] - Examples
* [x] - Mark experimental
* [ ] - Review
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-732
### How should this be tested?
Will be updated with examples
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <moon@apache.org>
Closes #836 from Leemoonsoo/ZEPPELIN-732-up and squashes the following commits:
|
||
|
astroshim
|
c348161df0 |
[ZEPPELIN-1023] Add more credential apis.
### What is this PR for? This PR is for supporting various Credential APIs for users. ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1023 ### How should this be tested? - to create credential information. ``` curl -XPUT -H "Content-Type: application/json" "http://localhost:8080/api/credential" -d '{"entity" : "e1", "username" : "user1", "password" : "testpass"}' ``` - to get credential information. ``` curl -XGET -H "Content-Type: application/json" "http://localhost:8080/api/credential" ``` - to remove credential entity information. ``` curl -XDELETE -H "Content-Type: application/json" "http://localhost:8080/api/credential/e1" ``` - to remove all credential information. ``` curl -XDELETE -H "Content-Type: application/json" "http://localhost:8080/api/credential" ``` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Author: AhyoungRyu <fbdkdud93@hanmail.net> Author: HyungSung <hsshim@nflabs.com> Closes #1030 from astroshim/ZEPPELIN-1023 and squashes the following commits: |
||
|
AhyoungRyu
|
09f0ebda9b |
[DOC] Update doc image in Explore Zeppelin UI page
### What is this PR for? There were several changes in Zeppelin UI after #860, #1006, #1013, #1081 so update screenshot of documents accordingly. ### What type of PR is it? Documentation ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Author: Mina Lee <mina.hyeji.lee@gmail.com> Author: Mina Lee <minalee@apache.org> Closes #1089 from minahlee/doc/ZEPPELIN-1002 and squashes the following commits: |
||
|
AhyoungRyu
|
5975125f18 |
[ZEPPELIN-1018] Apply auto "Table of Contents" generator to Zeppelin docs website
### What is this PR for? I added auto TOC(Table of Contents) generator for Zeppelin documentation website. TOC can help people looking through whole contents at a glance and finding what they want quickly. I just added `<div id="toc"></div>` to the each documentation header. [`toc`](https://github.com/apache/zeppelin/compare/master...AhyoungRyu:ZEPPELIN-1018?expand=1#diff-85af09fb498a5667ea455391533f945dR3) recognize `<h2>` & `<h3>` as a title in the docs and it automatically generate TOC. So I set a rule for this work. (I'll write this rule on `docs/CONTRIBUTING.md` or [docs/howtocontributewebsite](https://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/development/howtocontributewebsite.html)). ``` # Level-1 Heading <- Use only for the main title of the page ## Level-2 Heading <- Start with this one ### Level-3 heading <- Only use this one for child of Level-2 toc only recognize Level-2 & Level-3 ``` Please see the below attached screenshot image. ### What type of PR is it? Improvement & Documentation ### Todos * [x] - Add TOC generator * [x] - Apply TOC(`<div id="toc"></div>`) to every documentation and reorganize each headers(apply the above rule) * [x] - Fix some broken code block in several docs * [x] - Apply TOC to `r.md` (Currently R docs has some duplicated info since [this one]( |
||
|
Lee moon soo
|
4efb39f450 |
[ZEPPELIN-1046] bin/install-interpreter.sh for netinst package
### What is this PR for? Implementation of bin/install-interpreter.sh for netinst package which suggested in the [discussion](http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Ask-opinion-regarding-0-6-0-release-package-tp3298p3314.html). Some usages will be ``` # download all interpreters provided by Apache Zeppelin project bin/install-interpreter.sh --all # download an interpreter with name (for example markdown interpreter) bin/install-interpreter.sh --name md # download an (3rd party) interpreter with specific maven artifact name bin/install-interpreter.sh --name md -t org.apache.zeppelin:zeppelin-markdown:0.6.0-SNAPSHOT ``` If it looks fine, i'll continue the work (refactor code, and add test) ### What type of PR is it? Feature ### Todos * [x] - working implementation * [x] - refactor * [x] - add test ### What is the Jira issue? * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/ * Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533] ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: Lee moon soo <moon@apache.org> Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1042 from Leemoonsoo/netinst and squashes the following commits: |
||
|
AhyoungRyu
|
5ddc1ef87e |
[ZEPPELIN-996] Improve first page and dropdown menu in documentation site
### What is this PR for? Current Zeppelin documentation site is little bit hard to find a way for Zeppelin beginners. It will not easy to improve this at a time, but I did the below as a start of this work. 1. Restructured dropdown menu and added each category names 2. Added a overview list(with short description) to first page of website (index.md) so that people can look through the overall contents in Zeppelin website at a glance. (as [Apache Spark](http://spark.apache.org/docs/latest/#where-to-go-from-here) and [Apache Mesos](http://mesos.apache.org/documentation/latest/) does) Please see the attached screenshot images :) ### What type of PR is it? Improvement & Documentation ### Todos * [x] - Change outdated screenshot images * [x] - Combine `text.md`, `table.md` and `html.md` to `basicdisplaysystem.md` * [x] - Fix dead link in `virtual_machine.md` * [x] - Improve dropdown menu and reorder * [x] - Improve first page(`index.md`) * [x] - Combine with #995 after it is merged into master ### What is the Jira issue? [ZEPPELIN-996](https://issues.apache.org/jira/browse/ZEPPELIN-996) ### How should this be tested? 1. Apply this patch and [build the docs website with jekyll](https://github.com/apache/zeppelin/tree/master/docs#build-documentation) 2. Check the first page(index.html) and dropdown menu ### Screenshots (if appropriate) - Dropdown menu  - First page <img width="717" alt="screen shot 2016-06-14 at 1 28 58 pm" src="https://cloud.githubusercontent.com/assets/10060731/16058631/3ab2cb6c-3234-11e6-95f4-180290df3d02.png"> <img width="694" alt="screen shot 2016-06-14 at 1 29 11 pm" src="https://cloud.githubusercontent.com/assets/10060731/16058639/43d68918-3234-11e6-921c-28436bfca33d.png"> <img width="649" alt="screen shot 2016-06-14 at 1 29 39 pm" src="https://cloud.githubusercontent.com/assets/10060731/16058650/501ec6d6-3234-11e6-9292-53ae84acc18a.png"> <img width="684" alt="screen shot 2016-06-14 at 1 29 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/16058643/4637c8f2-3234-11e6-9b12-a233906f4c8b.png"> <img width="650" alt="screen shot 2016-06-14 at 1 30 12 pm" src="https://cloud.githubusercontent.com/assets/10060731/16058655/56c5af22-3234-11e6-8d29-9b7937728948.png"> <img width="636" alt="screen shot 2016-06-14 at 1 30 22 pm" src="https://cloud.githubusercontent.com/assets/10060731/16058656/58d1187e-3234-11e6-9171-ab7390b4a526.png"> ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1004 from AhyoungRyu/ZEPPELIN-996 and squashes the following commits: |
