### What is this PR for?
Simple PR to only apply dynamic forms for LivySqlInterpreter, this make the behavior consistent with zeppelin's built-in spark interpreter.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3034
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2651 from zjffdu/ZEPPELIN-3034 and squashes the following commits:
39eff37 [Jeff Zhang] ZEPPELIN-3034. Only apply dynamic form for LivySqlnterpreter
### What is this PR for?
Allow an override of default Spark logic in building the URL path and redirect the Zeppelin user directly to a configured reverse proxy path
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2949
### How should this be tested?
* Set "zeppelin.spark.uiWebUrl" to http://localhost/spark-reverse-proxy-path
* Run Spark code
* Open Spark UI
* Expected to be redirected to http://localhost/spark-reverse-proxy-path
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? N
* Is there breaking changes for older versions? N
* Does this needs documentation? N
Author: Nelson Costa <nelson.costa85@gmail.com>
Closes#2596 from necosta/zeppelin2949 and squashes the following commits:
2ae8f2d [Nelson Costa] [ZEPPELIN-2949] Last amendments
118339f [Nelson Costa] [ZEPPELIN-2949] Updated documentation
826bad4 [Nelson Costa] [ZEPPELIN-2949] Updated documentation
0a56299 [Nelson Costa] [ZEPPELIN-2949] Allow custom Spark UI URL
### What is this PR for?
Follow up of #2577. Main changes on Interpreter
* Add throw `InterpreterException` which is checked exception for the abstract methods of `Interpreter`, this would enforce the interpreter implementation to throw `InterpreterException`.
* field name refactoring.
* `property` -> `properties`
* `getProperty()` --> `getProperties()`
* Introduce launcher layer for interpreter launching. Currently we only use shell script to launch interpreter, but it could be any other service or component to launch interpreter, such as livy server , other 3rd party tools or even we may create a separate module for interpreter launcher
* abstract cass `InterpreterLauncher`
* For now, only 2 implementation: `ShellScriptLauncher` & `SparkInterpreterLauncher`. We could add method in class `Interpreter` to allow interpreter to specify its own launcher class, but it could be future work.
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2685
### How should this be tested?
Unit test is covered. `ShellScriptLauncherTest` & `SparkInterpreterLauncherTest`
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2592 from zjffdu/ZEPPELIN-2685 and squashes the following commits:
17dc2f1 [Jeff Zhang] address comments
e545cc3 [Jeff Zhang] ZEPPELIN-2685. Improvement on Interpreter class
### What is this PR for?
This PR is trying to allow user to add custom http headers when calling livy rest api. User just need to specify `zeppelin.livy.http.headers` in livy interpreter setting
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2953
### How should this be tested?
Outline the steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2600 from zjffdu/ZEPPELIN-2953 and squashes the following commits:
720d8d0 [Jeff Zhang] ZEPPELIN-2953 Allow custom http header for livy interpreter
### What is this PR for?
This contribution would to introduce Neo4j Cypher intepreter and at the same time provides base APIs that enable other graph databases (or graph framworks such as GraphX or Giraph).
### What type of PR is it?
[Feature]
### Todos
* [x] - Provide base APIs to manage graph results (under: org.apache.zeppelin.interpreter.graph.*)
* [x] - Create the Neo4j intepreter
### What is the Jira issue?
[[ZEPPELIN-2761]](https://issues.apache.org/jira/browse/ZEPPELIN-2761)
### How should this be tested?
Donwload and execute [Neo4j](https://neo4j.com/download/?ref=home) v3.x, you can also pull a [Docker image](https://neo4j.com/developer/docker/).
In order to execute test cases, if you are running Java 7, you need to also provide an environment variable telling the tests where to find Java 8, because Neo4j-the-database needs it to run.
```bash
export NEO4J_JAVA=<path/to/java/home>
```
Use this statement to create a dummy dataset
```bash
%neo4j
UNWIND range(1,100) as id
CREATE (p:Person {id:id, name: "Name " + id, age: id % 3}) WITH collect(p) as people
UNWIND people as p1
UNWIND range(1,10) as friend
WITH p1, people[(p1.id + friend) % size(people)] as p2
CREATE (p1)-[:KNOWS {years: abs(p2.id - p2.id)}]->(p2)
```
Then you can write some simple queries like:
```bash
%neo4j
MATCH (p:Person)-[r:KNOWS]-(p1:Person)
RETURN p, r, p1
LIMIT 10;
```
```bash
%neo4j
MATCH (p:Person)-[r:KNOWS]-(p1:Person)
RETURN p.id AS ID_A, p.name AS NAME_A, r.years AS YEARS, p1.id AS ID_B, p1.name AS NAME_B
LIMIT 20;
```
### Video
### Questions:
* Does the licenses files need update? **Yes**
* Is there breaking changes for older versions? **No**
* Does this needs documentation? **Yes**
Author: conker84 <santand@gmail.com>
Closes#2478 from conker84/master and squashes the following commits:
5fb97ce [conker84] review by felixcheung 18/09
1e3d611 [conker84] Review by felixcheung (09/08)
aaf618e [conker84] Review by 1ambda 26/07
b5c2c66 [conker84] Added docs
9eb568d [conker84] Final behavior after discussion
2f88e98 [conker84] Moved business logic to the connection manager
8e4690e [conker84] Create connection manager class
35b4e29 [conker84] First commit
### What is this PR for?
This is the first version for supporting yarn-cluster of `SparkInterpreter`. I just delegate all the function to `spark-submit` as yarn-cluster is natively supported by spark, we don't need to reinvent the wheel. But there's still improvement to be done in future, e.g. I put some spark specific logic in `InterpreterSetting` which is not a good practise. I plan to improve it when I refactor the `Interpreter` class (ZEPPELIN-2685).
Besides that, I also add `MiniHadoopCluster` & `MiniZeppelin` which help for the integration test of yarn-client & yarn-cluster mode, otherwise I have to manually verify yarn-client & yarn-cluster mode which would easily cause regression issue in future.
To be noticed:
* SPARK_HOME must be specified for yarn-cluster mode
* HADOOP_CONF_DIR must be specified for yarn-cluster mode
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
https://github.com/zjffdu/zeppelin/tree/ZEPPELIN-2898
### How should this be tested?
System test is added in `SparkInterpreterIT`.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2577 from zjffdu/ZEPPELIN-2898 and squashes the following commits:
9da7c4b [Jeff Zhang] ZEPPELIN-2898. Support Yarn-Cluster for Spark Interpreter
### What is this PR for?
Straightforward change for allowing pass non spark.* properties to spark app driver. See discussion here http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Setting-spark-config-properties-in-Zeppelin-0-7-2-td6143.html
### What type of PR is it?
[Bug Fix ]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2893
### How should this be tested?
Unit test is added
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2561 from zjffdu/ZEPPELIN-2893 and squashes the following commits:
1342527 [Jeff Zhang] ZEPPELIN-2893. Pass non spark.* of spark interpreter properties to spark app driver
### What is this PR for?
With ZEPPELIN-2841, it had changed the default working directory of Shell Interpreter from the relative path where Zeppelin is running to user-home. This is to make the configuration optional.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2903
### How should this be tested?
by default when the user runs shell interpreter and executes `pwd` will the path where Zeppelin server is running, but when `shell.working.directory.user.home` is set to true in Zeppelin's interpreter setting, it will point to the user's home directory by which the interpreter is running.
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2566 from prabhjyotsingh/ZEPPELIN-2903 and squashes the following commits:
9934df52a [Prabhjyot Singh] add doc for shell.working.directory.user.home
9164ed2c0 [Prabhjyot Singh] Make setting of working directory to user-home optional for shell interpreter
### What is this PR for?
This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well.
For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter.
Main features:
* IPython interpreter support
** All the ipython features are available, including visualization, ipython magics.
* ZeppelinContext support
* Streaming output support
* Support Ipython in PySpark
Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here.
In unit test, I only verify the following 3 popular visualization library. could add more later.
* matplotlib
* bokeh
* ggplot
### What type of PR is it?
[Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2753
### How should this be tested?
Unit test is added.
### Screenshots (if appropriate)
Verify bokeh in IPython Interpreter
Verify matplotlib
Verify ZeppelinContext
Verify Streaming
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits:
e869f31 [Jeff Zhang] address comments
b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
### What is this PR for?
upgrade Beam interpreter to use the latest version of Apache Beam.
### What type of PR is it?
[Improvement]
### Todos
*
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2865
### How should this be tested?
* Start the Zeppelin server
* The prefix of interpreter is %beam and then write your code with required imports and the runner
Refer to `docs/interpreter/beam.md` for an example;
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes, updated `docs/interpreter/beam.md` and `README.md`
Author: mingmxu <mingmxu@ebay.com>
Closes#2541 from XuMingmin/ZEPPELIN-2865 and squashes the following commits:
520f0fd7 [mingmxu] restore the notice message of scala-2.10
93b3e24d [mingmxu] upgrade to Apache Beam 2.0.0
### What is this PR for?
Complementing Zeppelin-1501 (support for Cassandra with SSL in interpreter) with documentation updates.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2835
* [Zeppelin-2835]
### How should this be tested?
Check that documentation for cassandra interpreter includes cassandra.ssl options.
### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
No
Author: Alex Lourie <alex@instaclustr.com>
Author: Alex Lourie <djay.il@gmail.com>
Closes#2517 from alourie/Zeppelin-2835-addSSL-doco and squashes the following commits:
16719766 [Alex Lourie] Updates
9058ee45 [Alex Lourie] Fix the highlight
02625c94 [Alex Lourie] Adding documentation for Cassandra interpreter SSL
I was unaware of this really nice feature, which led me to writing my own "zeppelin_show" function.
### What is this PR for?
(tiny) improvement of the docs
Author: TwUxTLi51Nus <TwUxTLi51Nus@users.noreply.github.com>
Closes#2488 from TwUxTLi51Nus/patch-1 and squashes the following commits:
7ab6f41 [TwUxTLi51Nus] Add z.show() feature to docs
### What is this PR for?
Since it is quite useful to expose the application info for user to monitor and debug, so here propose to enable "zeppelin.livy.displayAppInfo" to true as default.
### What type of PR is it?
Improvement
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2716
### How should this be tested?
Manual verification.
Author: jerryshao <sshao@hortonworks.com>
Closes#2459 from jerryshao/ZEPPELIN-2716 and squashes the following commits:
c9506ab [jerryshao] Address UT failure
32d3f72 [jerryshao] Change the default value of zeppelin.livy.displayAppInfo to true
### What is this PR for?
fix: broken image URLs in 0.8.0-SNAPSHOT doc
using the path `/asset` (the absolute path) for image URLs is actually invalid. That's because each version has its own image directory. So they should use the relative path. `{{BASE_PATH}}`
```
➜ asf-zeppelin tree site | grep asset
├── assets # root asset, we shouldn't use it in versioned doc.
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ ├── assets
```
### What type of PR is it?
[Bug Fix]
### Todos
DONE
### What is the Jira issue?
[ZEPPELIN-2707](https://issues.apache.org/jira/browse/ZEPPELIN-2707)
### How should this be tested?
1. cd `docs/`
2. build: `bundle exec jekyll build --safe`
3. check whether links in `_site` include `/docs/0.8.0-SNAPSHOT` as prefix or not
### Screenshots (if appropriate)
#### Current
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/interpreter/overview.html
#### After
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#2450 from 1ambda/ZEPPELIN-2707/should-use-its-own-asset-directory and squashes the following commits:
fb70214a [1ambda] fix: Use its own asset dir
### What is this PR for?
Pig 0.17.0 has just been released. This PR is to upgrade pig to 0.17.0 and support spark engine which is a big milestone of pig 0.17.0
Main Changes:
* Upgrade pig to 0.17.0
* Remove some code using java reflection in `PigUtils.java`, as pig 0.17.0 has some improvement and expose new apis which could be used pig interpreter.
* Support spark engine
### What type of PR is it?
[Improvement | Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://github.com/zjffdu/zeppelin/tree/ZEPPELIN-2615
### How should this be tested?
Unit test is added and also manually test spark yarn-client mode in pig tutorial note.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2431 from zjffdu/ZEPPELIN-2615 and squashes the following commits:
d4e9a6d [Jeff Zhang] Address comments
4b4e3db [Jeff Zhang] ZEPPELIN-2615. Upgrade pig to 0.17.0 to support spark engine
### What is this PR for?
Kerberos ticket and renew lifetime are set to 1 hour. On accessing secure Hadoop from shell interpreter, it does kinit and returns result successfully but after 1 hour, the ticket gets expired and Hadoop list fails with below exception.
```
%sh
hadoop fs -ls /
17/01/05 09:29:45 WARN ipc.Client: Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "zeppelin1.hwxblr.com/10.0.1.57"; destination host is: "zeppelin1.hwxblr.com":8020;
ExitValue: 1
```
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
* [ZEPPELIN-1907](https://issues.apache.org/jira/browse/ZEPPELIN-1907)
### How should this be tested?
On a Kerberos enabled cluster, run this paragraph
```
%sh
hdfs dfs -ls /user/zeppelin/
```
Wait for key-tab to expire (or run `kdestroy`), and re-run the same paragraph.
### Screenshots (if appropriate)
Before:
<img width="1438" alt="screen shot 2017-06-13 at 3 44 30 pm" src="https://user-images.githubusercontent.com/674497/27078184-511ed810-5050-11e7-8afa-90247f33047a.png">
After:
<img width="1438" alt="screen shot 2017-06-13 at 3 44 04 pm" src="https://user-images.githubusercontent.com/674497/27078183-5109d690-5050-11e7-82e4-d79a5e98295f.png">
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2407 from prabhjyotsingh/ZEPPELIN-1907 and squashes the following commits:
ffd5f11b2 [Prabhjyot Singh] add bash after `
ecc1a7ce0 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-1907
9243c6ab9 [Prabhjyot Singh] replace `###` with `##`
443c407d3 [Prabhjyot Singh] add space before time(s)
adf23743b [Prabhjyot Singh] update documentation.
289b7d346 [Prabhjyot Singh] reset kinitFailCount on successful renew.
96bfdfe97 [Prabhjyot Singh] log more error
72b32ae25 [Prabhjyot Singh] add java doc
df6645a64 [Prabhjyot Singh] add KerberosInterpreter and move kinit loginc there.
856c8716e [Prabhjyot Singh] renew token periodically
ee741e483 [Prabhjyot Singh] @zjffdu review comments
7c539ef2e [Prabhjyot Singh] add null check
ab823d3ee [Prabhjyot Singh] relogin using keytab, and append message for the same
### What is this PR for?
Improving documentation page. Please check *TODO* and *Screenshots* sections for detail.
The motivation is described in [the JIRA ticket](https://issues.apache.org/jira/browse/ZEPPELIN-2583) and discussion is ongoing on the mailing list.
### What type of PR is it?
[Improvement | Documentation]
### Todos
* [x] - improved the navbar style
* [x] - improved the main page
* [x] - re-organized content structure
* [x] - added tutorial pages: `spark_with_zeppelin.md`, `python_with_zeppelin.md`, `sql_with_zeppelin.md` for overview
* [x] - added `multi_user_support.md` page to provide overview
* [x] - added the empty `interpreter_binding_mode` page. This will be handed in the different issue: [ZEPPELIN-2582](https://issues.apache.org/jira/browse/ZEPPELIN-2582)
* [x] - added the empty `trouble_shooting` page. This can be filled in the following PRs.
* [x] - added the empty `useful_developer_tools` page. This can be filled in the following PRs.
### What is the Jira issue?
[ZEPPELIN-2596](https://issues.apache.org/jira/browse/ZEPPELIN-2596)
### How should this be tested?
1. checkout
2. `cd docs`
3. `bundle install` (make sure that you have ruby 2.1.0+ and bundle)
4. `bundle exec jekyll serve --watch`
5. open `localhost:4000`
### Screenshots (if appropriate)
#### better navbar: before
#### better navbar: after
#### improved main page: before
#### improved main page: after
#### organized content structure: before
#### organized content structure: after
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - related with docs
Author: 1ambda <1amb4a@gmail.com>
Closes#2371 from 1ambda/updating-version-doc and squashes the following commits:
eb02fa967 [1ambda] fix: navbar focus color applies after folding
026379ed6 [1ambda] fix: Remove docs/.listen_test
a7dd4737b [1ambda] fix: sora's comment 1.2
18c5058f7 [1ambda] fix: resolve description in python_with_zeppelin.md
d3ad67c73 [1ambda] fix: sora's comment 4
d133dbbcc [1ambda] fix: resolve sora's comment 3
513c6ff2c [1ambda] fix: resolve sora's comment 1.1
4c2946928 [1ambda] fix: resovle sora's comment 2
1c3946ac6 [1ambda] fix: sora's comment 1
4d6e4267f [1ambda] fix: Resolve sola's comment 3
d0524cafe [1ambda] fix: Set less shadow for nav
5f1f998ba [1ambda] docs: Add useful_develop_tools.md
9dfd62c74 [1ambda] fix: Typo in installation.md
30f7d7e06 [1ambda] fix: Typo in helium ctrl
d6877e792 [1ambda] docs: Add python_with_zeppelin.md
7027e96c0 [1ambda] docs: Improve python conda, docker doc style
e55b50a9d [1ambda] fix: Invalid URLs
75ddeeaff [1ambda] docs: replace URIs in interpreter
5b43993a4 [1ambda] docs: Add sql_with_zeppelin
053794e84 [1ambda] docs: Add spark_with_zeppelin.md
d4d88b9c7 [1ambda] docs: Improve proxy doc
b46cdd126 [1ambda] docs: Add empty interpreter_binding_mode.md
06fcb239e [1ambda] docs: Add empty personalized_mode.md
4991cf0a7 [1ambda] docs: Update upgrading.md
53142b7a0 [1ambda] fix: Simplify install.md
8a5c1e721 [1ambda] docs: Add multi_user_support.md
34095775e [1ambda] fix: Increase font size to 15px
a03b04b33 [1ambda] fix: Remove sample text from trouble_shooting.md
199842590 [1ambda] fix: Remove docker doc link
66a2a7d26 [1ambda] docs: Improve impersonation page
0a6e3fc1d [1ambda] docs: Improve install doc
ccd999ed5 [1ambda] docs: Improve helium doc
f8d742d08 [1ambda] fix: an invalid link in navbar
b7aa5f884 [1ambda] fix: URLs in development
61a175d94 [1ambda] docs: Update install.md
4c56de5c4 [1ambda] fix: URLs in setup
0b1d63513 [1ambda] fix: URLs in quickstart
28970a4fe [1ambda] feat: Add docs/usage
735946bca [1ambda] feat: rename /quickstart
b351cf237 [1ambda] fix: Add missing links
b70770b4f [1ambda] feat: Change URLs in nav, index
94e80aef6 [1ambda] fix: doens't display navbar version in small
6e0cab110 [1ambda] feat: Update doc section names
b9ce256ff [1ambda] feat: Hide version in navbar when md
f8bab52be [1ambda] fix: Better image display in index.md
eeb37d5b5 [1ambda] fix: Add RL padding for mobile browser
ceb60b5ee [1ambda] feat: Style collapsed nav for mobile browser
4ebafb4b6 [1ambda] commit
### What is this PR for?
There are some problems if meta is large (few schemas, each schema contains 500+ tables etc.).
Problems:
1. loading is very long
2. each update takes one connection if updates are long, the situation may arise that the entire pool will be busy
3. no cache
This PR solves these problems. Added cache and access by full path (schema.table, schema.table.column) + protection for release the connections
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2538
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Tinkoff DWH <tinkoff.dwh@gmail.com>
Closes#2343 from tinkoff-dwh/ZEPPELIN-2538 and squashes the following commits:
0991c6ab [Tinkoff DWH] [ZEPPELIN-2538] small improvement
e770d261 [Tinkoff DWH] [ZEPPELIN-2538] update description
a5788743 [Tinkoff DWH] [ZEPPELIN-2538] protection long download
f999488b [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2538
f26ab5da [Tinkoff DWH] [ZEPPELIN-2538] fix tests
d600fa16 [Tinkoff DWH] [ZEPPELIN-2538] rewrite sql completer to work with large data
### What is this PR for?
Fix JDBC doc after #2229.
### What type of PR is it?
[Bug Fix]
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2314 from prabhjyotsingh/minor/jdbc-doc-zeppelin-2367 and squashes the following commits:
e54a3a2b7 [Prabhjyot Singh] @AhyoungRyu review comments
0f396ac64 [Prabhjyot Singh] fix doc for zeppelin-2367
### What is this PR for?
Hive JDBC proxy user option should be available generically.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* [Zeppelin 2367](https://issues.apache.org/jira/browse/ZEPPELIN-2367)
### How should this be tested?
Enable Shiro authentication and set `zeppelin.jdbc.auth.type` as `SIMPLE` in the interpreter setting, and observe the connection string for the Hive.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need an update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? Yes
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2229 from prabhjyotsingh/ZEPPELIN-2367 and squashes the following commits:
84b5e55b2 [Prabhjyot Singh] add logger.warn for hive and impersonation
45c90a8e2 [Prabhjyot Singh] improve doc
9fee9d2a9 [Prabhjyot Singh] replace hive with generic method
a348e969a [Prabhjyot Singh] revert "zeppelin.jdbc.auth.kerberos.proxy.enable" behaviour
e2bdbb2ad [Prabhjyot Singh] include e as inner exception
c180f5ce2 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367
1802b453f [Prabhjyot Singh] remove hive string from logger
513987a28 [Prabhjyot Singh] apply genric logic to appendProxyUserToURL
3fa2b1e98 [Prabhjyot Singh] change name to appendProxyUserToURL
a75167415 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367
4c382eefa [Prabhjyot Singh] log user details as well
d51e770b2 [Prabhjyot Singh] add doc in jdbc.md
01b18b9d2 [Prabhjyot Singh] add doc (reverted from commit ee8a6b524c)
40489c89d [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367
ee8a6b524 [Prabhjyot Singh] add doc
8999d93ae [Prabhjyot Singh] ZEPPELIN-2367: Hive JDBC proxy user option should be avail even without kerberos
### What is this PR for?
General solution to execute precode. Refactoring jdbc precode using general solution. Task contains to subtasks: executeAfterOpen, executeBeforeClose. executeBeforeClose not done because we need the context so there is a solution only for executeAfterOpen.
### What type of PR is it?
Feature | Refactoring
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2216
### How should this be tested?
1. Add parameter zeppelin.PySparkInterpreter.precode `someVar='text'`
2. Execute
```
%pyspark
print(someVar)
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Tinkoff DWH <tinkoff.dwh@gmail.com>
Closes#2221 from tinkoff-dwh/ZEPPELIN-2216 and squashes the following commits:
1e3f3f7 [Tinkoff DWH] [ZEPPELIN-2216] fix path
e4cf72f [Tinkoff DWH] [ZEPPELIN-2216] added tests
5a482a0 [Tinkoff DWH] [ZEPPELIN-2216] fix tests
3977722 [Tinkoff DWH] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2216
c0436a2 [Tinkoff DWH] [ZEPPELIN-2216] general solution to precode. refactoring jdbc precode
### What is this PR for?
Target: Create ability to call http services with custom keystores in a groovy way.
The following should work:
```groovy
//connect to host xxx.yyy with special keystore
HTTP.get(
url: "https://xxx.yyy/zzz",
ssl: " HTTP.getKeystoreSSLContext('./xxx_yyy_keystore.jks', 'testpass') "
)
//take context initialization code from groovy interpreter properties
HTTP.get(
url: "https://xxx.yyy/zzz",
ssl: g.SSL_CONTEXT_FROM_GROOVY_INTERPRET_PROPERTIES
)
//connect to host xxx.yyy with trust all (do not check trust certificates - dev mode only)
HTTP.get(
url: "https://xxx.yyy/zzz",
ssl: " HTTP.getNaiveSSLContext() "
)
//
HTTP.get(
url: "https://xxx.yyy/zzz",
ssl: " MyCustomSSLBuilder.build() "
)
```
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
[ZEPPELIN-2443]
### How should this be tested?
follow the samples above or in documentation
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? YES
Author: dlukyanov <dlukyanov@ukr.net>
Closes#2287 from dlukyanov/master and squashes the following commits:
4baa22e [dlukyanov] ZEPPELIN-2443
…e to disable proxy behavior
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
### What type of PR is it?
[Improvement]
* add a configuration item zeppelin.jdbc.auth.kerberos.proxy.enable to disable kerberos behaviour
as we know， in current version of zeppelin， if we have kerberos auth configured，and using zeppelin with a user login，the jdbc interpreter will do a proxy behavior with the login user automatically，but in many cases， we do not want do this，and we do not want bind the zeppelin user system with kerberos auth system. I think it's make senses to add a configuration item to disable this behavior.
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
[ZEPPELIN-2353] (https://issues.apache.org/jira/browse/ZEPPELIN-2353)
### How should this be tested?
Outline the steps to test the PR here.
when we have kerberos auth configured, and using zeppelin with a login user
add configuration:
"zeppelin.jdbc.auth.kerberos.proxy.enable=false"
for jdbc iterpreter configuration, the jdbc iterpreter will not do the proxy behavior with the login user
### Screenshots (if appropriate)
<img width="1405" alt="2017-04-04 9 12 03" src="https://cloud.githubusercontent.com/assets/869480/24658501/7739e680-197c-11e7-90ab-c1938e31efc7.png">
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: LeiWang <wanglei6744@163.com>
Author: lei wang <lei1989@outlook.com>
Closes#2222 from wary/main-master and squashes the following commits:
1f26bc3 [LeiWang] Update jdbc.md
ba34a41 [LeiWang] Update JDBCInterpreter.java
444b6ab [LeiWang] Update JDBCInterpreter.java
ea11ce4 [LeiWang] Update jdbc.md
ee9cd18 [LeiWang] Update jdbc.md
e04fb38 [LeiWang] Update jdbc.md
75e4b9f [LeiWang] Update jdbc.md
b850a42 [LeiWang] Update JDBCInterpreter.java
067696a [lei wang] add configure zeppelin.jdbc.auth.kerberos.proxy for kerberos auth type to disable proxy behavior
### What is this PR for?
Groovy Interpreter
### What type of PR is it?
Feature
### Todos
* [ Tests ] - Task
* [ Documentation ] - Task
### What is the Jira issue?
[ZEPPELIN-2176]
### How should this be tested?
Follow the groovy interpreter documentation samples
### Questions:
* Does the licenses files need update? YES
* Is there breaking changes for older versions? NO
* Does this needs documentation? YES
Author: dlukyanov <dlukyanov@ukr.net>
Author: dm <dm>
Closes#2135 from dlukyanov/master and squashes the following commits:
faf213f [dlukyanov] ZEPPELIN-2176 comments from @AhyoungRyu - remove @author - remove commented code
89c3ed5 [dlukyanov] retry
ca65947 [dlukyanov] deprecated
3dd53e2 [dlukyanov] ZEPPELIN-2176 comments from @AhyoungRyu - Zeppelin follows Google Java code - interpreter alphabetical order in _navigation.html - direct link to MarkupBuilder in groovy help
fe08159 [dlukyanov] retry
ca8bea6 [dlukyanov] Update groovy.md
a5b37a1 [dlukyanov] ZEPPELIN-2176 https://github.com/apache/zeppelin/pull/2135#issuecomment-289308850 - Inside of docs directory, groovy.md will need some header to be compiled with Jekyll - Menu in docs also need link to groovy - .travis.yml we need add !groovy
4abf649 [dm] Merge branch 'master' of https://github.com/apache/zeppelin41a1702 [dlukyanov] ZEPPELIN-2176 https://github.com/apache/zeppelin/pull/2135#issuecomment-288829494 - implement shared script variables - move docs - implement run methods
dd388b3 [dlukyanov] retry
b34b42a [dlukyanov] retry
0d7732a [dlukyanov] retry
2646fa8 [dlukyanov] ZEPPELIN-2176 groovy interpreter, fix unchecked, add to configs, move HTTP.groovy to resources to simplify build, add default z-properties
5fa26e0 [dlukyanov] ZEPPELIN-2176 groovy interpreter, fix unchecked, add to configs, move HTTP.groovy to resources to simplify build, add default z-properties
aa427cd [dlukyanov] retry
addf167 [dlukyanov] retry
db4c35b [dlukyanov] Update README.md
fa779ea [dlukyanov] groovy interpreter
### What is this PR for?
For now, every time when I want to try new spark version, I have to change file `SparkVersion.java` and rebuild it. It is not so convenient, so I'd like to add property `zeppelin.spark. enableSupportedVersionCheck` for spark interpreter. So that I can try new spark version by setting this property as false, of course it is only for zeppelin developer.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2324
### How should this be tested?
Verify it in spark master
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2197 from zjffdu/ZEPPELIN-2324 and squashes the following commits:
f821839 [Jeff Zhang] address comments
4193ee3 [Jeff Zhang] ZEPPELIN-2324. Add property zeppelin.spark.unSupportedVersionCheck for trying new spark version
… to display results
### What is this PR for?
Livy SQL interpreter truncate result strings of size greater than 20. In some cases, we like to see the full string. We are adding a interpreter property **zeppelin.livy.spark.sql.field.truncate** to control whether to truncate strings or not. By default, **zeppelin.livy.spark.sql.field.truncate** is set to **true**.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1965
### How should this be tested?
Set zeppelin.livy.spark.sql.field.truncate to true or false
Run a SQL query which produces string values of length greater than 20.
Depending on the value of zeppelin.livy.spark.sql.field.truncate, the strings will either get truncated or not.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Benoy Antony <benoy@apache.org>
Closes#2201 from benoyantony/master and squashes the following commits:
bb006c0 [Benoy Antony] changed field name and description
9eae68b [Benoy Antony] added a null check to avoid testcase failures, another nullcheck for backward compatibility and added two new testcases
ab1ead2 [Benoy Antony] documented zeppelin.livy.spark.sql.truncate
b6252be [Benoy Antony] [ZEPPELIN-1965] Livy SQL Interpreter: Should use df.show(1000, false) to display results
### What is this PR for?
Livy server support https, but the currently livy interpreter doesn't support it. This PR is for for the supporting to connect with livy through https
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2261
### How should this be tested?
Tested manually on livy server with ssl enabled.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2139 from zjffdu/ZEPPELIN-2261 and squashes the following commits:
52fc204 [Jeff Zhang] address comment
53230c3 [Jeff Zhang] [ZEPPELIN-2261]. Support to connect with livy through https
### What is this PR for?
Separate precode by prefix. Added the ability to set different precode for different data sources
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2245
### How should this be tested?
1. Set properties
```
default.password 1
default.precode set search_path='test_path'
default.url jdbc:postgresql://localhost:5432/
default.user postgres
mysql.driver com.mysql.jdbc.Driver
mysql.password 1
mysql.precode set v=12
mysql.url jdbc:mysql://localhost:3306/
mysql.user root
```
2. Run
`show search_path`
3. Run
```
%jdbc(mysql)
select v
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Tinkoff DWH <tinkoff.dwh@gmail.com>
Closes#2121 from tinkoff-dwh/ZEPPELIN-2245 and squashes the following commits:
970c064 [Tinkoff DWH] [ZEPPELIN-2245] editing documentation
a136a0e [Tinkoff DWH] [ZEPPELIN-2245] documentation for usage of precode
f896ea8 [Tinkoff DWH] [ZEPPELIN-2245] separate precode into JDBCInterpreter
### What is this PR for?
Added new property "hive.proxy.user" to disable hive impersonation (on some clusters, this option is disabled) in order to make Hive Interpreter even without this
### What type of PR is it?
Feature
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1968
### How should this be tested?
Set "hive.proxy.user" to true in the jdbc interpreter setttings, and you should see "Using hive proxy user" in the jdbc logs.
If "hive.proxy.user" has another value, this is not mentionned in the logs
You can also test with the appropriate hive configuration, but this could take longer :)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Paolo Genissel <paolo.genissel-monsallier@1000mercis.com>
Closes#2051 from gfalcone/hive_impersonation and squashes the following commits:
a39d11c [Paolo Genissel] Fixed last NPE
1f7f685 [Paolo Genissel] Fixed NPE when getting hive.proxy.user property
433eefb [Paolo Genissel] Added documentation for feature
d6f0c62 [Paolo Genissel] Added property to disable hive user impersonation
### What is this PR for?
Adds property "precode". Value of property contains SQL which executes while opening connection.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1988
### How should this be tested?
1) Set property zeppelin.interpreter.precode =` set search_path='test, public' `
2) Execute `%jdbc show search_path`
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Tinkoff DWH <tinkoff.dwh@gmail.com>
Closes#2078 from tinkoff-dwh/ZEPPELIN-1988 and squashes the following commits:
cd46cce [Tinkoff DWH] [ZEPPELIN-1988] trim precode
42ffcb7 [Tinkoff DWH] [ZEPPELIN-1988] fix condition
7636b3f [Tinkoff DWH] Merge remote-tracking branch 'origin/master' into ZEPPELIN-1988
66d6ae4 [Tinkoff DWH] [ZEPPELIN-1988] fixes of review items
9d37bc4 [Tinkoff DWH] [ZEPPELIN-1988] fix
ba3477a [Tinkoff DWH] [ZEPPELIN-1988] add property "precode" to JDBCInterpreter
### What is this PR for?
`zeppelin.pyspark.python` is zeppelin configuration for the python exec on driver side, it won't affect executor side. It would be better to use `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` which is what spark use officially. So that user can define their own python exec in interpreter setting for different version of python rather than defining them `zeppelin-env.sh` which is shared globally.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2195
### How should this be tested?
Tested it manually.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2079 from zjffdu/ZEPPELIN-2195 and squashes the following commits:
fa71cb2 [Jeff Zhang] address comments
fd89a1e [Jeff Zhang] ZEPPELIN-2195. Use PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON over zeppelin.pyspark.python
### What is this PR for?
Add pig wiki page pig doc
### What type of PR is it?
[Documentation |]
### Todos
* [ ] - Task
### What is the Jira issue?
No jira created
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2004 from zjffdu/pig_doc and squashes the following commits:
e5a564a [Jeff Zhang] rename zeppelin to Zeppelin and pig to Pig
65458ff [Jeff Zhang] address comments and minor update on pig tutorial
c6cb5ff [Jeff Zhang] update pig tutorial
b8542de [Jeff Zhang] [MINOR] add pig wiki page to pig doc
### What is this PR for?
We don't have to maintain PostgresqlInterpreter because JDBCInterpreter covers all functions of PostgresqlInterpreter. It reduces maintenance costs.
### What type of PR is it?
[Feature]
### Todos
* [x] - Remove files and lines related to PostgresqlInterpreter
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2003
### How should this be tested?
N/A
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? Yes
* Does this needs documentation? Yes, but already documented in jdbc docs
Author: Jongyoul Lee <jongyoul@gmail.com>
Closes#1945 from jongyoul/ZEPPELIN-2003 and squashes the following commits:
5dc58f2 [Jongyoul Lee] Fixed doc
ec31f78 [Jongyoul Lee] Change docs to make a link to jdbc.md
bfd70ef [Jongyoul Lee] Removed Postgresql from docs and configurations
98a106e [Jongyoul Lee] Removed whole packages about Postgresql
### What is this PR for?
Add HTTP client to elasticsearch interpreter.
### What type of PR is it?
Feature
### Todos
* [X] - Source code
* [X] - Tests
* [X] - License
* [X] - Docs
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1821
### How should this be tested?
* Start an Elasticsearch node
* Configure the elasticsearch interpreter to use http
* Create queries in a note using elasticsearch
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Bruno Bonnin <bbonnin@gmail.com>
Author: Bruno Bonnin <bruno.bonnin@myscript.com>
Closes#1902 from bbonnin/master and squashes the following commits:
f5a539e [Bruno Bonnin] Remove commented code lines
86153a8 [Bruno Bonnin] Merge remote-tracking branch 'upstream/master'
2e1bbbd [Bruno Bonnin] Merge remote-tracking branch 'upstream/master'
19e888e [Bruno Bonnin] Remove bad code in test
523d155 [Bruno Bonnin] Replace Java 8 methods
6bcf369 [Bruno Bonnin] Fix issue with id containing special chars (/, #)
4e9812e [Bruno Bonnin] Merge elasticsearch/pom.xml
5a96ae0 [Bruno Bonnin] Merge branch 'master' into master
e2365fb [Bruno Bonnin] Update elasticsearch/pom.xml
28b9805 [Bruno Bonnin] Update img
549db39 [Bruno Bonnin] Add HTTP client to elasticsearch interpreter
f4c5ac3 [Bruno Bonnin] HTTP-based Elasticsearch client
### What is this PR for?
Refactor the livy interpreter to use multiple `InterpreterResult` for displaying appInfo.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1852
### How should this be tested?
Add integration test and also test it manually
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#1796 from zjffdu/ZEPPELIN-1852 and squashes the following commits:
724f22e [Jeff Zhang] update doc
e1c2eb9 [Jeff Zhang] ZEPPELIN-1852. Use multiple InterpreterResult for displaying appInfo
### What is this PR for?
Livy 0.3 support cancel operation, this PR is to support cancel in livy interpreter. First we would check the livy version, then based on the livy version, we would call the livy rest api to cancel the statement.
### What type of PR is it?
Improvement | Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1432
### How should this be tested?
Tested manually, because cancel is only avaible in livy 0.3 which is not released yet.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#1859 from zjffdu/ZEPPELIN-1432 and squashes the following commits:
83eaf83 [Jeff Zhang] minor update
200ca71 [Jeff Zhang] address comments
1cbeb26 [Jeff Zhang] add zeppelin.livy.pull_status.interval.millis
070fea0 [Jeff Zhang] ZEPPELIN-1432. Support cancellation of paragraph execution
### What is this PR for?
Add support for jceks stored password instead of using password in clear text. This security enhancement prevents any user from reading clear passwords from interpreter json using shell/py/spark etc. #1315 is the parent PR which fixed similar a issue in Active Directory (shiro.ini config).
### What type of PR is it?
Improvement
### Todos
### What is the Jira issue?
ZEPPELIN-1935
### How should this be tested?
Create a keystore file using the hadoop credential commandline, for this the hadoop commons should be in the classpath.
`hadoop credential create jdbc.password -provider jceks://file/user/zeppelin/conf/zeppelin.jceks`
Use the jceks file and the key to configure jdbc interpreter. Example interpreter setting:
#### Following are the supported settings for backward compatibility
- Settings with username and jceks
```
default.driver org.postgresql.Driver
default.jceks.credentialKey jdbc.password
default.jceks.file jceks://file/tmp/zeppelin.jceks
default.url jdbc:postgresql://rkamath-local-1:5432/
default.user rk-user
```
- Settings with user name and clear text password
```
default.driver org.postgresql.Driver
default.url jdbc:postgresql://rkamath-local-1:5432/
default.user rk-user
default.password password1
```
### Screenshots (if appropriate)
<img width="1392" alt="screen shot 2017-01-10 at 7 02 12 pm" src="https://cloud.githubusercontent.com/assets/2031306/21808016/5e602982-d767-11e6-88f9-3d15b9a7f0b8.png">
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Renjith Kamath <renjith.kamath@gmail.com>
Closes#1881 from r-kamath/ZEPPELIN-1935 and squashes the following commits:
5cc4db1 [Renjith Kamath] ZEPPELIN-1935 revert wildcard imports
dfabe3a [Renjith Kamath] Merge branch 'master' of https://github.com/apache/zeppelin into ZEPPELIN-1935
e62088a [Renjith Kamath] ZEPPELIN-1935 fix log message
d41d56c [Renjith Kamath] ZEPPELIN-1935 Add jceks stored password support for jdbc interpreter
### What is this PR for?
Two improvements for pig interpreter.
* Set job name via paragraph title if it exists, otherwise use the last line of pig script
* Allow to set any pig property in interpreter setting
### What type of PR is it?
[ Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1933
### How should this be tested?
Unit tested and manually tested.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#1885 from zjffdu/ZEPPELIN-1933 and squashes the following commits:
d2e1cd4 [Jeff Zhang] address comments
9cee380 [Jeff Zhang] ZEPPELIN-1933. Set pig job name and allow to set pig property in pig interpreter setting
### What is this PR for?
* Minor update for pig interpreter
* Add one sample pig tutorial note which do the same thing as the spark tutorial note.
### What type of PR is it?
[Improvement | Documentation ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1867
### How should this be tested?
Tested manually
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#1830 from zjffdu/ZEPPELIN-1867 and squashes the following commits:
1c0d819 [Jeff Zhang] rename note name
50198a1 [Jeff Zhang] add more description of tutorial note
88385f2 [Jeff Zhang] Add pig tutorial note
25216f8 [Jeff Zhang] ZEPPELIN-1867. Update document for pig interpreter and add one sample note
### What is this PR for?
This PR will add an example of batch processing with Flink to Zeppelin tutorial notebooks. There are no any Flink notebooks in the tutorial at the moment.
### What type of PR is it?
Improvement
### What is the Jira issue?
[ZEPPELIN-1787](https://issues.apache.org/jira/browse/ZEPPELIN-1787)
### How should this be tested?
You should open `Using Flink for batch processing` notebook from the `Zeppelin Tutorial` folder and run all paragraphs one by one
### Questions:
* Does the licenses files need update? - **no**
* Is there breaking changes for older versions? - **no**
* Does this needs documentation? - **no**
Author: Alexander Shoshin <Alexander_Shoshin@epam.com>
Closes#1758 from AlexanderShoshin/ZEPPELIN-1787 and squashes the following commits:
83cbffb [Alexander Shoshin] remove localhost url
5255e17 [Alexander Shoshin] Merge branch 'master' into ZEPPELIN-1787
0b9df56 [Alexander Shoshin] add a link for this notebook to Zeppelin documentation
593c47d [Alexander Shoshin] convert notebook to 0.7.0 format
9013620 [Alexander Shoshin] convert notebook to 0.6.2 format
fe2a39e [Alexander Shoshin] add download instruction, change "wget" to "curl"
f64b60a [Alexander Shoshin] [ZEPPELIN-1787] Add an example of Flink Notebook
### What is this PR for?
Add `README` file for developer, besides update the `livy.md` for the recent changes on livy interpreter.
### What type of PR is it?
[Documentation]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1853
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#1798 from zjffdu/ZEPPELIN-1853 and squashes the following commits:
940ebae [Jeff Zhang] remove upgrade
7233a4d [Jeff Zhang] address comments
72f40c7 [Jeff Zhang] ZEPPELIN-1853. Add README for livy interpreter
### What is this PR for?
Update scio doc - fix type $ should be %
### What type of PR is it?
Documentation
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1800
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Rafal Wojdyla <rav@spotify.com>
Closes#1754 from ravwojdyla/fix_1800 and squashes the following commits:
d4287a9 [Rafal Wojdyla] Update scio doc - fix type $ should be %
### What is this PR for?
Currently, the parameter "project" is defined with "kylin.query.project" in properties. It's not convenience when query Kylin among different projects. May I propose introducing %kylin(project_name) at the interpreter runtime? If not set, the default project will work, otherwise, will use explicit project_name for the query request.
### What type of PR is it?
Improvement
### Todos
DONE
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1512
### How should this be tested?
%kylin select ...
%kylin(new_project) select ....
### Screenshots (if appropriate)
### Questions:
- Does the licenses files need update?
No.
- Is there breaking changes for older versions?
No.
- Does this needs documentation?
Will do later.
Author: Yiming Liu <liuyiming.vip@gmail.com>
Closes#1477 from yiming187/ZEPPELIN-1512 and squashes the following commits:
b58ee7f [Yiming Liu] [ZEPPELIN-1512] Support Kylin project name in interpreter runtime
### What is this PR for?
hive-site.xml is required to configure HiveContext in SparkInterpreter.
So it'll be helpful if document provide at least simple example that can help user get some idea.
### What type of PR is it?
Improvement
### Todos
* [x] - Add simple example
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1728
### How should this be tested?
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#1708 from Leemoonsoo/ZEPPELIN-1728 and squashes the following commits:
98860ba [Lee moon soo] an example configure hive-site.xml into the classpath
### What is this PR for?
ZeppelinContext can be used to exchange DataFrames but there are some nasty tricks and typecasts.
It's good to provide some examples.
### What type of PR is it?
Documentation
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Mohammad Amin Khashkhashi Moghaddam <amin.moghaddamV@gmail.com>
Closes#1677 from m30m/patch-3 and squashes the following commits:
a039d5c [Mohammad Amin Khashkhashi Moghaddam] Add doc for exchanging data frames
### What is this PR for?
Allow users to issue insert/update/upsert statements from Zeppelin notes
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
[ZEPPELIN-1645](https://issues.apache.org/jira/browse/ZEPPELIN-1645)
### How should this be tested?
For Phoenix Interpreter:
Paragraph 1:
%jdbc
UPSERT INTO CX_TEST (ACCT_NBR, HSE_ID) VALUES ('a', 'b')
Paragraph 2:
%jdbc
select count(*) from cx_test
Result: 1
### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
Yes - JDBC README updated
Author: Randy Gelhausen <rgelhau@gmail.com>
Closes#1617 from randerzander/master and squashes the following commits:
1cf1ce3 [Randy Gelhausen] Added check for connection autocommit status
### What is this PR for?
This PR is for the multi-tenant of JDBC Interpreter.
User can create a user/password for JDBC account at the [Credential page](http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/security/datasource_authorization.html).
The `Entity` of `Credential` is match with JDBC interpreter group name.
If the account for JDBC is not setted in the `Interpreter property` then use `Credential`'s.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1567
### How should this be tested?
Please refer to testMultiTenant() of JDBCInterpreterTest/
### Screenshots (if appropriate)
### Questions:
- Does the licenses files need update? no
- Is there breaking changes for older versions? no
- Does this needs documentation? no
Author: astroshim <hsshim@nflabs.com>
Closes#1539 from astroshim/jdbc-impersonation and squashes the following commits:
46fce31 [astroshim] add explanation of InterpreterGroup
7a92236 [astroshim] fix doc and remove persist value.
63f5ea7 [astroshim] Merge branch 'master' into jdbc-impersonation
267277a [astroshim] rebase
649ff6e [astroshim] rebase
872fb49 [astroshim] fix ScioInterpreterTestCase
4387a5b [astroshim] Merge branch 'master' into jdbc-impersonation
47c463f [astroshim] update doc and html
d4eb178 [astroshim] fix docs
59aa9ff [astroshim] Merge branch 'master' into jdbc-impersonation
bf61afd [astroshim] fix testcase
5c0f5d7 [astroshim] rebase
79ba25b [astroshim] Merge branch 'master' into jdbc-impersonation
1f9c2c0 [astroshim] clean redundant code
a2f5687 [astroshim] fix impersonation
9962181 [astroshim] fix InterpreterOutput of PySparkInterpreterTest case
b55aceb [astroshim] Merge branch 'master' into jdbc-impersonation
24a8226 [astroshim] fix doc
086dfda [astroshim] fix testcase
34fe0a6 [astroshim] fix code for more simple.
fee7086 [astroshim] fix build error.
a305eca [astroshim] Merge branch 'master' into jdbc-impersonation
df80741 [astroshim] documentation for credential.
df1b1dc [astroshim] rebase and entity name convention.
63d6a1c [astroshim] change thrift version to 0.9.2
6573c1c [astroshim] change variable name
f311f34 [astroshim] fix typo
722e333 [astroshim] change testcase name
9161937 [astroshim] clean code
3dafdf0 [astroshim] add testcase
373d5f1 [astroshim] pass replName to Interpreter and use credential info for jdbc auth.
