### What is this PR for?
Support encrypting passwords using a private key
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2952
### How should this be tested?
- Set the env variable `ZEPPELIN_CREDENTIALS_ENCRYPT_KEY=something`
- Save a few credentials
- Check that the `credentials.json` file is storing encrypted passwords
- Restart server using the same env variable for `ZEPPELIN_CREDENTIALS_ENCRYPT_KEY`
- The credentials should still be decryptable
### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
Yes
Author: Herval Freire <hfreire@twitter.com>
Closes#2599 from herval/encrypt-credentials and squashes the following commits:
e5857d8 [Herval Freire] missing license
1d4bc04 [Herval Freire] documentation....?
82ae8f4 [Herval Freire] added license
c3e0ead [Herval Freire] encrypt credentials.json with AES
### What is this PR for?
This is the first version for supporting yarn-cluster of `SparkInterpreter`. I just delegate all the function to `spark-submit` as yarn-cluster is natively supported by spark, we don't need to reinvent the wheel. But there's still improvement to be done in future, e.g. I put some spark specific logic in `InterpreterSetting` which is not a good practise. I plan to improve it when I refactor the `Interpreter` class (ZEPPELIN-2685).
Besides that, I also add `MiniHadoopCluster` & `MiniZeppelin` which help for the integration test of yarn-client & yarn-cluster mode, otherwise I have to manually verify yarn-client & yarn-cluster mode which would easily cause regression issue in future.
To be noticed:
* SPARK_HOME must be specified for yarn-cluster mode
* HADOOP_CONF_DIR must be specified for yarn-cluster mode
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
https://github.com/zjffdu/zeppelin/tree/ZEPPELIN-2898
### How should this be tested?
System test is added in `SparkInterpreterIT`.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2577 from zjffdu/ZEPPELIN-2898 and squashes the following commits:
9da7c4b [Jeff Zhang] ZEPPELIN-2898. Support Yarn-Cluster for Spark Interpreter
### What is this PR for?
This is a refactoring PR of ZEPPELIN-1515. Because hadoop's FileSystem API not only works with hdfs, but also other hadoop compatible filesystem. So in this PR I rename it to `FileSystemNotebookRepo`
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2933
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2588 from zjffdu/ZEPPELIN-2933 and squashes the following commits:
45d1e9b [Jeff Zhang] ZEPPELIN-2993. Code Refactoring of ZEPPELIN-1515 follow up
### What is this PR for?
Straightforward change for allowing pass non spark.* properties to spark app driver. See discussion here http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Setting-spark-config-properties-in-Zeppelin-0-7-2-td6143.html
### What type of PR is it?
[Bug Fix ]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2893
### How should this be tested?
Unit test is added
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2561 from zjffdu/ZEPPELIN-2893 and squashes the following commits:
1342527 [Jeff Zhang] ZEPPELIN-2893. Pass non spark.* of spark interpreter properties to spark app driver
### What is this PR for?
With ZEPPELIN-2841, it had changed the default working directory of Shell Interpreter from the relative path where Zeppelin is running to user-home. This is to make the configuration optional.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2903
### How should this be tested?
by default when the user runs shell interpreter and executes `pwd` will the path where Zeppelin server is running, but when `shell.working.directory.user.home` is set to true in Zeppelin's interpreter setting, it will point to the user's home directory by which the interpreter is running.
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2566 from prabhjyotsingh/ZEPPELIN-2903 and squashes the following commits:
9934df52a [Prabhjyot Singh] add doc for shell.working.directory.user.home
9164ed2c0 [Prabhjyot Singh] Make setting of working directory to user-home optional for shell interpreter
### What is this PR for?
The idea of this PR is to provide a new kind of user : Runner.
Basically, what it does is that it just removes write authorization and allow user to read and run note.
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
[ZEPPELIN-2848] https://issues.apache.org/jira/browse/ZEPPELIN-2848
### How should this be tested?
- Log in as admin
- Create new notebook and create a paragraph with the interpreter you want
- Assign runner right to user1
- Log in as user1
- Try to run the paragraph (should work)
- Try to modify the paragraph (should fail)
- Log in as user2
- Try to run the paragraph (should fail)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? Yes
* Does this needs documentation? No
Author: Paolo Genissel <paolo.genissel-monsallier@1000mercis.com>
Author: gfalcone <paologenissel@gmail.com>
Author: Paolo Genissel <paologenissel@gmail.com>
Closes#2526 from gfalcone/new_type_runner and squashes the following commits:
96bba66 [gfalcone] Fix typo on notebook_authorization.md
8ab4512 [gfalcone] Update notebook_authorization.md
22a1eb3 [Paolo Genissel] Fixed typo
d621792 [Paolo Genissel] Fix NotebookSecurityRestApiTest
a67af0f [Paolo Genissel] Fix test
5c43ca9 [Paolo Genissel] Added new type of user
### What is this PR for?
This is the first step for implement IPython Interpreter in Zeppelin. I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well.
For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter.
Main features:
* IPython interpreter support
** All the ipython features are available, including visualization, ipython magics.
* ZeppelinContext support
* Streaming output support
* Support Ipython in PySpark
Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here.
In unit test, I only verify the following 3 popular visualization library. could add more later.
* matplotlib
* bokeh
* ggplot
### What type of PR is it?
[Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2753
### How should this be tested?
Unit test is added.
### Screenshots (if appropriate)
Verify bokeh in IPython Interpreter
Verify matplotlib
Verify ZeppelinContext
Verify Streaming
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2474 from zjffdu/ZEPPELIN-2753 and squashes the following commits:
e869f31 [Jeff Zhang] address comments
b0b5c95 [Jeff Zhang] [ZEPPELIN-2753] Basic Implementation of IPython Interpreter
### What is this PR for?
Adding a section in Shiro Authentication about how to enable secure cookie via Shiro. Shiro do support configuring 'HttpOnly' flag in response cookie. A Zeppelin user, who is security conscious, should know how to enable this in Zeppelin's Shiro configuration.
### What type of PR is it?
Documentation
### What is the Jira issue?
ZEPPELIN-2873
### How should this be tested?
Doc changes. CI test should pass.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Vipin Rathor <v.rathor@gmail.com>
Closes#2545 from VipinRathor/ZEPPELIN-2873 and squashes the following commits:
c7b7995 [Vipin Rathor] Updated doc as per the review comments.
fec8d7e [Vipin Rathor] ZEPPELIN-2873 - Add documentation on Zeppelin Shiro's abliity to configure secure cookie
### What is this PR for?
This PR is trying to add hdfs as another implementation for `NotebookRepo`. There's another PR about using webhdfs to implement that. Actually hdfs client library is compatibility cross major versions. See http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility, if using webhdfs, the code become more complicated and may lose some features of hdfs.
This PR is also required for HA of zeppelin, so that multiple zeppelin instances can share notes via hdfs. I add hadoop-client in pom file. So zeppelin will package hadoop client jar into its binary distribution. This is because zeppelin may be installed in a gateway machine where no hadoop is installed (only hadoop configuration file is existed in this machine) And since the hadoop client will work with multiple versions of hadoop, so it is fine to package into binary distribution. Spark also package hadoop client jar in its binary distribution.
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1515
### How should this be tested?
Unit test is added. Also manually verify it in a single node cluster.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2455 from zjffdu/ZEPPELIN-1515 and squashes the following commits:
b3e83ab [Jeff Zhang] ZEPPELIN-1515. Notebook: HDFS as a backend storage (Read & Write Mode)
### What is this PR for?
upgrade Beam interpreter to use the latest version of Apache Beam.
### What type of PR is it?
[Improvement]
### Todos
*
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2865
### How should this be tested?
* Start the Zeppelin server
* The prefix of interpreter is %beam and then write your code with required imports and the runner
Refer to `docs/interpreter/beam.md` for an example;
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes, updated `docs/interpreter/beam.md` and `README.md`
Author: mingmxu <mingmxu@ebay.com>
Closes#2541 from XuMingmin/ZEPPELIN-2865 and squashes the following commits:
520f0fd7 [mingmxu] restore the notice message of scala-2.10
93b3e24d [mingmxu] upgrade to Apache Beam 2.0.0
### What is this PR for?
Current screenshot images for "Settings" and "About Zeppelin" is outdated since UI is updated. So it'll be better update them with latest one.
### What type of PR is it?
Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2856
### How should this be tested?
Just seeing file changes will be enough :)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: EomJeongyeon <lune6482@naver.com>
Closes#2537 from EomJeongyeon/ZEPPELIN-2856 and squashes the following commits:
153a891 [EomJeongyeon] update screenshot images with new about_menu.png and settings_menu.png
### What is this PR for?
To change the order of content in index documentation. Now the last two contents is "External Resources" and "Available Interpreters". This PR change the order to "Available Interpreters" and "External Resources".
### What type of PR is it?
[Documentation]
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2853
### How should this be tested?
1. Read the doc
### Screenshots (if appropriate)
Before
<img width="726" alt="screen shot 2017-08-15 at 3 08 14 pm" src="https://user-images.githubusercontent.com/19610222/29304040-d9f9adac-81cb-11e7-9aa3-0a325511e267.png">
After
<img width="615" alt="screen shot 2017-08-15 at 3 04 44 pm" src="https://user-images.githubusercontent.com/19610222/29304001-a26fca6a-81cb-11e7-8cf2-be10eef781d9.png">
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Loun Lee <beneficial02@gmail.com>
Closes#2530 from beneficial02/feature/change_doc_order and squashes the following commits:
10a7172 [Loun Lee] feature: change the order of doc content
### What is this PR for?
At compile time Zeppelin is downloading several external resources.
I want to be able to provide alternative URLs to compile when internet is not available(i.e. behind corporate proxy).
### What type of PR is it?
[Improvement]
### What is the Jira issue?
[ZEPPELIN-2766]
### How should this be tested?
```
mvn -DskipTests -Dplugin.frontend.downloadRoot=<alernative-url> -Dplugin.frontend.nodeDownloadRoot=<alernative-url> -Dplugin.frontend.yarnDownloadRoot=<alernative-url> -Dplugin.frontend.npmDownloadRoot=<alernative-url> -Dpypi.repo.url=<alernative-url> clean package
```
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? NO
Author: andrea <andrea.peruffo1982@gmail.com>
Closes#2483 from andreaTP/softCodeRepos and squashes the following commits:
6894bcd5 [andrea] added docs for additional configurations
647417c3 [andrea] fix typo
1e5fd85f [andrea] remove deprecated downloadRoot
0aea864a [andrea] Keep online resources configurable
### What is this PR for?
Complementing Zeppelin-1501 (support for Cassandra with SSL in interpreter) with documentation updates.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2835
* [Zeppelin-2835]
### How should this be tested?
Check that documentation for cassandra interpreter includes cassandra.ssl options.
### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
No
Author: Alex Lourie <alex@instaclustr.com>
Author: Alex Lourie <djay.il@gmail.com>
Closes#2517 from alourie/Zeppelin-2835-addSSL-doco and squashes the following commits:
16719766 [Alex Lourie] Updates
9058ee45 [Alex Lourie] Fix the highlight
02625c94 [Alex Lourie] Adding documentation for Cassandra interpreter SSL
### What is this PR for?
This PR adds support for such a configuration which can give access to user who belongs to "any of" the roles defined in Shiro configuration. By default, as per Shiro implementation, user is allowed only when he/she belongs to "all" the roles defined.
This PR fixes the problem for static users/roles in Shiro as well as Active Directory and/or LDAP based user-group-roles mapping.
### What type of PR is it?
Improvement
### TODO
* [x] - Add documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2825
### How should this be tested?
Add the following in shiro.ini:
[main]
...
anyofroles = org.apache.zeppelin.utils.AnyOfRolesAuthorizationFilter
[urls]
...
/api/interpreter/** = authc, **anyofroles**[admin, role1]
/api/configurations/** = authc, roles[admin]
/api/credential/** = authc, roles[admin]
### Screenshots (if appropriate)
Not applicable
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Vipin Rathor <v.rathor@gmail.com>
Closes#2515 from VipinRathor/ZEPPELIN-2825 and squashes the following commits:
01deb25b0 [Vipin Rathor] ZEPPELIN-2825 - Added license header
2105810f7 [Vipin Rathor] ZEPPELIN-2825 - Fix formatting for doc changes
95a9b4f41 [Vipin Rathor] ZEPPELIN-2825 - Fix formatting in doc changes
3f49d8413 [Vipin Rathor] ZEPPELIN-2825 - Add documentation for supporting any of the Shiro roles
c5fc9deac [Vipin Rathor] ZEPPELIN-2825 - Fix Zeppelin to support any of the Shiro roles
### What is this PR for?
The HTTP Strict-Transport-Security response header (often abbreviated as HSTS) is a security feature that lets a web site tell browsers that it should only be communicated with using HTTPS, instead of using HTTP.
Note: The Strict-Transport-Security header is ignored by the browser when your site is accessed using HTTP; this is because an attacker may intercept HTTP connections and inject the header or remove it. When your site is accessed over HTTPS with no certificate errors, the browser knows your site is HTTPS capable and will honor the Strict-Transport-Security header.
The HTTP X-XSS-Protection response header is a feature of Internet Explorer, Chrome and Safari that stops pages from loading when they detect reflected cross-site scripting (XSS) attacks.
### What type of PR is it?
[Bug Fix | Improvement ]
### What is the Jira issue?
* [ZEPPELIN-2775](https://issues.apache.org/jira/browse/ZEPPELIN-2775)
### How should this be tested?
Make a curl call to Zeppelin? Go to Chrome Browser and select "More Tools" -> "Developer Tools" from the right-side menu. Under Network Section, select any request and check for "Response Headers". You should see below headers along with existing ones.
> strict-transport-security:max-age=631138519
> x-xss-protection:1; mode=block
<img width="1436" alt="screen shot 2017-07-14 at 8 19 14 pm" src="https://user-images.githubusercontent.com/6433184/28217231-16ce6cee-68d2-11e7-91aa-77ad083612c7.png">
### Questions:
* Does this needs documentation?
Author: krishna-pandey <krish.pandey21@gmail.com>
Closes#2492 from krishna-pandey/ZEPPELIN-2775 and squashes the following commits:
7d9978e49 [krishna-pandey] Modified Documentation as per review.
6733289ed [krishna-pandey] Adding documentation for HTTP Security Headers
754d2d71e [krishna-pandey] Supplying String instead of Int (required for Response Header)
468231cc6 [krishna-pandey] Added configurable Strict-Transport-Security and X-XSS-Protection Headers
The github preview is fine but the html on https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/security/shiroauthentication.html#ldap is not displaying the code fragment correctly.
### What is this PR for?
Fixes the documentation orientation for shiro LDAPRealm
### What type of PR is it?
[Documentation]
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2804
### How should this be tested?
Build the documentation and verify that the problem as stated in image is resolved.
### Screenshots (if appropriate)
<img width="856" alt="screen shot 2017-07-21 at 10 08 04 am" src="https://user-images.githubusercontent.com/5103613/28454775-c91fbe1e-6dfc-11e7-9601-6279be44f124.png">
### Questions:
* Does the licenses files need update? N
* Is there breaking changes for older versions? N
* Does this needs documentation? N
Author: Sohaib Iftikhar <sohaib1692@gmail.com>
Closes#2500 from sohaibiftikhar/patch-1 and squashes the following commits:
94e18433 [Sohaib Iftikhar] Fixed documentation for shiro LDAPRealm
### What is this PR for?
Fixing bug on enabling Helium plugins (artifact is empty, not null) when sending curl command.
After noticing out-of-date documentation, I updated it
Also noticed issues on missing try/catch expressions and missing info messages on enable/disable packages.
### What type of PR is it?
[Bug Fix | Documentation | Refactoring]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2699
### How should this be tested?
Run "curl -X POST http://localhost:8080/api/helium/enable/{packageName}"
Check that the package gets enabled
### Questions:
* Does the licenses files need update? N
* Is there breaking changes for older versions? N
* Does this needs documentation? Y (Done)
Author: Nelson Costa <nelson.costa85@gmail.com>
Closes#2446 from necosta/zeppelin2699 and squashes the following commits:
473f9b43 [Nelson Costa] [ZEPPELIN-2699] Fixed initialization issue
45b77513 [Nelson Costa] [ZEPPELIN-2699] More corrections
c8230218 [Nelson Costa] [ZEPPELIN-2699] Bug-fix on gathering package info
2b1ff8ae [Nelson Costa] [ZEPPELIN-2699] Changes after code review
a71b9fb9 [Nelson Costa] [ZEPPELIN-2699] Changes after code-review
5d179218 [Nelson Costa] [ZEPPELIN-2699] Helium REST API bug-fixes. Also updated documentation
I was unaware of this really nice feature, which led me to writing my own "zeppelin_show" function.
### What is this PR for?
(tiny) improvement of the docs
Author: TwUxTLi51Nus <TwUxTLi51Nus@users.noreply.github.com>
Closes#2488 from TwUxTLi51Nus/patch-1 and squashes the following commits:
7ab6f41 [TwUxTLi51Nus] Add z.show() feature to docs
### What is this PR for?
Since it is quite useful to expose the application info for user to monitor and debug, so here propose to enable "zeppelin.livy.displayAppInfo" to true as default.
### What type of PR is it?
Improvement
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2716
### How should this be tested?
Manual verification.
Author: jerryshao <sshao@hortonworks.com>
Closes#2459 from jerryshao/ZEPPELIN-2716 and squashes the following commits:
c9506ab [jerryshao] Address UT failure
32d3f72 [jerryshao] Change the default value of zeppelin.livy.displayAppInfo to true
### What is this PR for?
When "spark.ui.enabled" property is set to "false" we should not show the Spark UI button.
We keep the same behaviour when this property does not exist or when it exists and it's set to true.
### What type of PR is it?
[ Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2714
### How should this be tested?
1. Open Zeppelin, go to Interpreter > Spark
1. Click spark ui > msg: "No spark application running"
1. Go to a paragraph, run "sc.parallelize(1 to 100).count()", check "SPARK JOB" button
1. Go to Interpreter > Spark > Set "spark.ui.enabled" to "false"
1. Rerun paragraph > "SPARK JOB" button not visible
1. Go to Interpreter > Spark > "spark ui" button not visible
Also, test with "spark.ui.enabled"="true" and other workflow combinations
### Questions:
* Does the licenses files need update? N
* Is there breaking changes for older versions? N
* Does this needs documentation? N
Author: Nelson Costa <nelson.costa85@gmail.com>
Closes#2456 from necosta/zeppelin2714 and squashes the following commits:
278a23e [Nelson Costa] [ZEPPELIN-2714] Fixed broken unit-test
edf29cc [Nelson Costa] [ZEPPELIN-2714] Minor final fix
294dea8 [Nelson Costa] [ZEPPELIN-2714] Improvements to change request
a85864c [Nelson Costa] [ZEPPELIN-2714] Soft-code Spark UI button visualization
### What is this PR for?
I spoiled the previous PR #2251
Added widgets (string, text, url, password, url, checkbox) to properties of interpreters. Those are widgets for properties customization. Properties must have the ability to customize the display (for example password).
### What type of PR is it?
Feature
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2403
### How should this be tested?
- remove conf/interpreter.json
- Try new form (create, edit) of interpreter settings
### Screenshots (if appropriate)
edit
view
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Tinkoff DWH <tinkoff.dwh@gmail.com>
Author: isys.mreshetov <m.reshetov@i-sys.ru>
Closes#2268 from tinkoff-dwh/ZEPPELIN-2403 and squashes the following commits:
75a10464 [isys.mreshetov] ZEPPELIN-2403 imports fix
7be8ddff [isys.mreshetov] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
585fc364 [isys.mreshetov] ZEPPELIN-2403 documentation fix
4b633993 [isys.mreshetov] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
726c1f31 [isys.mreshetov] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
b17dfb59 [isys.mreshetov] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
098fbd14 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
a5f13272 [Tinkoff DWH] [ZEPPELIN-2403] checkstyle fix
fd25c467 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
e35ff58f [Tinkoff DWH] [ZEPPELIN-2403] fix checkstyle
7c25b6db [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
10ce996a [Tinkoff DWH] [ZEPPELIN-2403] merge widget and type
ca1e2bf7 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
99daca6d [Tinkoff DWH] [ZEPPELIN-2403] fix rest api test
f735c0a9 [Tinkoff DWH] [ZEPPELIN-2403] fix test
c6d24c4c [Tinkoff DWH] [ZEPPELIN-2403] converter for old settings to new (with widgets)
76a98083 [Tinkoff DWH] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2403
b41e7a3f [Tinkoff DWH] ZEPPELIN-2403 checkstyle
637cb0a1 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
e92713c7 [Tinkoff DWH] [ZEPPELIN-2403] generalized types, added new types
07160e00 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
a495137f [Tinkoff DWH] ZEPPELIN-2403 eslint fix
fd8d2781 [Tinkoff DWH] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2403_backup
4f271d9b [Tinkoff DWH] ZEPPELIN-2403 rename to widget added new widgets string, number, url
dd5d6c80 [Tinkoff DWH] ZEPPELIN-2403 did properties immutable, added new type 'checkbox'
14353b12 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
12499ae1 [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2403
45f5f627 [Tinkoff DWH] ZEPPELIN-2403 added interpreter property types
### What is this PR for?
fix: broken image URLs in 0.8.0-SNAPSHOT doc
using the path `/asset` (the absolute path) for image URLs is actually invalid. That's because each version has its own image directory. So they should use the relative path. `{{BASE_PATH}}`
```
➜ asf-zeppelin tree site | grep asset
├── assets # root asset, we shouldn't use it in versioned doc.
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ ├── assets
```
### What type of PR is it?
[Bug Fix]
### Todos
DONE
### What is the Jira issue?
[ZEPPELIN-2707](https://issues.apache.org/jira/browse/ZEPPELIN-2707)
### How should this be tested?
1. cd `docs/`
2. build: `bundle exec jekyll build --safe`
3. check whether links in `_site` include `/docs/0.8.0-SNAPSHOT` as prefix or not
### Screenshots (if appropriate)
#### Current
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/interpreter/overview.html
#### After
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#2450 from 1ambda/ZEPPELIN-2707/should-use-its-own-asset-directory and squashes the following commits:
fb70214a [1ambda] fix: Use its own asset dir
### What is this PR for?
Pig 0.17.0 has just been released. This PR is to upgrade pig to 0.17.0 and support spark engine which is a big milestone of pig 0.17.0
Main Changes:
* Upgrade pig to 0.17.0
* Remove some code using java reflection in `PigUtils.java`, as pig 0.17.0 has some improvement and expose new apis which could be used pig interpreter.
* Support spark engine
### What type of PR is it?
[Improvement | Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://github.com/zjffdu/zeppelin/tree/ZEPPELIN-2615
### How should this be tested?
Unit test is added and also manually test spark yarn-client mode in pig tutorial note.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2431 from zjffdu/ZEPPELIN-2615 and squashes the following commits:
d4e9a6d [Jeff Zhang] Address comments
4b4e3db [Jeff Zhang] ZEPPELIN-2615. Upgrade pig to 0.17.0 to support spark engine
### What is this PR for?
Integrating Open ID connect login into Zeppelin leveraging Shiro(already present) and Pac4J( that needs to be in the classpath).
Modifications done here should not affect any existing mechanisms but simply integrates and enable new once.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
[ZEPPELIN-2598]
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: andrea <andrea.peruffo1982@gmail.com>
Closes#2373 from andreaTP/keycloak and squashes the following commits:
2d5af6d [andrea] merge with master
31270f2 [andrea] Merge remote-tracking branch 'apache/master' into keycloak
80c375d [andrea] fixed style issues on master
7ec8cdd [andrea] reverted to a minimum
545686d [andrea] fix username after direct login
03bbe30 [andrea] updates
4ce9121 [andrea] Enabling authentication with OpenId connect
### What is this PR for?
Kerberos ticket and renew lifetime are set to 1 hour. On accessing secure Hadoop from shell interpreter, it does kinit and returns result successfully but after 1 hour, the ticket gets expired and Hadoop list fails with below exception.
```
%sh
hadoop fs -ls /
17/01/05 09:29:45 WARN ipc.Client: Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "zeppelin1.hwxblr.com/10.0.1.57"; destination host is: "zeppelin1.hwxblr.com":8020;
ExitValue: 1
```
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
* [ZEPPELIN-1907](https://issues.apache.org/jira/browse/ZEPPELIN-1907)
### How should this be tested?
On a Kerberos enabled cluster, run this paragraph
```
%sh
hdfs dfs -ls /user/zeppelin/
```
Wait for key-tab to expire (or run `kdestroy`), and re-run the same paragraph.
### Screenshots (if appropriate)
Before:
<img width="1438" alt="screen shot 2017-06-13 at 3 44 30 pm" src="https://user-images.githubusercontent.com/674497/27078184-511ed810-5050-11e7-8afa-90247f33047a.png">
After:
<img width="1438" alt="screen shot 2017-06-13 at 3 44 04 pm" src="https://user-images.githubusercontent.com/674497/27078183-5109d690-5050-11e7-82e4-d79a5e98295f.png">
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2407 from prabhjyotsingh/ZEPPELIN-1907 and squashes the following commits:
ffd5f11b2 [Prabhjyot Singh] add bash after `
ecc1a7ce0 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-1907
9243c6ab9 [Prabhjyot Singh] replace `###` with `##`
443c407d3 [Prabhjyot Singh] add space before time(s)
adf23743b [Prabhjyot Singh] update documentation.
289b7d346 [Prabhjyot Singh] reset kinitFailCount on successful renew.
96bfdfe97 [Prabhjyot Singh] log more error
72b32ae25 [Prabhjyot Singh] add java doc
df6645a64 [Prabhjyot Singh] add KerberosInterpreter and move kinit loginc there.
856c8716e [Prabhjyot Singh] renew token periodically
ee741e483 [Prabhjyot Singh] @zjffdu review comments
7c539ef2e [Prabhjyot Singh] add null check
ab823d3ee [Prabhjyot Singh] relogin using keytab, and append message for the same
### What is this PR for?
Improving documentation page. Please check *TODO* and *Screenshots* sections for detail.
The motivation is described in [the JIRA ticket](https://issues.apache.org/jira/browse/ZEPPELIN-2583) and discussion is ongoing on the mailing list.
### What type of PR is it?
[Improvement | Documentation]
### Todos
* [x] - improved the navbar style
* [x] - improved the main page
* [x] - re-organized content structure
* [x] - added tutorial pages: `spark_with_zeppelin.md`, `python_with_zeppelin.md`, `sql_with_zeppelin.md` for overview
* [x] - added `multi_user_support.md` page to provide overview
* [x] - added the empty `interpreter_binding_mode` page. This will be handed in the different issue: [ZEPPELIN-2582](https://issues.apache.org/jira/browse/ZEPPELIN-2582)
* [x] - added the empty `trouble_shooting` page. This can be filled in the following PRs.
* [x] - added the empty `useful_developer_tools` page. This can be filled in the following PRs.
### What is the Jira issue?
[ZEPPELIN-2596](https://issues.apache.org/jira/browse/ZEPPELIN-2596)
### How should this be tested?
1. checkout
2. `cd docs`
3. `bundle install` (make sure that you have ruby 2.1.0+ and bundle)
4. `bundle exec jekyll serve --watch`
5. open `localhost:4000`
### Screenshots (if appropriate)
#### better navbar: before
#### better navbar: after
#### improved main page: before
#### improved main page: after
#### organized content structure: before
#### organized content structure: after
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - related with docs
Author: 1ambda <1amb4a@gmail.com>
Closes#2371 from 1ambda/updating-version-doc and squashes the following commits:
eb02fa967 [1ambda] fix: navbar focus color applies after folding
026379ed6 [1ambda] fix: Remove docs/.listen_test
a7dd4737b [1ambda] fix: sora's comment 1.2
18c5058f7 [1ambda] fix: resolve description in python_with_zeppelin.md
d3ad67c73 [1ambda] fix: sora's comment 4
d133dbbcc [1ambda] fix: resolve sora's comment 3
513c6ff2c [1ambda] fix: resolve sora's comment 1.1
4c2946928 [1ambda] fix: resovle sora's comment 2
1c3946ac6 [1ambda] fix: sora's comment 1
4d6e4267f [1ambda] fix: Resolve sola's comment 3
d0524cafe [1ambda] fix: Set less shadow for nav
5f1f998ba [1ambda] docs: Add useful_develop_tools.md
9dfd62c74 [1ambda] fix: Typo in installation.md
30f7d7e06 [1ambda] fix: Typo in helium ctrl
d6877e792 [1ambda] docs: Add python_with_zeppelin.md
7027e96c0 [1ambda] docs: Improve python conda, docker doc style
e55b50a9d [1ambda] fix: Invalid URLs
75ddeeaff [1ambda] docs: replace URIs in interpreter
5b43993a4 [1ambda] docs: Add sql_with_zeppelin
053794e84 [1ambda] docs: Add spark_with_zeppelin.md
d4d88b9c7 [1ambda] docs: Improve proxy doc
b46cdd126 [1ambda] docs: Add empty interpreter_binding_mode.md
06fcb239e [1ambda] docs: Add empty personalized_mode.md
4991cf0a7 [1ambda] docs: Update upgrading.md
53142b7a0 [1ambda] fix: Simplify install.md
8a5c1e721 [1ambda] docs: Add multi_user_support.md
34095775e [1ambda] fix: Increase font size to 15px
a03b04b33 [1ambda] fix: Remove sample text from trouble_shooting.md
199842590 [1ambda] fix: Remove docker doc link
66a2a7d26 [1ambda] docs: Improve impersonation page
0a6e3fc1d [1ambda] docs: Improve install doc
ccd999ed5 [1ambda] docs: Improve helium doc
f8d742d08 [1ambda] fix: an invalid link in navbar
b7aa5f884 [1ambda] fix: URLs in development
61a175d94 [1ambda] docs: Update install.md
4c56de5c4 [1ambda] fix: URLs in setup
0b1d63513 [1ambda] fix: URLs in quickstart
28970a4fe [1ambda] feat: Add docs/usage
735946bca [1ambda] feat: rename /quickstart
b351cf237 [1ambda] fix: Add missing links
b70770b4f [1ambda] feat: Change URLs in nav, index
94e80aef6 [1ambda] fix: doens't display navbar version in small
6e0cab110 [1ambda] feat: Update doc section names
b9ce256ff [1ambda] feat: Hide version in navbar when md
f8bab52be [1ambda] fix: Better image display in index.md
eeb37d5b5 [1ambda] fix: Add RL padding for mobile browser
ceb60b5ee [1ambda] feat: Style collapsed nav for mobile browser
4ebafb4b6 [1ambda] commit
### What is this PR for?
Added a new REST API that provides more detailed info. on the exact source code Git commit Id that created the build. Useful to track down bugs and replicate issues.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Add more Git properties
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2622
### How should this be tested?
Build Zeppelin. Open http://localhost:8080/api/git
### Screenshots (if appropriate)
Expected output: {"status":"OK","message":"Zeppelin Git info","body":{"commitId":"4c2d295","timestamp":"2017-06-06 19:00:01"}}
### Questions:
* Does the licenses files need update? N
* Is there breaking changes for older versions? N
* Does this needs documentation? Y (Done)
Author: Nelson Costa <nelson.costa85@gmail.com>
Closes#2398 from necosta/zeppelin2622 and squashes the following commits:
dc76a24 [Nelson Costa] [ZEPPELIN-2622] Keep 'About Zeppelin' window only displaying version
426b1ea [Nelson Costa] [ZEPPELIN-2622] Minor adjustments
2d20455 [Nelson Costa] [ZEPPELIN-2622] Merged version and git REST APIs
9ff7d60 [Nelson Costa] [ZEPPELIN-2622] Added missing license
75795f1 [Nelson Costa] [ZEPPELIN-2622] Add Zeppelin Git info REST API
### What is this PR for?
Complements ZEPPELIN-2278, this is a fix for "connection timeout" bug when trying to run "npm install" behind a proxy. Instead of connecting to the official nodejs site from behind a proxy, we should use the already soft-coded env variable ZEPPELIN_HELIUM_NPM_REGISTRY repository.
### What type of PR is it?
Bug Fix
### ToDos
* [x] - Soft-code npm folder name as well (currently hardcoded "npm") - To be agreed on...
### What is the Jira issue?
* [ZEPPELIN-2214 - Bug on connecting to the official nodejs site from behind a proxy]
### How should this be tested?
Enable any Helium plugin from behind a proxy
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Nelson Costa <nelson.costa85@gmail.com>
Closes#2349 from necosta/zeppelin2214 and squashes the following commits:
6d2a52e [Nelson Costa] [ZEPPELIN-2214] Soft-code Node, NPM and YarnPkg download/installer urls for Helium module
### What is this PR for?
Currently allowing authentication for selected roles/groups of an LDAP realm is not possible. The LDAPRealm allows for mapping of roles to groups but only allows authorization on URLs with respect to groups. No group based checks are carried out during authentication. This PR allows for group based authentication using LdapRealm.
### What type of PR is it?
[Improvement]
### Todos
* [x] - Merge #932 - This PR also merges changes from 932 so that needs to be merged first.
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2539
### How should this be tested?
Build and configure `shiro.ini` to use the LdapRealm and verify that the realm works as before along with the added functionality of allowing only certain user groups for authentication if the `allowedRolesForAuthentication` config is set in the init. If this configuration is absent authentication should work as before without verifying roles. A sample shiro.ini is pasted here for testing purposes.
```
[main]
ldapRealm = org.apache.zeppelin.realm.LdapRealm
ldapRealm.userDnTemplate = uid={0},ou=people,dc=my-company,dc=net
ldapRealm.searchBase = dc=my-company,dc=net
ldapRealm.userSearchBase = ou=people,dc=my-company,dc=net
ldapRealm.groupSearchBase = ou=groups,dc=my-company,dc=net
ldapRealm.contextFactory.url = ldaps://auth.my-company.net:636
ldapRealm.contextFactory.authenticationMechanism = simple
ldapRealm.userObjectClass = posixAccount
ldapRealm.groupObjectClass = posixGroup
ldapRealm.authorizationEnabled = true
ldapRealm.memberAttribute = memberUid
ldapRealm.memberAttributeValueTemplate=uid={0},ou=people,dc=my-company,dc=net
ldapRealm.rolesByGroup = GLOBAL_ADMINS:admin,HKG_USERS:user
ldapRealm.allowedRolesForAuthentication = admin,user
ldapRealm.userSearchAttributeName = uid
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
shiro.loginUrl = /api/login
securityManager.sessionManager = $sessionManager
securityManager.sessionManager.globalSessionTimeout = 86400000
securityManager.realms = $ldapRealm
[urls]
/api/version = anon
/api/login = authc
/api/login/logout = authc
/** = authc, roles[admin,user]
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Y (documentation updated in PR)
Author: Sohaib Iftikhar <iftikhar.sohaib@gmail.com>
Closes#2354 from sohaibiftikhar/ldaprealm and squashes the following commits:
f706b22 [Sohaib Iftikhar] Added role based authentication(not to be confused with authorization) for shiro
### What is this PR for?
There are some problems if meta is large (few schemas, each schema contains 500+ tables etc.).
Problems:
1. loading is very long
2. each update takes one connection if updates are long, the situation may arise that the entire pool will be busy
3. no cache
This PR solves these problems. Added cache and access by full path (schema.table, schema.table.column) + protection for release the connections
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2538
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Tinkoff DWH <tinkoff.dwh@gmail.com>
Closes#2343 from tinkoff-dwh/ZEPPELIN-2538 and squashes the following commits:
0991c6ab [Tinkoff DWH] [ZEPPELIN-2538] small improvement
e770d261 [Tinkoff DWH] [ZEPPELIN-2538] update description
a5788743 [Tinkoff DWH] [ZEPPELIN-2538] protection long download
f999488b [Tinkoff DWH] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2538
f26ab5da [Tinkoff DWH] [ZEPPELIN-2538] fix tests
d600fa16 [Tinkoff DWH] [ZEPPELIN-2538] rewrite sql completer to work with large data
### What is this PR for?
Proxy configuration implementation in Helium bundle factory. Used by Node, NPM and YarnPkg installers.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2152
### How should this be tested?
Trigger Helium plugins behind proxy
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Nelson Costa <nelson.costa85@gmail.com>
Closes#2363 from necosta/zeppelin2152 and squashes the following commits:
f681e73 [Nelson Costa] [ZEPPELIN-2152] Added documentation
5e29cca [Nelson Costa] [ZEPPELIN-2152] Fixed format issues
f55e6e2 [Nelson Costa] [ZEPPELIN-2152] Fixed bug in secure/insecure proxy. Added proxy for HeliumOnlineRegistry
d9a086a [Nelson Costa] [ZEPPELIN-2152] Added proxy config support in Helium bundle factory
### What is this PR for?
Add ability to change the log level at runtime
### What type of PR is it?
Improvement
### Todos
* [x] - Update/Add the interpreter rest API documentation(edited: added a new Zeppelin server API doc page)
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2543
### How should this be tested?
>➜ ~ curl -X PUT http://localhost:8080/api/log/level/debug
{"status":"OK"}
>➜ ~ curl -X PUT http://localhost:8080/api/log/level/adx
{"status":"NOT_ACCEPTABLE","message":"Not a valid LOG level"}
### Screenshots (if appropriate) NA
### Questions:
* Does the licenses files need update? NA
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes(Will need to the interpreter Rest api doc, if the approach looks fine)
Author: Karup <karuppayyar@qubole.com>
Closes#2341 from karuppayya/ZEPPELIN-2543 and squashes the following commits:
4017a83 [Karup] Add documenation, fix response message
e4782c3 [Karup] Add abitiy to change the log level at runtime
### What is this PR for?
This PR fixes a security hole in documentation when 'SIMPLE' authentication mechanism is defined in Shiro configuration (http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/security/shiroauthentication.html). With that, user can log in without entering his/her password. Zeppelin documentation should recommend the correct value which is 'simple'.
### What type of PR is it?
Documentation
### What is the Jira issue?
N/A
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Vipin Rathor <v.rathor@gmail.com>
Closes#2331 from VipinRathor/fix-SIMPLE-auth-doc and squashes the following commits:
a1f0e48 [Vipin Rathor] [minor] Replace 'SIMPLE' auth with 'simple' auth everywhere in documents
### What is this PR for?
Fix JDBC doc after #2229.
### What type of PR is it?
[Bug Fix]
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2314 from prabhjyotsingh/minor/jdbc-doc-zeppelin-2367 and squashes the following commits:
e54a3a2b7 [Prabhjyot Singh] @AhyoungRyu review comments
0f396ac64 [Prabhjyot Singh] fix doc for zeppelin-2367
### What is this PR for?
Use single `Dockerfile` for each release since [apache infra uses tag pushes](https://issues.apache.org/jira/browse/INFRA-12781) to build an image.
- https://issues.apache.org/jira/browse/INFRA-12781
After release process finishes, dockerhub will build using the pushed tag.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
[ZEPPELIN-2492](https://issues.apache.org/jira/browse/ZEPPELIN-2492)
### How should this be tested?
1. `./dev/change_zeppelin_version.sh 0.8.0-SNAPSHOT 0.7.1`
2. Check that version is properly set: `vi scrtips/docker/zeppelin/bin/Dockerfile`
3. Build docker image `cd scripts/docker/zeppelin/bin; docker build -t zeppelin:0.7.1 ./`
4. Run the image: `docker run -p 8080:8080 --rm --name zeppelin zeppelin:0.7.1`
### Screenshots (if appropriate)
NONE
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#2318 from 1ambda/ZEPPELIN-2492/use-single-dockerfile-for-each-tag and squashes the following commits:
483bec3 [1ambda] docs: Update README for Dockerfile
5826c8c [1ambda] fix: Use single dockerfile for tag push
### What is this PR for?
Hive JDBC proxy user option should be available generically.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* [Zeppelin 2367](https://issues.apache.org/jira/browse/ZEPPELIN-2367)
### How should this be tested?
Enable Shiro authentication and set `zeppelin.jdbc.auth.type` as `SIMPLE` in the interpreter setting, and observe the connection string for the Hive.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need an update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? Yes
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#2229 from prabhjyotsingh/ZEPPELIN-2367 and squashes the following commits:
84b5e55b2 [Prabhjyot Singh] add logger.warn for hive and impersonation
45c90a8e2 [Prabhjyot Singh] improve doc
9fee9d2a9 [Prabhjyot Singh] replace hive with generic method
a348e969a [Prabhjyot Singh] revert "zeppelin.jdbc.auth.kerberos.proxy.enable" behaviour
e2bdbb2ad [Prabhjyot Singh] include e as inner exception
c180f5ce2 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367
1802b453f [Prabhjyot Singh] remove hive string from logger
513987a28 [Prabhjyot Singh] apply genric logic to appendProxyUserToURL
3fa2b1e98 [Prabhjyot Singh] change name to appendProxyUserToURL
a75167415 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367
4c382eefa [Prabhjyot Singh] log user details as well
d51e770b2 [Prabhjyot Singh] add doc in jdbc.md
01b18b9d2 [Prabhjyot Singh] add doc (reverted from commit ee8a6b524c)
40489c89d [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367
ee8a6b524 [Prabhjyot Singh] add doc
8999d93ae [Prabhjyot Singh] ZEPPELIN-2367: Hive JDBC proxy user option should be avail even without kerberos
### What is this PR for?
Just noticed some grammar inconsistencies that I thought I could clean up while getting myself started on Zeppelin. Thanks for a great project!
### What type of PR is it?
Documentation
Author: Michael Ressler <mressler@diamondkinetics.com>
Closes#2308 from mressler/master and squashes the following commits:
aa39597 [Michael Ressler] Some grammar updates.
### What is this PR for?
The example in the doc would fail due to typo, straightforward fix.
### What type of PR is it?
[Hot Fix]
### Todos
* [ ] - Task
### How should this be tested?
Outline the steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2307 from zjffdu/doc_hotfix and squashes the following commits:
133fe72 [Jeff Zhang] [HOTFIX]. Minor doc fix for back-end-angular.md
### What is this PR for?
General solution to execute precode. Refactoring jdbc precode using general solution. Task contains to subtasks: executeAfterOpen, executeBeforeClose. executeBeforeClose not done because we need the context so there is a solution only for executeAfterOpen.
### What type of PR is it?
Feature | Refactoring
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2216
### How should this be tested?
1. Add parameter zeppelin.PySparkInterpreter.precode `someVar='text'`
2. Execute
```
%pyspark
print(someVar)
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Tinkoff DWH <tinkoff.dwh@gmail.com>
Closes#2221 from tinkoff-dwh/ZEPPELIN-2216 and squashes the following commits:
1e3f3f7 [Tinkoff DWH] [ZEPPELIN-2216] fix path
e4cf72f [Tinkoff DWH] [ZEPPELIN-2216] added tests
5a482a0 [Tinkoff DWH] [ZEPPELIN-2216] fix tests
3977722 [Tinkoff DWH] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2216
c0436a2 [Tinkoff DWH] [ZEPPELIN-2216] general solution to precode. refactoring jdbc precode
### What is this PR for?
Created `Dockerfile` for released bin
- based on **Ubuntu:16.04 (LTS)** for desktop usage
- **JDK 8**
- **R** with basic packages
- **Python 2** with basic packages
- **miniconda2** for `%python.conda`
### Details
We already discussed about using alpine image in https://github.com/apache/zeppelin/pull/1761.
- However, it's not designed for desktop usage
- Doesn't have some official packages (R, ...)
- Not familiar to users for desktop OS
That the reason why ubuntu is used in base image
```
zeppelin base b3818f9ae4b1 11 hours ago 1.67 GB
zeppelin 0.6.2 c0a4d8556f92 7 hours ago 2.29 GB
zeppelin 0.7.0 c4a5ad0d04bd 8 hours ago 2.5 GB
zeppelin 0.7.1 54173b77743b 7 hours ago 2.49 GB
```
### What type of PR is it?
[Feature]
### Todos
* [x] - base image
* [x] - script for creating bin images
* [x] - bin image template
### What is the Jira issue?
[ZEPPELIN-1711](https://issues.apache.org/jira/browse/ZEPPELIN-1711)
### How should this be tested?
1. build base image `cd scripts/docker/zeppelin/base; docker build -t zeppelin:base ./`
2. build bin image `cd scripts/docker/zeppelin/0.7.1; docker build -t zeppelin:0.7.1 ./`
3. execute docker images
```
docker run -p 8080:8080 --rm --name zeppelin zeppelin:0.7.1
```
since it takes time to build, you can use already [published docker images](https://hub.docker.com/r/1ambda/docker-zeppelin/)
```
docker run -p 8080:8080 --rm --name zeppelin 1ambda/docker-zeppelin:0.7.1
```
4. should be able to run spark, python and R tutorials
### Screenshots (if appropriate)
NO
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - YES, updated
Author: 1ambda <1amb4a@gmail.com>
Closes#2264 from 1ambda/ZEPPELIN-1711/bin-dockerfile and squashes the following commits:
69a0b1f [1ambda] docs: Update docker.md
ced897f [1ambda] fix: DON'T remove /tmp
1f6da76 [1ambda] feat: Dockerfiles for 060, 070, 071
0fc3f75 [1ambda] feat: Add template for bin image
5cba56e [1ambda] feat: Use ubuntu for base image
### What is this PR for?
Target: Create ability to call http services with custom keystores in a groovy way.
The following should work:
```groovy
//connect to host xxx.yyy with special keystore
HTTP.get(
url: "https://xxx.yyy/zzz",
ssl: " HTTP.getKeystoreSSLContext('./xxx_yyy_keystore.jks', 'testpass') "
)
//take context initialization code from groovy interpreter properties
HTTP.get(
url: "https://xxx.yyy/zzz",
ssl: g.SSL_CONTEXT_FROM_GROOVY_INTERPRET_PROPERTIES
)
//connect to host xxx.yyy with trust all (do not check trust certificates - dev mode only)
HTTP.get(
url: "https://xxx.yyy/zzz",
ssl: " HTTP.getNaiveSSLContext() "
)
//
HTTP.get(
url: "https://xxx.yyy/zzz",
ssl: " MyCustomSSLBuilder.build() "
)
```
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
[ZEPPELIN-2443]
### How should this be tested?
follow the samples above or in documentation
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? YES
Author: dlukyanov <dlukyanov@ukr.net>
Closes#2287 from dlukyanov/master and squashes the following commits:
4baa22e [dlukyanov] ZEPPELIN-2443
### What is this PR for?
Documentation - Adding an example use of prinicpalSuffix to the Active Directory authentication documentation. It took us a while to figure out that the lack of a principalSuffix was our problem and then a bit more hunting to figure out how to automatically include it
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
Outline the steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: Paul Brenner <paul-brenner@users.noreply.github.com>
Closes#2281 from paul-brenner/patch-1 and squashes the following commits:
2ff57a9 [Paul Brenner] added principalSuffix to AD examples
### What is this PR for?
What I did for the documents:
* Highlight codes
* Follow JSON syntax
* Remove white spaces
And in my opinion, [here](https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/writingzeppelinvisualization.html#1-create-a-npm-package) is ambiguous:
> "Normally, you can add any dependencies in package.json however Zeppelin Visualization package only allows two dependencies: zeppelin-vis and zeppelin-tabledata."
Does it want to say "you can add any dependencies in package.json, but you must include two dependencies: zeppelin-vis and zeppelin-tabledata."?
### What type of PR is it?
[Documentation]
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? NO
Author: Jun Kim <i2r.jun@gmail.com>
Closes#2236 from tae-jun/helium-doc and squashes the following commits:
63505e9 [Jun Kim] Fix ambiguous sentence
3e775cf [Jun Kim] Highlight codes, follow JSON syntax, and remove white spaces
