zeppelin/docs/interpreter/shell.md

85 lines
2.9 KiB
Markdown
Raw Normal View History

---
layout: page
[ZEPPELIN-1219] Add searching feature to Zeppelin docs site ### What is this PR for? As more and more document pages are added, it's really hard to find specific pages. So I added searching feature to Zeppelin documentation site([jekyll](https://jekyllrb.com/) based site) using [lunr.js](http://lunrjs.com/). - **How does it work?** I created [`search_data.json`](https://github.com/AhyoungRyu/zeppelin/blob/6e02423f541cc406e4e41031629609a276a9f481/docs/search_data.json) which is used for docs info template. `lunr.js` combines all of the text from all of the docs in `docs/` into `_site/search_data.json`. It looks like below. ![screen shot 2016-08-03 at 4 49 59 am](https://cloud.githubusercontent.com/assets/10060731/17342828/f2908be8-5935-11e6-8eee-b189677c0531.png) All the info are comes from [Jekyll YAML front matter](https://jekyllrb.com/docs/frontmatter/) variables. (i.e. title, group, description.. that's why I rewrote all docs' title and description.) [search.js](https://github.com/AhyoungRyu/zeppelin/blob/6e02423f541cc406e4e41031629609a276a9f481/docs/assets/themes/zeppelin/js/search.js) will do this job using this data! ### What type of PR is it? Improvement & Feature ### Todos * [x] - Keep consistency for all docs pages' `Title` * [x] - Add some overview sentences to all docs pages' `Description` section (this will be used as the result preview) * [x] - Add apache license header to all docs page (some pages are missing the license header currently) * [x] - Add LICENSE for `lunr.min.js` ### What is the Jira issue? [ZEPPELIN-1219](https://issues.apache.org/jira/browse/ZEPPELIN-1219) ### How should this be tested? 1. Apply this patch and build `ZEPPELIN_HOME/docs` dir -> please see [docs/README.md#build-documentation](https://github.com/apache/zeppelin/tree/master/docs#build-documentation) 2. Click `search` icon in navbar and go to `search.html` page 3. Type anything you want to search in the search bar (i.e. type `python`, `spark`, `dynamic` ... ) ### Screenshots (if appropriate) ![screen shot 2016-08-03 at 4 42 28 pm](https://cloud.githubusercontent.com/assets/10060731/17357851/d092e2ca-5999-11e6-9917-a3d4113e6e43.png) ![search](https://cloud.githubusercontent.com/assets/10060731/17357828/b2486cd6-5999-11e6-873b-121fac033b03.gif) ### Questions: * Does the licenses files need update? Yes, for `lunr.min.js` * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1266 from AhyoungRyu/ZEPPELIN-1219 and squashes the following commits: 7ec8854 [AhyoungRyu] Modify 'no result' sentence 91b71a7 [AhyoungRyu] Remove Apache license header since JSON doesn't allow comment 34afd5d [AhyoungRyu] Add Apache license header to search_data.json 6784282 [AhyoungRyu] Minor search page UI update 0389d28 [AhyoungRyu] Make index.md not to be searched 9f1ba42 [AhyoungRyu] Disable enterkey press & change icon bd4956a [AhyoungRyu] Add docs.js & search.js to exclude list in pom.xml 624b051 [AhyoungRyu] Add Apache license header to search.js 1381152 [AhyoungRyu] Fix search result skipping issue 6e775f5 [AhyoungRyu] Make pleasecontribute.md not to be searched ee11136 [AhyoungRyu] Fix some typos fa01299 [AhyoungRyu] Refine 'description' in some docs as @bzz suggested da0cff9 [AhyoungRyu] Exclude lunr.min.js 36ba7f1 [AhyoungRyu] Add lunr.min.js license info f6a05a6 [AhyoungRyu] Apply css style for the search results 68eb997 [AhyoungRyu] Attach 'Apache Zeppelin ZEPPELIN_VERSION Documentation: ' to title d908c37 [AhyoungRyu] Add searching page a951fa6 [AhyoungRyu] Add search icon to navbar 0688a79 [AhyoungRyu] Keep consistency all docs' front matter for the right search result 040f532 [AhyoungRyu] Add template for storing docs info based on jekyll front matter 0705bd6 [AhyoungRyu] Add js files: lunr.min.js & search.js
2016-08-06 05:50:25 +00:00
title: "Shell interpreter for Apache Zeppelin"
description: "Shell interpreter uses Apache Commons Exec to execute external processes."
group: interpreter
---
[ZEPPELIN-1219] Add searching feature to Zeppelin docs site ### What is this PR for? As more and more document pages are added, it's really hard to find specific pages. So I added searching feature to Zeppelin documentation site([jekyll](https://jekyllrb.com/) based site) using [lunr.js](http://lunrjs.com/). - **How does it work?** I created [`search_data.json`](https://github.com/AhyoungRyu/zeppelin/blob/6e02423f541cc406e4e41031629609a276a9f481/docs/search_data.json) which is used for docs info template. `lunr.js` combines all of the text from all of the docs in `docs/` into `_site/search_data.json`. It looks like below. ![screen shot 2016-08-03 at 4 49 59 am](https://cloud.githubusercontent.com/assets/10060731/17342828/f2908be8-5935-11e6-8eee-b189677c0531.png) All the info are comes from [Jekyll YAML front matter](https://jekyllrb.com/docs/frontmatter/) variables. (i.e. title, group, description.. that's why I rewrote all docs' title and description.) [search.js](https://github.com/AhyoungRyu/zeppelin/blob/6e02423f541cc406e4e41031629609a276a9f481/docs/assets/themes/zeppelin/js/search.js) will do this job using this data! ### What type of PR is it? Improvement & Feature ### Todos * [x] - Keep consistency for all docs pages' `Title` * [x] - Add some overview sentences to all docs pages' `Description` section (this will be used as the result preview) * [x] - Add apache license header to all docs page (some pages are missing the license header currently) * [x] - Add LICENSE for `lunr.min.js` ### What is the Jira issue? [ZEPPELIN-1219](https://issues.apache.org/jira/browse/ZEPPELIN-1219) ### How should this be tested? 1. Apply this patch and build `ZEPPELIN_HOME/docs` dir -> please see [docs/README.md#build-documentation](https://github.com/apache/zeppelin/tree/master/docs#build-documentation) 2. Click `search` icon in navbar and go to `search.html` page 3. Type anything you want to search in the search bar (i.e. type `python`, `spark`, `dynamic` ... ) ### Screenshots (if appropriate) ![screen shot 2016-08-03 at 4 42 28 pm](https://cloud.githubusercontent.com/assets/10060731/17357851/d092e2ca-5999-11e6-9917-a3d4113e6e43.png) ![search](https://cloud.githubusercontent.com/assets/10060731/17357828/b2486cd6-5999-11e6-873b-121fac033b03.gif) ### Questions: * Does the licenses files need update? Yes, for `lunr.min.js` * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1266 from AhyoungRyu/ZEPPELIN-1219 and squashes the following commits: 7ec8854 [AhyoungRyu] Modify 'no result' sentence 91b71a7 [AhyoungRyu] Remove Apache license header since JSON doesn't allow comment 34afd5d [AhyoungRyu] Add Apache license header to search_data.json 6784282 [AhyoungRyu] Minor search page UI update 0389d28 [AhyoungRyu] Make index.md not to be searched 9f1ba42 [AhyoungRyu] Disable enterkey press & change icon bd4956a [AhyoungRyu] Add docs.js & search.js to exclude list in pom.xml 624b051 [AhyoungRyu] Add Apache license header to search.js 1381152 [AhyoungRyu] Fix search result skipping issue 6e775f5 [AhyoungRyu] Make pleasecontribute.md not to be searched ee11136 [AhyoungRyu] Fix some typos fa01299 [AhyoungRyu] Refine 'description' in some docs as @bzz suggested da0cff9 [AhyoungRyu] Exclude lunr.min.js 36ba7f1 [AhyoungRyu] Add lunr.min.js license info f6a05a6 [AhyoungRyu] Apply css style for the search results 68eb997 [AhyoungRyu] Attach 'Apache Zeppelin ZEPPELIN_VERSION Documentation: ' to title d908c37 [AhyoungRyu] Add searching page a951fa6 [AhyoungRyu] Add search icon to navbar 0688a79 [AhyoungRyu] Keep consistency all docs' front matter for the right search result 040f532 [AhyoungRyu] Add template for storing docs info based on jekyll front matter 0705bd6 [AhyoungRyu] Add js files: lunr.min.js & search.js
2016-08-06 05:50:25 +00:00
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
{% include JB/setup %}
# Shell interpreter for Apache Zeppelin
<div id="toc"></div>
## Overview
Shell interpreter uses [Apache Commons Exec](https://commons.apache.org/proper/commons-exec) to execute external processes.
In Zeppelin notebook, you can use ` %sh ` in the beginning of a paragraph to invoke system shell and run commands.
> **Note :** Currently each command runs as the user Zeppelin server is running as.
## Configuration
At the "Interpreters" menu in Zeppelin dropdown menu, you can set the property value for Shell interpreter.
<table class="table-configuration">
<tr>
<th>Name</th>
<th>Value</th>
<th>Description</th>
</tr>
<tr>
<td>shell.command.timeout.millisecs</td>
<td>60000</td>
<td>Shell command time out in millisecs</td>
</tr>
<tr>
<td>shell.working.directory.user.home</td>
<td>false</td>
<td>If this set to true, the shell's working directory will be set to user home</td>
</tr>
<tr>
<td>zeppelin.shell.auth.type</td>
<td></td>
<td>Types of authentications' methods supported are SIMPLE, and KERBEROS</td>
</tr>
<tr>
<td>zeppelin.shell.principal</td>
<td></td>
<td>The principal name to load from the keytab</td>
</tr>
<tr>
<td>zeppelin.shell.keytab.location</td>
<td></td>
<td>The path to the keytab file</td>
</tr>
</table>
## Example
The following example demonstrates the basic usage of Shell in a Zeppelin notebook.
[ZEPPELIN-2707][DOCS][HOTFIX] fix: broken image URLs in 0.8.0-SNAPSHOT doc ### What is this PR for? fix: broken image URLs in 0.8.0-SNAPSHOT doc using the path `/asset` (the absolute path) for image URLs is actually invalid. That's because each version has its own image directory. So they should use the relative path. `{{BASE_PATH}}` ``` ➜ asf-zeppelin tree site | grep asset ├── assets # root asset, we shouldn't use it in versioned doc. │ │ ├── assets │ │ ├── assets │ │ ├── assets │ │ ├── assets │ │ ├── assets │ │ ├── assets │ │ ├── assets │ │ ├── assets │ │ ├── assets │ ├── assets ``` ### What type of PR is it? [Bug Fix] ### Todos DONE ### What is the Jira issue? [ZEPPELIN-2707](https://issues.apache.org/jira/browse/ZEPPELIN-2707) ### How should this be tested? 1. cd `docs/` 2. build: `bundle exec jekyll build --safe` 3. check whether links in `_site` include `/docs/0.8.0-SNAPSHOT` as prefix or not ### Screenshots (if appropriate) #### Current http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/interpreter/overview.html ![image](https://user-images.githubusercontent.com/4968473/27685112-bcca5846-5d07-11e7-89bb-3965015cf5a5.png) #### After ![image](https://user-images.githubusercontent.com/4968473/27685218-3490fad8-5d08-11e7-9133-0e148689986e.png) ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO Author: 1ambda <1amb4a@gmail.com> Closes #2450 from 1ambda/ZEPPELIN-2707/should-use-its-own-asset-directory and squashes the following commits: fb70214a [1ambda] fix: Use its own asset dir
2017-06-29 11:13:41 +00:00
<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/shell-example.png" />
[ZEPPELIN-2596] Improving documentation page ### What is this PR for? Improving documentation page. Please check *TODO* and *Screenshots* sections for detail. The motivation is described in [the JIRA ticket](https://issues.apache.org/jira/browse/ZEPPELIN-2583) and discussion is ongoing on the mailing list. ### What type of PR is it? [Improvement | Documentation] ### Todos * [x] - improved the navbar style * [x] - improved the main page * [x] - re-organized content structure * [x] - added tutorial pages: `spark_with_zeppelin.md`, `python_with_zeppelin.md`, `sql_with_zeppelin.md` for overview * [x] - added `multi_user_support.md` page to provide overview * [x] - added the empty `interpreter_binding_mode` page. This will be handed in the different issue: [ZEPPELIN-2582](https://issues.apache.org/jira/browse/ZEPPELIN-2582) * [x] - added the empty `trouble_shooting` page. This can be filled in the following PRs. * [x] - added the empty `useful_developer_tools` page. This can be filled in the following PRs. ### What is the Jira issue? [ZEPPELIN-2596](https://issues.apache.org/jira/browse/ZEPPELIN-2596) ### How should this be tested? 1. checkout 2. `cd docs` 3. `bundle install` (make sure that you have ruby 2.1.0+ and bundle) 4. `bundle exec jekyll serve --watch` 5. open `localhost:4000` ### Screenshots (if appropriate) #### better navbar: before ![2596_before_nav](https://cloud.githubusercontent.com/assets/4968473/26542353/89004e7a-4494-11e7-89c0-28d608f5f375.gif) #### better navbar: after ![2596_after_nav](https://cloud.githubusercontent.com/assets/4968473/26542356/8bfb7b90-4494-11e7-9979-0bcaef8ba97b.gif) #### improved main page: before ![2596_before_main](https://cloud.githubusercontent.com/assets/4968473/26542358/8f35b0be-4494-11e7-8a6c-e74ec52fc384.gif) #### improved main page: after ![2596_after_main](https://cloud.githubusercontent.com/assets/4968473/26542366/93b333c8-4494-11e7-981f-3f7b4545868f.gif) #### organized content structure: before ![2596_before_content](https://cloud.githubusercontent.com/assets/4968473/26542398/ad81ac26-4494-11e7-9a17-70dff41396fb.gif) #### organized content structure: after ![2596_after_content](https://cloud.githubusercontent.com/assets/4968473/26542403/b0a42ad2-4494-11e7-8bd3-8a5bd194c6af.gif) ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - related with docs Author: 1ambda <1amb4a@gmail.com> Closes #2371 from 1ambda/updating-version-doc and squashes the following commits: eb02fa967 [1ambda] fix: navbar focus color applies after folding 026379ed6 [1ambda] fix: Remove docs/.listen_test a7dd4737b [1ambda] fix: sora's comment 1.2 18c5058f7 [1ambda] fix: resolve description in python_with_zeppelin.md d3ad67c73 [1ambda] fix: sora's comment 4 d133dbbcc [1ambda] fix: resolve sora's comment 3 513c6ff2c [1ambda] fix: resolve sora's comment 1.1 4c2946928 [1ambda] fix: resovle sora's comment 2 1c3946ac6 [1ambda] fix: sora's comment 1 4d6e4267f [1ambda] fix: Resolve sola's comment 3 d0524cafe [1ambda] fix: Set less shadow for nav 5f1f998ba [1ambda] docs: Add useful_develop_tools.md 9dfd62c74 [1ambda] fix: Typo in installation.md 30f7d7e06 [1ambda] fix: Typo in helium ctrl d6877e792 [1ambda] docs: Add python_with_zeppelin.md 7027e96c0 [1ambda] docs: Improve python conda, docker doc style e55b50a9d [1ambda] fix: Invalid URLs 75ddeeaff [1ambda] docs: replace URIs in interpreter 5b43993a4 [1ambda] docs: Add sql_with_zeppelin 053794e84 [1ambda] docs: Add spark_with_zeppelin.md d4d88b9c7 [1ambda] docs: Improve proxy doc b46cdd126 [1ambda] docs: Add empty interpreter_binding_mode.md 06fcb239e [1ambda] docs: Add empty personalized_mode.md 4991cf0a7 [1ambda] docs: Update upgrading.md 53142b7a0 [1ambda] fix: Simplify install.md 8a5c1e721 [1ambda] docs: Add multi_user_support.md 34095775e [1ambda] fix: Increase font size to 15px a03b04b33 [1ambda] fix: Remove sample text from trouble_shooting.md 199842590 [1ambda] fix: Remove docker doc link 66a2a7d26 [1ambda] docs: Improve impersonation page 0a6e3fc1d [1ambda] docs: Improve install doc ccd999ed5 [1ambda] docs: Improve helium doc f8d742d08 [1ambda] fix: an invalid link in navbar b7aa5f884 [1ambda] fix: URLs in development 61a175d94 [1ambda] docs: Update install.md 4c56de5c4 [1ambda] fix: URLs in setup 0b1d63513 [1ambda] fix: URLs in quickstart 28970a4fe [1ambda] feat: Add docs/usage 735946bca [1ambda] feat: rename /quickstart b351cf237 [1ambda] fix: Add missing links b70770b4f [1ambda] feat: Change URLs in nav, index 94e80aef6 [1ambda] fix: doens't display navbar version in small 6e0cab110 [1ambda] feat: Update doc section names b9ce256ff [1ambda] feat: Hide version in navbar when md f8bab52be [1ambda] fix: Better image display in index.md eeb37d5b5 [1ambda] fix: Add RL padding for mobile browser ceb60b5ee [1ambda] feat: Style collapsed nav for mobile browser 4ebafb4b6 [1ambda] commit
2017-06-19 10:13:57 +00:00
If you need further information about **Zeppelin Interpreter Setting** for using Shell interpreter,
[ZEPPELIN-1907] Shell Interpreter does not renew ticket on secure cluster ### What is this PR for? Kerberos ticket and renew lifetime are set to 1 hour. On accessing secure Hadoop from shell interpreter, it does kinit and returns result successfully but after 1 hour, the ticket gets expired and Hadoop list fails with below exception. ``` %sh hadoop fs -ls / 17/01/05 09:29:45 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757) at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618) at org.apache.hadoop.ipc.Client.call(Client.java:1449) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "zeppelin1.hwxblr.com/10.0.1.57"; destination host is: "zeppelin1.hwxblr.com":8020; ExitValue: 1 ``` ### What type of PR is it? [Bug Fix] ### What is the Jira issue? * [ZEPPELIN-1907](https://issues.apache.org/jira/browse/ZEPPELIN-1907) ### How should this be tested? On a Kerberos enabled cluster, run this paragraph ``` %sh hdfs dfs -ls /user/zeppelin/ ``` Wait for key-tab to expire (or run `kdestroy`), and re-run the same paragraph. ### Screenshots (if appropriate) Before: <img width="1438" alt="screen shot 2017-06-13 at 3 44 30 pm" src="https://user-images.githubusercontent.com/674497/27078184-511ed810-5050-11e7-8afa-90247f33047a.png"> After: <img width="1438" alt="screen shot 2017-06-13 at 3 44 04 pm" src="https://user-images.githubusercontent.com/674497/27078183-5109d690-5050-11e7-82e4-d79a5e98295f.png"> ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: Prabhjyot Singh <prabhjyotsingh@gmail.com> Author: Prabhjyot Singh <prabhjyotsingh@gmail.com> Closes #2407 from prabhjyotsingh/ZEPPELIN-1907 and squashes the following commits: ffd5f11b2 [Prabhjyot Singh] add bash after ` ecc1a7ce0 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-1907 9243c6ab9 [Prabhjyot Singh] replace `###` with `##` 443c407d3 [Prabhjyot Singh] add space before time(s) adf23743b [Prabhjyot Singh] update documentation. 289b7d346 [Prabhjyot Singh] reset kinitFailCount on successful renew. 96bfdfe97 [Prabhjyot Singh] log more error 72b32ae25 [Prabhjyot Singh] add java doc df6645a64 [Prabhjyot Singh] add KerberosInterpreter and move kinit loginc there. 856c8716e [Prabhjyot Singh] renew token periodically ee741e483 [Prabhjyot Singh] @zjffdu review comments 7c539ef2e [Prabhjyot Singh] add null check ab823d3ee [Prabhjyot Singh] relogin using keytab, and append message for the same
2017-06-23 16:21:00 +00:00
please read [What is interpreter setting?](../usage/interpreter/overview.html#what-is-interpreter-setting) section first.
## Kerberos refresh interval
For changing the default behavior of when to renew Kerberos ticket following changes can be made in `conf/zeppelin-env.sh`.
```bash
# Change Kerberos refresh interval (default value is 1d). Allowed postfix are ms, s, m, min, h, and d.
export LAUNCH_KERBEROS_REFRESH_INTERVAL=4h
# Change kinit number retries (default value is 5), which means if the kinit command fails for 5 retries consecutively it will close the interpreter.
export KINIT_FAIL_THRESHOLD=10
```