mirror of
https://github.com/apache/zeppelin
synced 2026-05-24 09:38:26 +00:00
15 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Lee moon soo
|
82749aa15b |
[ZEPPELIN-3982] add how to setup development mode for Kubernetes support
### What is this PR for?
This pr adds docs that explains development mode for Kubernetes support.
### What type of PR is it?
Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3982
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes #3298 from Leemoonsoo/ZEPPELIN-3982 and squashes the following commits:
|
||
|
Lee moon soo
|
b13651cedf |
[ZEPPELIN-3840] Zeppelin on Kubernetes
### What type of PR is it? This PR adds ability to run Zeppelin on Kubernetes. It aims - Zero configuration to start Zeppelin on Kubernetes. (and Spark on Kubernetes) - Run everything on Kubernetes: Zeppelin, Interpreters, Spark. - Highly customizable to adopt various user configurations and extensions. Key features are - Provides zeppelin-server.yaml file for `kubectl` to run Zeppelin server - All interpreters are automatically running as a Pod. - Spark interpreter automatically configured to use [Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html) - Reverse proxy is configured to access Spark UI To do - [x] Document how reverse proxy for Spark UI works and how to configure custom domain. - [x] Document how to customize zeppelin-server and interpreter yaml. - [x] Document new configurations - [x] Document how to mount volume for notebook and configurations ### How it works #### Run Zeppelin Server on Kubernetes `k8s/zeppelin-server.yaml` is provided to run Zeppelin Server with few sidecars and configurations. This file is easy to publish (user can easily consume it using `curl`), highly customizable while it includes all the necessary things. #### K8s Interpreter launcher This PR adds new module, `launcher-k8s-standard` under `zeppelin/zeppelin-plugins/launcher/k8s-standard/` directory. This launcher is [automatically being selected](https://github.com/apache/zeppelin/pull/3240/files#diff-82fddd2ffb77aaffc4b9cf7b5b1eaa79) when Zeppelin is running on Kubernetes. The launcher both handles Spark interpreter and All other interpreters. The launcher launches interpreter as a Pod using template [k8s/interpreter/100-interpreter-pod.yaml](https://github.com/apache/zeppelin/pull/3240/files#diff-d9ce62e2c992d32f0184d7edb862f3c4). Reason filename has `100-` in prefix is because all files in the directory is consumed in alphabetical order by launcher on interpreter start/stop. User can drop more files here to extend/customize interpreter, and filename can be used to control order. The template is rendered by [jinjava](https://github.com/HubSpot/jinjava). #### Spark interpreter When interpreter group is `spark`, K8sRemoteInterpreterProcess [sets necessary spark configuration](https://github.com/apache/zeppelin/pull/3240/files#diff-6d1d3084f55bdd519e39ede4a619e73dR297) automatically to use [Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html). User doesn't have to configure anything. It uses client mode. #### Spark UI We may make user manually configure port-forward or do something to access Spark UI, but that's not optimal. It is the best when Spark UI is automatically accessible when user have access to Zeppelin UI, without any extra configuration. To enable this, Zeppelin server Pod has a reverse proxy as a sidecar, and it split traffic to Zeppelin server and Spark UI running in the other Pod. It assume both `service.domain.com` and `*.service.domain.com` point the nginx proxy address. `service.domain.com` is directed to ZeppelinServer, `*.service.domain.com` is directed to interpreter Pod. `<port>-<interpreter pod svc name>.service.domain.com` is convention to access any application running in interpreter Pod. If Spark interpreter Pod is running with a name `spark-axefeg` and Spark UI is running on port 4040, ``` 4040-spark-axefeg.service.domain.com ``` is the address to access Spark UI. Default service domain is [local.zeppelin-project.org:8080](https://github.com/apache/zeppelin/pull/3240/files#diff-56ccb2e2c2617b27dbaae866d9431e51R22), while `local.zeppelin-project.org` and `*.local.zeppelin-project.org` point `127.0.0.1`, and it works with `kubectl port-forward`. ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3840 ### How should this be tested? Prepare a Kubernetes cluster with enough resources (cpus > 5, mem > 6g). If you're using [minikube](https://github.com/kubernetes/minikube), check your capacity using `kubectl describe node` command before start. You'll need to build Zeppelin docker image and Spark docker image to test. Please follow guide docs/quickstart/kubernetes.md. To quickly try without building docker images, I have uploaded pre-built image on docker hub `moon/zeppelin:0.9.0-SNAPSHOT`, `moon/spark:2.4.0`. Try following command ``` ZEPPELIN_SERVER_YAML="curl -s https://raw.githubusercontent.com/Leemoonsoo/zeppelin/kubernetes/k8s/zeppelin-server.yaml" $ZEPPELIN_SERVER_YAML | sed 's/apache\/zeppelin:0.9.0-SNAPSHOT/moon\/zeppelin:0.9.0-SNAPSHOT/' | sed 's/spark:2.4.0/moon\/spark:2.4.0/' | kubectl apply -f - ``` And port forward ``` kubectl port-forward zeppelin-server 8080:80 ``` And browse http://localhost:8080 To clean up ``` $ZEPPELIN_SERVER_YAML | sed 's/apache\/zeppelin:0.9.0-SNAPSHOT/moon\/zeppelin:0.9.0-SNAPSHOT/' | sed 's/spark:2.4.0/moon\/spark:2.4.0/' | kubectl delete -f - ``` ### Screenshots (if appropriate) See this video https://youtu.be/7E4ZGn4pnTo ### Future work - Per interpreter docker image - Blocking communication between interpreter Pod. - Spark Interpreter Pod has Role CRUD for any pod/service in the same namespace. Which should be restricted to only Spark executors Pod. - Per note interpreter mode by default when Zeppelin is running on Kubernetes ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Lee moon soo <leemoonsoo@gmail.com> Author: Lee moon soo <moon@apache.org> Closes #3240 from Leemoonsoo/kubernetes and squashes the following commits: |
||
|
Maxim Belousov
|
0ca247b1b8 |
[Docs] Update java version to 1.8
### What is this PR for? This continues https://github.com/apache/zeppelin/pull/3029 (ZEPPELIN-621 - Bump Java version to Java 8) ### What type of PR is it? Bug Fix ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Maxim Belousov <mebelousov@ya.ru> Closes #3048 from mebelousov/java1.8doc and squashes the following commits: |
||
|
1ambda
|
1c23f21388 |
[ZEPPELIN-2707][DOCS][HOTFIX] fix: broken image URLs in 0.8.0-SNAPSHOT doc
### What is this PR for?
fix: broken image URLs in 0.8.0-SNAPSHOT doc
using the path `/asset` (the absolute path) for image URLs is actually invalid. That's because each version has its own image directory. So they should use the relative path. `{{BASE_PATH}}`
```
➜ asf-zeppelin tree site | grep asset
├── assets # root asset, we shouldn't use it in versioned doc.
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ │ ├── assets
│ ├── assets
```
### What type of PR is it?
[Bug Fix]
### Todos
DONE
### What is the Jira issue?
[ZEPPELIN-2707](https://issues.apache.org/jira/browse/ZEPPELIN-2707)
### How should this be tested?
1. cd `docs/`
2. build: `bundle exec jekyll build --safe`
3. check whether links in `_site` include `/docs/0.8.0-SNAPSHOT` as prefix or not
### Screenshots (if appropriate)
#### Current
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/interpreter/overview.html
#### After
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes #2450 from 1ambda/ZEPPELIN-2707/should-use-its-own-asset-directory and squashes the following commits:
|
||
|
1ambda
|
4b6d3e5574 |
[ZEPPELIN-2596] Improving documentation page
### What is this PR for? Improving documentation page. Please check *TODO* and *Screenshots* sections for detail. The motivation is described in [the JIRA ticket](https://issues.apache.org/jira/browse/ZEPPELIN-2583) and discussion is ongoing on the mailing list. ### What type of PR is it? [Improvement | Documentation] ### Todos * [x] - improved the navbar style * [x] - improved the main page * [x] - re-organized content structure * [x] - added tutorial pages: `spark_with_zeppelin.md`, `python_with_zeppelin.md`, `sql_with_zeppelin.md` for overview * [x] - added `multi_user_support.md` page to provide overview * [x] - added the empty `interpreter_binding_mode` page. This will be handed in the different issue: [ZEPPELIN-2582](https://issues.apache.org/jira/browse/ZEPPELIN-2582) * [x] - added the empty `trouble_shooting` page. This can be filled in the following PRs. * [x] - added the empty `useful_developer_tools` page. This can be filled in the following PRs. ### What is the Jira issue? [ZEPPELIN-2596](https://issues.apache.org/jira/browse/ZEPPELIN-2596) ### How should this be tested? 1. checkout 2. `cd docs` 3. `bundle install` (make sure that you have ruby 2.1.0+ and bundle) 4. `bundle exec jekyll serve --watch` 5. open `localhost:4000` ### Screenshots (if appropriate) #### better navbar: before  #### better navbar: after  #### improved main page: before  #### improved main page: after  #### organized content structure: before  #### organized content structure: after  ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - related with docs Author: 1ambda <1amb4a@gmail.com> Closes #2371 from 1ambda/updating-version-doc and squashes the following commits: |
||
|
Michael Ressler
|
e4f399f1b0 |
Some grammar updates.
### What is this PR for?
Just noticed some grammar inconsistencies that I thought I could clean up while getting myself started on Zeppelin. Thanks for a great project!
### What type of PR is it?
Documentation
Author: Michael Ressler <mressler@diamondkinetics.com>
Closes #2308 from mressler/master and squashes the following commits:
|
||
|
AhyoungRyu
|
c87fa53a3a |
[ZEPPELIN-2298] Remove -Ppyspark build profile
### What is this PR for? Currently users who build Zeppelin from source need to include `-Ppyspark` to use `%pyspark` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Ppyspark` and make pyspark related libraries automatically downloaded when we build Zeppelin. ### What type of PR is it? Improvement ### Todos * [x] - remove the rest of `-Ppyspark` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after getting feedback ### What is the Jira issue? [ZEPPELIN-2298](https://issues.apache.org/jira/browse/ZEPPELIN-2298) ### How should this be tested? 1. Apply this patch 2. Build source with below command ``` mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies' ``` Aftr this step, there will be `pyspark` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Ppyspark` build profile. 4. Restart Zeppelin. To make sure, run any python code e.g. ``` %pyspark print("Hello "+z.input("name")) ``` It should be run successfully without any error ### Screenshots (if appropriate) tl;dr Without `-Ppyspark` profile - Before <img width="856" alt="screen shot 2017-04-02 at 2 50 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584778/0e8ec6b0-17b4-11e7-9f0d-f2599fd7bd63.png"> - After <img width="893" alt="screen shot 2017-04-02 at 2 28 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584779/10b7ed68-17b4-11e7-90d4-aa95eb9bba2d.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no I want to include `SparkR` by default(= remove `-PsparkR` build profile) like this as a next step. I want to ask how Zeppelin community think about this. Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #2213 from AhyoungRyu/ZEPPELIN-2298/includePysparkByDefault and squashes the following commits: |
||
|
rawkintrevo
|
a648746b50 |
[ZEPPELIN-1798][DOCS] Update docs with benv instead of env in Flink e…
### What is this PR for?
Several Flink examples reference `env` instead of `benv` which was changed per [ZEPPELIN-1461](https://github.com/apache/zeppelin/pull/1409)
Also update [http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/quickstart/install_with_flink_and_spark_cluster.html] to reference updated Flink Version
### What type of PR is it?
Documentation
### Todos
* [x] - Update https://zeppelin.apache.org/docs/0.6.2/interpreter/flink.html
* [x] - Update http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/quickstart/install_with_flink_and_spark_cluster.html
### What is the Jira issue?
[ZEPPELIN-1798](https://issues.apache.org/jira/browse/ZEPPELIN-1798)
### How should this be tested?
No tests
### Screenshots (if appropriate)
n/a
### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
This IS documentation that should have been in earlier PR
Author: rawkintrevo <trevor.d.grant@gmail.com>
Closes #1772 from rawkintrevo/ZEPPELIN-1798 and squashes the following commits:
|
||
|
oganizang
|
575467055f |
Add missing TOC(Table of Contents) in install_with_flink_and_spark_cluster.md
### What is this PR for?
Add missing TOC(Table of Contents) in install_with_flink_and_spark_cluster.md
### What type of PR is it?
Documentation
### What is the Jira issue?
no Jira issue for this
Author: oganizang <jangow77@gmail.com>
Closes #1543 from oganizang/master and squashes the following commits:
|
||
|
Damien CORNEAU
|
15a6ee5208 |
Change path to tutorial images
### What is this PR for?
Fixing the image path of the tutorial made in #418
### What type of PR is it?
Hot Fix
### What is the Jira issue?
None, just a Hotfix
### How should this be tested?
Build, move into full website, you should see the images
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Damien CORNEAU <corneadoug@gmail.com>
Closes #1474 from corneadoug/fix/FlinkTutorialImg and squashes the following commits:
|
||
|
rawkintrevo
|
523868ed8d |
[DOCS] Zeppelin Flink Spark tutorial
Moon recommended I migrate a How To tutorial I wrote into the website.
http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-users/201511.mbox/browser
I didn't create a JIRA issue, it doesn't look like that is nessicary for website updates? If it is, I'm sorry I'll go make one. Sorry, I'm new.
Author: rawkintrevo <trevor.d.grant@gmail.com>
Closes #418 from rawkintrevo/zeppelin-flink-spark-tutorial and squashes the following commits:
|
||
|
AhyoungRyu
|
85d4df4f0c |
[ZEPPELIN-1219] Add searching feature to Zeppelin docs site
### What is this PR for? As more and more document pages are added, it's really hard to find specific pages. So I added searching feature to Zeppelin documentation site([jekyll](https://jekyllrb.com/) based site) using [lunr.js](http://lunrjs.com/). - **How does it work?** I created [`search_data.json`]( |
||
|
AhyoungRyu
|
09f0ebda9b |
[DOC] Update doc image in Explore Zeppelin UI page
### What is this PR for? There were several changes in Zeppelin UI after #860, #1006, #1013, #1081 so update screenshot of documents accordingly. ### What type of PR is it? Documentation ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Author: Mina Lee <mina.hyeji.lee@gmail.com> Author: Mina Lee <minalee@apache.org> Closes #1089 from minahlee/doc/ZEPPELIN-1002 and squashes the following commits: |
||
|
AhyoungRyu
|
5975125f18 |
[ZEPPELIN-1018] Apply auto "Table of Contents" generator to Zeppelin docs website
### What is this PR for? I added auto TOC(Table of Contents) generator for Zeppelin documentation website. TOC can help people looking through whole contents at a glance and finding what they want quickly. I just added `<div id="toc"></div>` to the each documentation header. [`toc`](https://github.com/apache/zeppelin/compare/master...AhyoungRyu:ZEPPELIN-1018?expand=1#diff-85af09fb498a5667ea455391533f945dR3) recognize `<h2>` & `<h3>` as a title in the docs and it automatically generate TOC. So I set a rule for this work. (I'll write this rule on `docs/CONTRIBUTING.md` or [docs/howtocontributewebsite](https://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/development/howtocontributewebsite.html)). ``` # Level-1 Heading <- Use only for the main title of the page ## Level-2 Heading <- Start with this one ### Level-3 heading <- Only use this one for child of Level-2 toc only recognize Level-2 & Level-3 ``` Please see the below attached screenshot image. ### What type of PR is it? Improvement & Documentation ### Todos * [x] - Add TOC generator * [x] - Apply TOC(`<div id="toc"></div>`) to every documentation and reorganize each headers(apply the above rule) * [x] - Fix some broken code block in several docs * [x] - Apply TOC to `r.md` (Currently R docs has some duplicated info since [this one]( |
||
|
Mina Lee
|
70ab1a376d |
[ZEPPELIN-952] Refine website style
### What is this PR for? - update document style (font, line-spacing) - apply same formats for documents - fix broke document styles ### What type of PR is it? Documentation ### What is the Jira issue? [ZEPPELIN-952](https://issues.apache.org/jira/browse/ZEPPELIN-952) ### Screenshots (if appropriate) **Before** <img width="1184" alt="screen shot 2016-06-04 at 9 51 38 pm" src="https://cloud.githubusercontent.com/assets/8503346/15803667/d0dd5ac2-2a9f-11e6-9ed0-ddc369a97612.png"> **After** <img width="1184" alt="screen shot 2016-06-04 at 9 15 08 pm" src="https://cloud.githubusercontent.com/assets/8503346/15803666/cd9212ea-2a9f-11e6-986e-17992a495ab6.png"> **Before** <img width="1183" alt="screen shot 2016-06-04 at 10 08 53 pm" src="https://cloud.githubusercontent.com/assets/8503346/15803695/03e73126-2aa1-11e6-8675-3ca437aeb833.png"> **After** <img width="1184" alt="screen shot 2016-06-04 at 10 08 18 pm" src="https://cloud.githubusercontent.com/assets/8503346/15803696/078ce866-2aa1-11e6-9044-4f5e16649eb4.png"> **Before** <img width="1184" alt="screen shot 2016-06-04 at 10 10 47 pm" src="https://cloud.githubusercontent.com/assets/8503346/15803704/5787e9ba-2aa1-11e6-804c-076a8f3aa852.png"> **After** <img width="1184" alt="screen shot 2016-06-04 at 10 11 22 pm" src="https://cloud.githubusercontent.com/assets/8503346/15803707/5afb5d0c-2aa1-11e6-98c7-7440db35bd2f.png"> **Before** <img width="188" alt="screen shot 2016-06-04 at 10 12 36 pm" src="https://cloud.githubusercontent.com/assets/8503346/15803719/92e5cc3e-2aa1-11e6-9a9f-e12150e78733.png"> **After** <img width="199" alt="screen shot 2016-06-04 at 10 12 55 pm" src="https://cloud.githubusercontent.com/assets/8503346/15803721/958e8c00-2aa1-11e6-8768-8350db6e7173.png"> ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Mina Lee <minalee@nflabs.com> Closes #962 from minahlee/ZEPPELIN-952 and squashes the following commits: |
