### What is this PR for?
Here's a possible systemd unit file to control the Zeppelin Java process using systemd commands. Typically on a Ubuntu Linux server for instance. I've also written a very short README to get noobs started.
### What type of PR is it?
Feature
### Todos
* Review code.
* Merge.
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4015
### How should this be tested?
* Install systemd unit file.
* Test.
* Report back.
### Screenshots (if appropriate)
* None required.
### Questions:
* Does the licenses files need update? No.
* Is there breaking changes for older versions? No.
* Does this needs documentation? Yes. Written in a README file for the time being. Might need to write a lengthy documentation page if contribution is accepted.
Author: Patrice Clement <monsieurp@gentoo.org>
Closes#3313 from monsieurp/ZEPPELIN-4015 and squashes the following commits:
6ebfe5596 [Patrice Clement] ZEPPELIN-4015: add a systemd unit file to launch the Zeppelin daemon via systemd commands.
### What is this PR for?
Configure IPython interpreter in Docker image by installing necessary dependencies.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3989
### How should this be tested?
Build docker image and see if %python interpreter works with Ipython kernel.
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <moon@apache.org>
Closes#3303 from Leemoonsoo/ZEPPELIN-3989 and squashes the following commits:
e64ac428e [Lee moon soo] install ipython interpreter dependencies
### What type of PR is it?
This PR adds ability to run Zeppelin on Kubernetes. It aims
- Zero configuration to start Zeppelin on Kubernetes. (and Spark on Kubernetes)
- Run everything on Kubernetes: Zeppelin, Interpreters, Spark.
- Highly customizable to adopt various user configurations and extensions.
Key features are
- Provides zeppelin-server.yaml file for `kubectl` to run Zeppelin server
- All interpreters are automatically running as a Pod.
- Spark interpreter automatically configured to use [Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html)
- Reverse proxy is configured to access Spark UI
To do
- [x] Document how reverse proxy for Spark UI works and how to configure custom domain.
- [x] Document how to customize zeppelin-server and interpreter yaml.
- [x] Document new configurations
- [x] Document how to mount volume for notebook and configurations
### How it works
#### Run Zeppelin Server on Kubernetes
`k8s/zeppelin-server.yaml` is provided to run Zeppelin Server with few sidecars and configurations.
This file is easy to publish (user can easily consume it using `curl`), highly customizable while it includes all the necessary things.
#### K8s Interpreter launcher
This PR adds new module, `launcher-k8s-standard` under `zeppelin/zeppelin-plugins/launcher/k8s-standard/` directory. This launcher is [automatically being selected](https://github.com/apache/zeppelin/pull/3240/files#diff-82fddd2ffb77aaffc4b9cf7b5b1eaa79) when Zeppelin is running on Kubernetes. The launcher both handles Spark interpreter and All other interpreters.
The launcher launches interpreter as a Pod using template [k8s/interpreter/100-interpreter-pod.yaml](https://github.com/apache/zeppelin/pull/3240/files#diff-d9ce62e2c992d32f0184d7edb862f3c4).
Reason filename has `100-` in prefix is because all files in the directory is consumed in alphabetical order by launcher on interpreter start/stop. User can drop more files here to extend/customize interpreter, and filename can be used to control order. The template is rendered by [jinjava](https://github.com/HubSpot/jinjava).
#### Spark interpreter
When interpreter group is `spark`, K8sRemoteInterpreterProcess [sets necessary spark configuration](https://github.com/apache/zeppelin/pull/3240/files#diff-6d1d3084f55bdd519e39ede4a619e73dR297) automatically to use [Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html). User doesn't have to configure anything. It uses client mode.
#### Spark UI
We may make user manually configure port-forward or do something to access Spark UI, but that's not optimal. It is the best when Spark UI is automatically accessible when user have access to Zeppelin UI, without any extra configuration.
To enable this, Zeppelin server Pod has a reverse proxy as a sidecar, and it split traffic to Zeppelin server and Spark UI running in the other Pod. It assume both `service.domain.com` and `*.service.domain.com` point the nginx proxy address. `service.domain.com` is directed to ZeppelinServer, `*.service.domain.com` is directed to interpreter Pod.
`<port>-<interpreter pod svc name>.service.domain.com` is convention to access any application running in interpreter Pod. If Spark interpreter Pod is running with a name `spark-axefeg` and Spark UI is running on port 4040,
```
4040-spark-axefeg.service.domain.com
```
is the address to access Spark UI. Default service domain is [local.zeppelin-project.org:8080](https://github.com/apache/zeppelin/pull/3240/files#diff-56ccb2e2c2617b27dbaae866d9431e51R22), while `local.zeppelin-project.org` and `*.local.zeppelin-project.org` point `127.0.0.1`, and it works with `kubectl port-forward`.
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3840
### How should this be tested?
Prepare a Kubernetes cluster with enough resources (cpus > 5, mem > 6g).
If you're using [minikube](https://github.com/kubernetes/minikube), check your capacity using `kubectl describe node` command before start.
You'll need to build Zeppelin docker image and Spark docker image to test. Please follow guide docs/quickstart/kubernetes.md.
To quickly try without building docker images, I have uploaded pre-built image on docker hub `moon/zeppelin:0.9.0-SNAPSHOT`, `moon/spark:2.4.0`. Try following command
```
ZEPPELIN_SERVER_YAML="curl -s https://raw.githubusercontent.com/Leemoonsoo/zeppelin/kubernetes/k8s/zeppelin-server.yaml"
$ZEPPELIN_SERVER_YAML | sed 's/apache\/zeppelin:0.9.0-SNAPSHOT/moon\/zeppelin:0.9.0-SNAPSHOT/' | sed 's/spark:2.4.0/moon\/spark:2.4.0/' | kubectl apply -f -
```
And port forward
```
kubectl port-forward zeppelin-server 8080:80
```
And browse http://localhost:8080
To clean up
```
$ZEPPELIN_SERVER_YAML | sed 's/apache\/zeppelin:0.9.0-SNAPSHOT/moon\/zeppelin:0.9.0-SNAPSHOT/' | sed 's/spark:2.4.0/moon\/spark:2.4.0/' | kubectl delete -f -
```
### Screenshots (if appropriate)
See this video https://youtu.be/7E4ZGn4pnTo
### Future work
- Per interpreter docker image
- Blocking communication between interpreter Pod.
- Spark Interpreter Pod has Role CRUD for any pod/service in the same namespace. Which should be restricted to only Spark executors Pod.
- Per note interpreter mode by default when Zeppelin is running on Kubernetes
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <leemoonsoo@gmail.com>
Author: Lee moon soo <moon@apache.org>
Closes#3240 from Leemoonsoo/kubernetes and squashes the following commits:
0100a36f2 [Lee moon soo] update how it works on docs, add some comments on yaml files
423412a93 [Lee moon soo] zeppelin.k8s.mode -> zeppelin.run.mode
4e7d8170d [Lee moon soo] localtest.me -> local.zeppelin-project.org
993a0e44e [Lee moon soo] document configurations
9ab6fc420 [Lee moon soo] address code review
22e090f61 [Lee moon soo] logger -> LOGGER
11960dd59 [Lee moon soo] update corresponding test as well
3b652a48e [Lee moon soo] Make spark executor set ownerreference correctly
1a3a07098 [Lee moon soo] Set ownerreference to Role and Rolebinding of interpreter
e2dc88a19 [Lee moon soo] suppress error log when wait target is already removed
fa36c18e3 [Lee moon soo] Make spark master configurable
b4f58a9a1 [Lee moon soo] sig term for quick termination
64a56b5c9 [Lee moon soo] Add docs
e9ce64fe7 [Lee moon soo] update dockerfile
ec09b8b88 [Lee moon soo] add test
3078bac55 [Lee moon soo] spark ui support
9341fcbfe [Lee moon soo] install kubectl and configure log4j in docker image
0f7c0d4e8 [Lee moon soo] add license
f30561189 [Lee moon soo] rename file
2b579ff12 [Lee moon soo] let user override namespace
f4166ad04 [Lee moon soo] make spark container image configurable
0d472ea52 [Lee moon soo] load properties and environment variables
b0e2c36c6 [Lee moon soo] Rbac role, rolebinding
2960dcb87 [Lee moon soo] configure namespace
a4072e6b9 [Lee moon soo] add signal handler
7a8736756 [Lee moon soo] configure spark on kubernetes
263d859d4 [Lee moon soo] use headless service for interpreter pod
7fe9823b1 [Lee moon soo] interpreter pod cascade delete on zeppelin-server delete
86e876435 [Lee moon soo] add services on RBAC
18b8f68cb [Lee moon soo] print spec file contents on debug log
0dea3836b [Lee moon soo] create and connect interpreter pod
9f1b7a169 [Lee moon soo] run kubernetes launcher
2fd2ac8c3 [Lee moon soo] kubernetes mode configuration
58f9f1909 [Lee moon soo] add rbac
36cf391a4 [Lee moon soo] correct plugin name
52bb6c7e1 [Lee moon soo] add k8s dir in package
5f602a65e [Lee moon soo] K8sRemoteInterpreterProcess
07489f76d [Lee moon soo] kubectl with exec
d2f3d5b7e [Lee moon soo] add k8s-standard launcher module
### What is this PR for?
This PR switches the project to use Java 8 - we already have dependency on Spark 2.2.0 that is Java 8+ only.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-621
Author: Alex Ott <alexott@gmail.com>
Closes#3029 from alexott/ZEPPELIN-621 and squashes the following commits:
a2e0cd8ea [Alex Ott] next try to make tests working
52a41d033 [Alex Ott] explicitly specify profiles for Scala 2.10 for some tests
7dc92d972 [Alex Ott] Merge branch 'ZEPPELIN-621' of github.com:alexott/zeppelin into ZEPPELIN-621
4d5113225 [Alex Ott] switch to Java 8 in the TravisCI as well
8e14ddb92 [Alex Ott] Mention Java 8 as dependency
3abd1425e [Alex Ott] update Vagrant to 16.04 + Java 8
fc69773cf [Alex Ott] bump Java to 1.8
cce03d184 [Alex Ott] switch to Java 8 in the TravisCI as well
51b9eccfb [Alex Ott] Mention Java 8 as dependency
764bd3417 [Alex Ott] update Vagrant to 16.04 + Java 8
07e80bb21 [Alex Ott] bump Java to 1.8
### What is this PR for?
trivial pom file change
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3192
### How should this be tested?
* Travis pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjffdu@apache.org>
Closes#2748 from zjffdu/ZEPPELIN-3192 and squashes the following commits:
635acde [Jeff Zhang] ZEPPELIN-3192. Bump up Zeppelin version to 0.9.0-SNAPSHOT
### What is this PR for?
The PR updates Spark version to the current 2.1.2, since the one which is present now (2.1.1) is not available anymore.
### What type of PR is it?
Fix
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3017
### How should this be tested?
manually
Author: mark91 <marcogaido91@gmail.com>
Closes#2636 from mgaido91/ZEPPELIN-3017 and squashes the following commits:
b3cfb44 [mark91] [ZEPPELIN-3017] fix Spark version in Dockerfiles
### What is this PR for?
Using OpenJDK at distributing docker image will reduce legal threats.
### What type of PR is it?
Bug Fix
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2861
### How should this be tested?
`docker build scripts/docker/zeppelin/bin`
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? **NO**
* Is there breaking changes for older versions? **NO**
* Does this needs documentation? **NO**
Author: Jinkyu Yi <jincreator@jincreator.net>
Closes#2536 from jincreator/ZEPPELIN-2861 and squashes the following commits:
3b4fbcb [Jinkyu Yi] [ZEPPELIN-2861] Use OpenJDK in docker image.
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
### What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
Outline the steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: sven0726 <sven0726@gmail.com>
Closes#2370 from sven0726/master and squashes the following commits:
15af90e [sven0726] change hadoop profile version
3b68a5f [sven0726] change the spark version of dockerfile
### What is this PR for?
Use single `Dockerfile` for each release since [apache infra uses tag pushes](https://issues.apache.org/jira/browse/INFRA-12781) to build an image.
- https://issues.apache.org/jira/browse/INFRA-12781
After release process finishes, dockerhub will build using the pushed tag.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
[ZEPPELIN-2492](https://issues.apache.org/jira/browse/ZEPPELIN-2492)
### How should this be tested?
1. `./dev/change_zeppelin_version.sh 0.8.0-SNAPSHOT 0.7.1`
2. Check that version is properly set: `vi scrtips/docker/zeppelin/bin/Dockerfile`
3. Build docker image `cd scripts/docker/zeppelin/bin; docker build -t zeppelin:0.7.1 ./`
4. Run the image: `docker run -p 8080:8080 --rm --name zeppelin zeppelin:0.7.1`
### Screenshots (if appropriate)
NONE
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO
Author: 1ambda <1amb4a@gmail.com>
Closes#2318 from 1ambda/ZEPPELIN-2492/use-single-dockerfile-for-each-tag and squashes the following commits:
483bec3 [1ambda] docs: Update README for Dockerfile
5826c8c [1ambda] fix: Use single dockerfile for tag push
### What is this PR for?
Created `Dockerfile` for released bin
- based on **Ubuntu:16.04 (LTS)** for desktop usage
- **JDK 8**
- **R** with basic packages
- **Python 2** with basic packages
- **miniconda2** for `%python.conda`
### Details
We already discussed about using alpine image in https://github.com/apache/zeppelin/pull/1761.
- However, it's not designed for desktop usage
- Doesn't have some official packages (R, ...)
- Not familiar to users for desktop OS
That the reason why ubuntu is used in base image
```
zeppelin base b3818f9ae4b1 11 hours ago 1.67 GB
zeppelin 0.6.2 c0a4d8556f92 7 hours ago 2.29 GB
zeppelin 0.7.0 c4a5ad0d04bd 8 hours ago 2.5 GB
zeppelin 0.7.1 54173b77743b 7 hours ago 2.49 GB
```
### What type of PR is it?
[Feature]
### Todos
* [x] - base image
* [x] - script for creating bin images
* [x] - bin image template
### What is the Jira issue?
[ZEPPELIN-1711](https://issues.apache.org/jira/browse/ZEPPELIN-1711)
### How should this be tested?
1. build base image `cd scripts/docker/zeppelin/base; docker build -t zeppelin:base ./`
2. build bin image `cd scripts/docker/zeppelin/0.7.1; docker build -t zeppelin:0.7.1 ./`
3. execute docker images
```
docker run -p 8080:8080 --rm --name zeppelin zeppelin:0.7.1
```
since it takes time to build, you can use already [published docker images](https://hub.docker.com/r/1ambda/docker-zeppelin/)
```
docker run -p 8080:8080 --rm --name zeppelin 1ambda/docker-zeppelin:0.7.1
```
4. should be able to run spark, python and R tutorials
### Screenshots (if appropriate)
NO
### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - YES, updated
Author: 1ambda <1amb4a@gmail.com>
Closes#2264 from 1ambda/ZEPPELIN-1711/bin-dockerfile and squashes the following commits:
69a0b1f [1ambda] docs: Update docker.md
ced897f [1ambda] fix: DON'T remove /tmp
1f6da76 [1ambda] feat: Dockerfiles for 060, 070, 071
0fc3f75 [1ambda] feat: Add template for bin image
5cba56e [1ambda] feat: Use ubuntu for base image
### What is this PR for?
Currently users who build Zeppelin from source need to include `-Psparkr` to use `%r` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Psparkr` and make `r` related libraries automatically downloaded when we build Zeppelin like I did #2213
### What type of PR is it?
Improvement
### Todos
* [x] - remove the rest of `-Psparkr` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after #2213 merged
### What is the Jira issue?
[ZEPPELIN-2341](https://issues.apache.org/jira/browse/ZEPPELIN-2341)
### How should this be tested?
1. Apply this patch
2. Build source with below command
```
mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies'
```
Aftr this step, there will be `R` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Psparkr` build profile.
4. Restart Zeppelin. To make sure, run R tutorial note under `Zeppelin Tutorial` folder
It should be run successfully without any error
### Screenshots (if appropriate)
If we build without `-Psparkr`
- before : R related properties are not activated by default in Spark interpreter
- after
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Author: Ahyoung Ryu <ahyoungryu@apache.org>
Closes#2215 from AhyoungRyu/ZEPPELIN-2341/includeSparkRByDefault and squashes the following commits:
8db18cc [AhyoungRyu] Remove the rest of '-Psparkr' in docs & sh files
f891fd4 [Ahyoung Ryu] Merge branch 'master' into ZEPPELIN-2341/includeSparkRByDefault
445be3e [AhyoungRyu] Add SPARKR env to check each test case need to download r dep or not
67af02a [AhyoungRyu] Remove -PsparkR in travis file
a00466c [AhyoungRyu] Remove sparkr build profile in pom files
### What is this PR for?
Currently users who build Zeppelin from source need to include `-Ppyspark` to use `%pyspark` with embedded local Spark. But it's quite inconvenient to write this build profile every time we build i think. So I removed `-Ppyspark` and make pyspark related libraries automatically downloaded when we build Zeppelin.
### What type of PR is it?
Improvement
### Todos
* [x] - remove the rest of `-Ppyspark` build profile in `dev/create_release.sh`, `dev/publish_release.sh`, and `docs/install/build.md` after getting feedback
### What is the Jira issue?
[ZEPPELIN-2298](https://issues.apache.org/jira/browse/ZEPPELIN-2298)
### How should this be tested?
1. Apply this patch
2. Build source with below command
```
mvn clean package -DskipTests -pl 'zeppelin-interpreter, zeppelin-zengine, zeppelin-server, zeppelin-display, spark, spark-dependencies'
```
Aftr this step, there will be `pyspark` dir under `ZEPPELIN_HOME/interpreter/spark`. Before this PR, only `dep` dir and `zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar` is generated without `-Ppyspark` build profile.
4. Restart Zeppelin. To make sure, run any python code e.g.
```
%pyspark
print("Hello "+z.input("name"))
```
It should be run successfully without any error
### Screenshots (if appropriate)
tl;dr Without `-Ppyspark` profile
- Before
<img width="856" alt="screen shot 2017-04-02 at 2 50 57 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584778/0e8ec6b0-17b4-11e7-9f0d-f2599fd7bd63.png">
- After
<img width="893" alt="screen shot 2017-04-02 at 2 28 21 pm" src="https://cloud.githubusercontent.com/assets/10060731/24584779/10b7ed68-17b4-11e7-90d4-aa95eb9bba2d.png">
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
I want to include `SparkR` by default(= remove `-PsparkR` build profile) like this as a next step. I want to ask how Zeppelin community think about this.
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Closes#2213 from AhyoungRyu/ZEPPELIN-2298/includePysparkByDefault and squashes the following commits:
f7bcf06 [AhyoungRyu] Remove -Ppyspark in virtual_machine.md
458ac02 [AhyoungRyu] Remove the rest of -Ppyspark in blind side of Zeppelin :)
cee1e87 [AhyoungRyu] Change py4j.version -> python.py4j.version
ce43158 [AhyoungRyu] Change py4j.version -> spark.py4j.version
fa4fb36 [AhyoungRyu] Remove the rest of -Ppyspark
30aac81 [AhyoungRyu] Remove -Ppyspark build flag
### What is this PR for?
This PR is for making docker images for zeppelin releases. It contains a script for building image for each release. Another script is used for publishing images to zeppelin Dockerhub account.
This repo, https://github.com/mfelgamal/zeppelin-dockers, is a demonstration of this PR. It contains zeppelin-base image and an image for each zeppelin release.
### What type of PR is it?
[Feature]
### Todos
- Review Comments
- Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1386
### How should this be tested?
- run create_release script or publish_release script.
### Screenshots (if appropriate)
### Questions:
- Does the licenses files need update? no
- Is there breaking changes for older versions? no
- Does this needs documentation? yes
Author: mahmoudelgamal <mahmoudf.elgamal@gmail.com>
Author: mfelgamal <mahmoudf.elgamal@gmail.com>
Author: Mahmoud Elgamal <mahmoudf.elgamal@gmail.com>
Author: 1ambda <1amb4a@gmail.com>
Closes#1538 from mfelgamal/zeppelin-dockers and squashes the following commits:
cc8493f [Mahmoud Elgamal] Merge pull request #3 from 1ambda/fix/remove-startzeppelinsh
d48ecef [1ambda] fix: Remove start-zeppelin.sh
b64c680 [mahmoudelgamal] Remove gcc and g++ for decreasing the size
1f093d4 [mahmoudelgamal] Add script start-zeppelin to zeppelin-base
d2c744e [mahmoudelgamal] add scala to zeppelin-base
fd23970 [mahmoudelgamal] remove bash erorr message.
e1d4b77 [mahmoudelgamal] add R and python to zeppelin-base
e731cb4 [mahmoudelgamal] Add java-cacerts to zeppelin-base
e642309 [mahmoudelgamal] Add documentation and some modifications
231a414 [mahmoudelgamal] Add zeppelin-base image
ac06f3a [mahmoudelgamal] Make docker image for zeppelin release
48d0a01 [mfelgamal] Merge pull request #1 from apache/master
### What is this PR for?
This PR adds Mahout functionality for the Spark Interpreter.
### What type of PR is it?
Improvement
### Todos
- [x] Implement Mahout Interpreter in Spark
- [x] Add Unit Tests
- [x] Add Documentation
- [x] Add Example Notebook
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-116
### How should this be tested?
Open a Spark Notebook with Mahout enabled and run a few simple commands using the R-Like DSL and Spark Distributed Context (Mahout Specific)
### Screenshots (if appropriate)
### Questions:
- Does the licenses files need update?
No
- Is there breaking changes for older versions?
No
- Does this needs documentation?
Yes
Author: rawkintrevo <trevor.d.grant@gmail.com>
Closes#928 from rawkintrevo/mahout-terp and squashes the following commits:
ed6eff0 [rawkintrevo] [ZEPPELIN-116] renamed add_mahout_interpreters.py and overwrite_existing feature
e7d4e12 [rawkintrevo] [ZEPPELIN-116] Made add_mahout.py script more resilient
7e83832 [rawkintrevo] [ZEPPELIN-116] Add Mahout Interpreters
### What is this PR for?
This PR fixes wrong written NotebookID to NoteID.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1549
### Questions:
* Does the licenses files need update? No.
* Is there breaking changes for older versions? No.
* Does this needs documentation? No
Author: hyonzin <hyeonjin507@gmail.com>
Author: 정현진 <hyeonjin507@gmail.com>
Author: Mina Lee <minalee@apache.org>
Closes#1518 from hyonzin/ZEPPELIN-1549 and squashes the following commits:
2c5d461 [hyonzin] fix pullNoteID to pullNoteId
f843abd [hyonzin] Fix missed line
22aecb3 [hyonzin] Merge branch 'master' of https://github.com/apache/zeppelin into ZEPPELIN-1549
ac03666 [정현진] Merge pull request #1 from minahlee/ZEPPELIN-1549
8b3fffd [Mina Lee] Change notebook to note and fix indentation
000605f [hyonzin] Change clonedNotebookId to clonedNoteId
496695c [hyonzin] Change noteID to noteId
1e87463 [hyonzin] Remove tab indent
5647d37 [hyonzin] Rebase and solve conflicts
09bacd8 [hyonzin] Fix more lines unchanged
070bc2d [hyonzin] fix more in ZeppelinRestApiTest.java
24822a3 [hyonzin] Fix more code not changed (notebookIndex to noteSearchService)
4b4e1e8 [hyonzin] Fix detail (function's name) & Change some placeholder
429203d [hyonzin] Fix details & convention to camel
5fa270d [hyonzin] pull upstream master & fix some details
294bea5 [hyonzin] Fix some wrong written term: Notebook -> Note
cc0d315 [hyonzin] Change NotebookID variable name to NoteID
### What is this PR for?
This PR is for the documentation of running zeppelin with CDH docker environment.
and This PR is the part of https://issues.apache.org/jira/browse/ZEPPELIN-1198.
Tested CDH5.7 on ubuntu.
### What type of PR is it?
Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1281
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@nflabs.com>
Author: AhyoungRyu <ahyoungryu@apache.org>
Author: HyungSung <hsshim@nflabs.com>
Closes#1451 from astroshim/ZEPPELIN-1281 and squashes the following commits:
5dcb8c1 [astroshim] move configurations to right path and add excluding rat-plugin
09408e3 [HyungSung] Merge pull request #11 from AhyoungRyu/ZEPPELIN-1281-ahyoung
850119c [AhyoungRyu] Generate TOC & change some sentences
e687a53 [AhyoungRyu] Replace zeppelin_with_cdh.png to crop the url part
cc9a023 [AhyoungRyu] Remove main title link anchor
b525f68 [astroshim] separate cdh doc with spark_cluster_mode.md
e66993f [astroshim] fix doc
a7b5b2d [astroshim] cdh docker environment
### What is this PR for?
This PR is for the documentation of running zeppelin on production environments especially spark on mesos via Docker.
Related issue is https://github.com/apache/zeppelin/pull/1227 and https://github.com/apache/zeppelin/pull/1318 and I got a lot of hints from https://github.com/sequenceiq/hadoop-docker.
Tested on ubuntu.
### What type of PR is it?
Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1279
### How should this be tested?
You can refer to https://github.com/apache/zeppelin/blob/master/docs/README.md#build-documentation.
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@nflabs.com>
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Author: HyungSung <hsshim@nflabs.com>
Closes#1389 from astroshim/ZEPPELIN-1279 and squashes the following commits:
974366a [HyungSung] Merge pull request #10 from AhyoungRyu/ZEPPELIN-1279-ahyoung
076fdba [AhyoungRyu] Change zeppelin_mesos_conf.png file
1cbe9d3 [astroshim] fix spark version and mesos
2b821b4 [astroshim] fix docs
159bafc [astroshim] fix anchor
d8c43b4 [astroshim] add navigation
c808350 [astroshim] add image file and doc
a3b0ded [astroshim] create dockerfile for mesos
### What is this PR for?
This PR is for the documentation of running zeppelin on production environments especially spark on yarn.
Related issue is https://github.com/apache/zeppelin/pull/1227 and I got a lot of hints from https://github.com/sequenceiq/hadoop-docker.
Tested on ubuntu.
### What type of PR is it?
Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1280
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@nflabs.com>
Author: AhyoungRyu <fbdkdud93@hanmail.net>
Author: HyungSung <hsshim@nflabs.com>
Closes#1318 from astroshim/ZEPPELIN-1280 and squashes the following commits:
60958cd [astroshim] small changes for doc
6c44b7b [astroshim] Merge branch 'master' into ZEPPELIN-1280
dad297c [astroshim] update version
4c8d72d [astroshim] merge with Ayoung's
8c62cf1 [astroshim] fixed felixcheung pointed out.
86ca513 [HyungSung] Merge pull request #9 from AhyoungRyu/ZEPPELIN-1280-ahyoung
cde5f8d [AhyoungRyu] Modify document description so that this docs can be searched
9e9390c [AhyoungRyu] Minor update for spark_cluster_mode.md
633c930 [astroshim] running zeppelin on yarn
### What is this PR for?
Change maven version from 3.3.3 to 3.3.9 in vagrant script and its documentation due to path to 3.3.3 doesn't exist (return 404 from apache mirror)
### What type of PR is it?
Bug Fix | Documentation
### What is the Jira issue?
ZEPPELIN-1299
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Jesang Yoon <yoonjs2@kanizsalab.com>
Closes#1294 from yoonjs2/ZEPPELIN-1299 and squashes the following commits:
1d2591f [Jesang Yoon] Change all appearances 3.3.3 to 3.3.9 in documentation
5ac7de7 [Jesang Yoon] Change maven version from 3.3.3 to 3.3.9
### What is this PR for?
This PR is for documentation for running zeppelin on production environments.
### What type of PR is it?
Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1198
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: astroshim <hsshim@nflabs.com>
Closes#1227 from astroshim/ZEPPELIN-1198/standalone and squashes the following commits:
53a32f2 [astroshim] add 'via Docker'
61a0e5e [astroshim] add apache license header
83fdef6 [astroshim] doc for spark standalone
### What is this PR for?
This is extension to https://github.com/apache/zeppelin/pull/983, with missing references of incubation.
### What type of PR is it?
[Improvement]
### What is the Jira issue?
[ZEPPELIN-980](https://issues.apache.org/jira/browse/ZEPPELIN-980)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Prabhjyot Singh <prabhjyotsingh@gmail.com>
Closes#992 from prabhjyotsingh/ZEPPELIN-980 and squashes the following commits:
5fa97b7 [Prabhjyot Singh] missing "incubation-" references
### What is this PR for?
Updates Virtual Machine documentation to include SparkR information.
### What type of PR is it?
Documentation
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Steinmetz <jeffrey.steinmetz@gmail.com>
Closes#826 from jeffsteinmetz/sparkr-doc-updates and squashes the following commits:
5d34ffb [Jeff Steinmetz] Include r packages in last paragraph
d8f7827 [Jeff Steinmetz] Include r packages in last paragraph
3f38a5a [Jeff Steinmetz] remove extranious brackets
0fc8a11 [Jeff Steinmetz] add r packages list per comment. use anchor links consistent with markdown documenation per github
2e419fe [Jeff Steinmetz] vagrant download like changed. update to a link that doesn not 404.
d8a1fa4 [Jeff Steinmetz] update VM readme and VM install docs to reference R interpreter additions
679e87a [Jeff Steinmetz] update VM readme and VM install docs to reference R interpreter additions
### What is this PR for?
Improvement of Virtual Machine Script to support R interpretor
### What type of PR is it?
Improvement of Virtual Machine Script to support R interpretor
### Todos
* [x] - Test with #208
* [x] - Test with #702
### Is there a relevant Jira issue?
Zeppelin-700
### How should this be tested?
Follow the steps in this Read Me to build a VM from scratch:
https://github.com/apache/incubator-zeppelin/blob/master/scripts/vagrant/zeppelin-dev/README.md
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? Added to all headers
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes, this will be a separate PR to update docs and README
Author: Jeff Steinmetz <jeffrey.steinmetz@gmail.com>
Closes#751 from jeffsteinmetz/ZEPPELIN-700 and squashes the following commits:
e03dba5 [Jeff Steinmetz] update to support R interpreter sparkr build profile
a9a2052 [Jeff Steinmetz] add base64enc, repr and htmltools
bdcdf5f [Jeff Steinmetz] removed packages not required for pr702. repr not needed - base64encode not needed - htmltools not needed
d497994 [Jeff Steinmetz] fix so that devtools will install
5643fc6 [Jeff Steinmetz] plotting in r interpreter requires the repr package. install devtools package first so repr can be installed.
fb18a2b [Jeff Steinmetz] plotting in r interpreter requires the repr package. install devtools package first so repr can be installed.
ef8f638 [Jeff Steinmetz] ZEPPELIN-700. Add Ansible R role to Virtual Machine to support dependencies for the R Interpreter
b940bcd [Jeff Steinmetz] ZEPPELIN-700. Add Ansible R role to Virtual Machine to support dependencies for the R Interpreter
### What is this PR for?
Lately, an official location of Zeppelin was changed from **https://github.com/apache/incubator-zeppelin.git** to **http://git.apache.org/incubator-zeppelin.git**. So I changed the old location in several documentations to the latest one.
### What type of PR is it?
Documentation
### Todos
### Is there a relevant Jira issue?
[ZEPPELIN-657](https://issues.apache.org/jira/browse/ZEPPELIN-657)
### How should this be tested?
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Ryu Ah young <fbdkdud93@hanmail.net>
Closes#700 from AhyoungRyu/ZEPPELIN-657 and squashes the following commits:
2144bcc [Ryu Ah young] ZEPPELIN-657: Change 'release' -> 'branch' in howtocontribute.md
813b438 [Ryu Ah young] ZEPPELIN-657: Change 'release' -> 'branch'
23646e5 [Ryu Ah young] ZEPPELIN-657: Fix the wrong locations
c4bef6c [Ryu Ah young] ZEPPELIN-657: fix a typo in howtocontribute.md
a5fb130 [Ryu Ah young] ZEPPELIN-657: Change the official location of Zeppelin
Realized I used an older Ubuntu 12 vm box file in Vagrant script.
Should be Ubuntu 14.04
Author: Jeff Steinmetz <jeffrey.steinmetz@gmail.com>
Closes#567 from jeffsteinmetz/vagrant-vm and squashes the following commits:
3b81445 [Jeff Steinmetz] use ubuntu/trusty64 Official Ubuntu Server 14.04 LTS (Trusty Tahr) build
fbaf2bd [Jeff Steinmetz] use ubuntu/trusty64 Official Ubuntu Server 14.04 LTS (Trusty Tahr) build
This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source. For pyspark users, this script also includes several helpful Python Libraries and one obscure configuration to help with matplotlib plotting inside Zeppelin.
The virtual machine consists of:
- Ubuntu Server 14.04 LTS
- Node.js 0.12.7
- npm 2.11.3
- ruby 1.9.3 + bundler, rake, make (only required for building jekyll documentation)
- Maven 3.3.3
- Git
- Unzip
- libfontconfig to avoid phatomJs missing dependency issues
- openjdk-7-jdk
- Python addons: pip, matplotlib, scipy, numpy, pandas
- Changes are made to the python backend configuration so that pyspark can render matplotlib plots without an interactive window
Author: Jeff Steinmetz <jeffrey.steinmetz@gmail.com>
Closes#460 from jeffsteinmetz/master and squashes the following commits:
cfb1034 [Jeff Steinmetz] add ruby 1.9.3 install + bundler, make and rake for jekyll doc support. Add vagrant port 4000 forwarding for jekyll documentation build watcher
9a78355 [Jeff Steinmetz] remove apache python script from commit
97a5819 [Jeff Steinmetz] remove apache python script. remove nodesource and use official node.js distribution download
0b169a6 [Jeff Steinmetz] remove license from apache-mirror-selector.py. Simplify node install
a86c55f [Jeff Steinmetz] typo fixed. licenses in supporting readme and apache mirror python script
3ec6a6b [Jeff Steinmetz] add virtual machine to docs
4b70e63 [Jeff Steinmetz] remove reference to python backend config in readme.
5c31125 [Jeff Steinmetz] add apache-mirror-selector to ansible playbook
11fdc0d [Jeff Steinmetz] removed matplotlib global Agg config. Updated vagrant virtual box settings with hostname and vm name. upated python example to set Agg
afc26a4 [Jeff Steinmetz] roll back to original top level README.md
de07f00 [Jeff Steinmetz] update readme based on comments. Add license
853ddca [Jeff Steinmetz] vagrant virtual machine and readme
ff8078a [Jeff Steinmetz] vagrant virtual machine and readme
fb31785 [Jeff Steinmetz] initial commit of vagrant virtual machine script
5888360 [Jeff Steinmetz] merge upstream
1eb27eb [Jeff Steinmetz] expanded build instructions to include pyspark, and clarified maven and node.js requirements
