Elgato_dark/zeppelin
1
0
Fork 0
mirror of https://github.com/apache/zeppelin synced 2026-05-24 09:38:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

address code review

Browse source
This commit is contained in:
Lee moon soo 2018-11-26 18:42:33 -08:00
parent 22e090f619
commit 9ab6fc4202
4 changed files with 24 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
docs/quickstart/kubernetes.md
View file

@ -21,7 +21,7 @@ limitations under the License.
# Zeppelin on Kubernetes
Zeppelin can run on clusters managed by [Kubernetes](https://kubernetes.io/). When Zeppelin runs in Pod, it creates pods for individual interpreter. Also Spark interpreter auto configured to use Spark on Kubernetes.
Zeppelin can run on clusters managed by [Kubernetes](https://kubernetes.io/). When Zeppelin runs in Pod, it creates pods for individual interpreter. Also Spark interpreter auto configured to use Spark on Kubernetes in client mode.
Key benefits are
@ -165,3 +165,13 @@ $ docker build -t <tag> .
```
Finally, set custom image `<tag>` just created to `image` and `ZEPPELIN_K8S_CONTAINER_IMAGE` env variable of `zeppelin-server` container spec in `zeppelin-server.yaml` file.
Currently, single docker image is being used in both Zeppelin server and Interpreter pods. Therefore,
| Pod | Number of instances | Image | Note |
| --- | --- | --- | --- |
| Zeppelin Server | 1 | Zeppelin docker image | User creates/deletes with kubectl command |
| Zeppelin Interpreters | n | Zeppelin docker image | Zeppelin Server creates/deletes |
| Spark executors | m | Spark docker image | Spark Interpreter creates/deletes |
Currently, size of Zeppelin docker image is quite big. Zeppelin project is planning to provides lightweight images for each individual interpreter in the future.

0
k8s/interpreter/100-interpreter-pod.yaml → k8s/interpreter/100-interpreter-spec.yaml
View file

9
zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
View file

@ -29,7 +29,6 @@ public class K8sRemoteInterpreterProcess extends RemoteInterpreterProcess {
private final Map<String, String> envs;
private final String zeppelinServiceHost;
private final String zeppelinServiceRpcPort;
private final int podCreateTimeoutSec = 180;
private final Gson gson = new Gson();
private final String podName;
@ -53,7 +52,8 @@ public class K8sRemoteInterpreterProcess extends RemoteInterpreterProcess {
String zeppelinServiceRpcPort,
boolean portForward,
String sparkImage,
int connectTimeout) {
int connectTimeout
) {
super(connectTimeout);
this.kubectl = kubectl;
this.specTempaltes = specTemplates;
@ -89,7 +89,7 @@ public class K8sRemoteInterpreterProcess extends RemoteInterpreterProcess {
public void start(String userName) throws IOException {
// create new pod
apply(specTempaltes, false);
kubectl.wait(String.format("pod/%s", getPodName()), "condition=Ready", podCreateTimeoutSec);
kubectl.wait(String.format("pod/%s", getPodName()), "condition=Ready", getConnectTimeout()/1000);
if (portForward) {
podPort = RemoteInterpreterUtils.findRandomAvailablePortOnAllLocalInterfaces();
@ -305,6 +305,9 @@ public class K8sRemoteInterpreterProcess extends RemoteInterpreterProcess {
options.append(" --master k8s://https://kubernetes.default.svc");
options.append(" --deploy-mode client");
if (properties.containsKey("spark.driver.memory")) {
options.append(" --driver-memory " + properties.get("spark.driver.memory"));
}
options.append(" --conf spark.kubernetes.namespace=" + kubectl.getNamespace());
options.append(" --conf spark.executor.instances=1");
options.append(" --conf spark.kubernetes.driver.pod.name=" + getPodName());

7
zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sStandardInterpreterLauncher.java
View file

@ -60,6 +60,13 @@ public class K8sStandardInterpreterLauncher extends InterpreterLauncher {
/**
* Check if i'm running inside of kubernetes or not.
* It should return truth regardless of ZeppelinConfiguration.getK8sMode().
*
* Normally, unless Zeppelin is running on Kubernetes, K8sStandardInterpreterLauncher shouldn't even have initialized.
* However, when ZeppelinConfiguration.getK8sMode() is force 'on', InterpreterSetting.getLauncherPlugin() will try
* to use K8sStandardInterpreterLauncher. This is useful for development. It allows Zeppelin server running on your
* IDE and creates your interpreters in Kubernetes. So any code changes on Zeppelin server or kubernetes yaml spec
* can be applied without re-building docker image.
* @return
*/
boolean isRunningOnKubernetes() {