mirror of
https://github.com/apache/zeppelin
synced 2026-05-24 09:38:26 +00:00
1b9c46dcfd
### What is this PR for? For now we spark configuration at runtime rather than pass them through `--conf`, it would cause several issues. - Some configuration has to be set through --conf, otherwise we need to duplicate code in SparkSubmit.scala (spark.yarn.keytab, spark.yarn.principal) - Some configuration would conflict with spark-defaults.conf. If you specify spark.master as yarn-client in spark-defaults.conf but specify spark.master as local in zeppelin side, you will see the spark interpreter fail to start due to this inconsistency. - As ZEPPELIN-1460 described, it is hard to figure what is the effective configuration. - We can not use yarn-cluster mode although it is not supported now, but I think it is necessary to do that as zeppelin needs to support multiple users. This PR would pass all the spark related configuration to spark-submit through `--conf`, so that it is easy to know and guarantee that configuration on zeppelin interpreter setting take precedence over spark-defaults.conf. And it is also good for maintenance that upstream change (any change about configuration in spark) would not affect us. ### What type of PR is it? [Improvement] ### Todos - [ ] - Task ### What is the Jira issue? - https://issues.apache.org/jira/browse/ZEPPELIN-1263 ### How should this be tested? Tested spark 1.6 spark 2.0 on both yarn-client mode and embedded mode. ### Screenshots (if appropriate)  ### Questions: - Does the licenses files need update? No - Is there breaking changes for older versions? No - Does this needs documentation? No … Author: Jeff Zhang <zjffdu@apache.org> Closes #1446 from zjffdu/ZEPPELIN-1263 and squashes the following commits: |
||
|---|---|---|
| .. | ||
| common.cmd | ||
| common.sh | ||
| functions.cmd | ||
| functions.sh | ||
| install-interpreter.sh | ||
| interpreter.cmd | ||
| interpreter.sh | ||
| zeppelin-daemon.sh | ||
| zeppelin.cmd | ||
| zeppelin.sh | ||
