Elgato_dark/zeppelin
1
0
Fork 0
mirror of https://github.com/apache/zeppelin synced 2026-05-24 09:38:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Initial version of files for 3388

Browse source
This commit is contained in:
Sanjay Dasgupta 2018-04-29 23:58:58 +05:30
parent 468cea2907
commit 64e4d7e1ba
5 changed files with 241 additions and 157 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
docs/_includes/themes/zeppelin/_navigation.html
View file

@ -63,6 +63,7 @@
<li><a href="{{BASE_PATH}}/usage/other_features/customizing_homepage.html">Customizing Zeppelin Homepage</a></li>
<li><a href="{{BASE_PATH}}/usage/other_features/notebook_actions.html">Notebook Actions</a></li>
<li><a href="{{BASE_PATH}}/usage/other_features/cron_scheduler.html">Cron Scheduler</a></li>
<li><a href="{{BASE_PATH}}/usage/other_features/zeppelin_context.html">Zeppelin Context</a></li>
<li role="separator" class="divider"></li>
<li class="title"><span>REST API</span></li>
<li><a href="{{BASE_PATH}}/usage/rest_api/interpreter.html">Interpreter API</a></li>
@ -168,9 +169,9 @@
<li><a href="{{BASE_PATH}}/development/contribution/how_to_contribute_website.html">How to Contribute (website)</a></li>
<li role="separator" class="divider"></li>
<li class="title"><span>External Resources</span></li>
<li><a target="_blank" href="https://zeppelin.apache.org/community.html">Mailing List</a></li>
<li><a target="_blank" href="https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Home">Apache Zeppelin Wiki</a></li>
<li><a target="_blank" href="http://stackoverflow.com/questions/tagged/apache-zeppelin">Stackoverflow Questions about Zeppelin</a></li>
<li><a target="_blank" href="">Mailing List</a></li>
<li><a target="_blank" href="">Apache Zeppelin Wiki</a></li>
<li><a target="_blank" href="">Stackoverflow Questions about Zeppelin</a></li>
</ul>
</li>
<li>
@ -182,3 +183,4 @@
</nav><!--/.navbar-collapse -->
</div>
</div>

2
docs/index.md
View file

@ -74,6 +74,7 @@ limitations under the License.
* [Personalized Mode](./usage/other_features/personalized_mode.html)
* [Customizing Zeppelin Homepage](./usage/other_features/customizing_homepage.html) with one of your notebooks
* [Notebook actions](./usage/other_features/notebook_actions.html)
* [Zeppelin-Context](./usage/other_features/zeppelin_context.html)
* REST API: available REST API list in Apache Zeppelin
* [Interpreter API](./usage/rest_api/interpreter.html)
* [Zeppelin Server API](./usage/rest_api/zeppelin_server.html)
@ -160,3 +161,4 @@ limitations under the License.
* [Mailing List](https://zeppelin.apache.org/community.html)
* [Apache Zeppelin Wiki](https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Home)
* [Stackoverflow Questions about Zeppelin (tag: `apache-zeppelin`)](http://stackoverflow.com/questions/tagged/apache-zeppelin)

4
docs/interpreter/shell.md
View file

@ -107,5 +107,5 @@ val members = spark.read.parquet(z.get("dataFileName"))
Object interpolation is disabled by default, and can be enabled (for the Shell interpreter) by
setting the value of the property `zeppelin.shell.interpolation` to `true` (see _Configuration_ above).
More details of this feature can be found in the Spark interpreter documentation under
[Object Interpolation](spark.html#object-interpolation)
More details of this feature can be found in [Zeppelin-Context](../usage/other_features/zeppelin_context.html)

154
docs/interpreter/spark.md
View file

@ -323,158 +323,7 @@ z.load("groupId:artifactId:version").local()
## ZeppelinContext
Zeppelin automatically injects `ZeppelinContext` as variable `z` in your Scala/Python environment. `ZeppelinContext` provides some additional functions and utilities.
### Exploring Spark DataFrames
`ZeppelinContext` provides a `show` method, which, using Zeppelin's `table` feature, can be used to nicely display a Spark DataFrame:
```
df = spark.read.csv('/path/to/csv')
z.show(df)
```
### Object Exchange
`ZeppelinContext` extends map and it's shared between Scala and Python environment.
So you can put some objects from Scala and read it from Python, vice versa.
<div class="codetabs">
<div data-lang="scala" markdown="1">
{% highlight scala %}
// Put object from scala
%spark
val myObject = ...
z.put("objName", myObject)
// Exchanging data frames
myScalaDataFrame = ...
z.put("myScalaDataFrame", myScalaDataFrame)
val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame]
{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
{% highlight python %}
# Get object from python
%spark.pyspark
myObject = z.get("objName")
# Exchanging data frames
myPythonDataFrame = ...
z.put("myPythonDataFrame", postsDf._jdf)
myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext)
{% endhighlight %}
</div>
</div>
### Object Interpolation
Some interpreters can interpolate object values from `z` into the paragraph text by using the
`{variable-name}` syntax. The value of any object previously `put` into `z` can be
interpolated into a paragraph text by using such a pattern containing the object's name.
The following example shows one use of this facility:
####In Scala cell:
```
z.put("minAge", 35)
```
####In later SQL cell:
```
%sql select * from members where age >= {minAge}
```
The interpolation of a `{var-name}` pattern is performed only when `z` contains an object with the specified name.
But the pattern is left unchanged if the named object does not exist in `z`.
Further, all `{var-name}` patterns within the paragraph text must must be translatable for any interpolation to occur --
translation of only some of the patterns in a paragraph text is never done.
In some situations, it is necessary to use { and } characters in a paragraph text without invoking the
object interpolation mechanism. For these cases an escaping mechanism is available --
doubled braces {{ and }} should be used. The following example shows the use of {{ and }} for passing a
regular expression containing just { and } into the paragraph text.
```
%sql select * from members where name rlike '[aeiou]{{3}}'
```
To summarize, patterns of the form `{var-name}` within the paragraph text will be interpolated only if a predefined
object of the specified name exists. Additionally, all such patterns within the paragraph text should also
be translatable for any interpolation to occur. Patterns of the form `{{any-text}}` are translated into `{any-text}`.
These translations are performed only when all occurrences of `{`, `}`, `{{`, and `}}` in the paragraph text conform
to one of the two forms described above. Paragraph text containing `{` and/or `}` characters used in any other way
(than `{var-name}` and `{{any-text}}`) is used as-is without any changes.
No error is flagged in any case. This behavior is identical to the implementation of a similar feature in
Jupyter's shell invocation using the `!` magic command.
This feature is disabled by default, and must be explicitly turned on for each interpreter independently
by setting the value of an interpreter-specific property to `true`.
Consult the _Configuration_ section of each interpreter's documentation
to find out if object interpolation is implemented, and the name of the parameter that must be set to `true` to
enable the feature. The name of the parameter used to enable this feature it is different for each interpreter.
For example, the SparkSQL and Shell interpreters use the parameter names `zeppelin.spark.sql.interpolation` and
`zeppelin.shell.interpolation` respectively.
At present only the SparkSQL and Shell interpreters support object interpolation.
### Form Creation
`ZeppelinContext` provides functions for creating forms.
In Scala and Python environments, you can create forms programmatically.
<div class="codetabs">
<div data-lang="scala" markdown="1">
{% highlight scala %}
%spark
/* Create text input form */
z.input("formName")
/* Create text input form with default value */
z.input("formName", "defaultValue")
/* Create select form */
z.select("formName", Seq(("option1", "option1DisplayName"),
("option2", "option2DisplayName")))
/* Create select form with default value*/
z.select("formName", "option1", Seq(("option1", "option1DisplayName"),
("option2", "option2DisplayName")))
{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
{% highlight python %}
%spark.pyspark
# Create text input form
z.input("formName")
# Create text input form with default value
z.input("formName", "defaultValue")
# Create select form
z.select("formName", [("option1", "option1DisplayName"),
("option2", "option2DisplayName")])
# Create select form with default value
z.select("formName", [("option1", "option1DisplayName"),
("option2", "option2DisplayName")], "option1")
{% endhighlight %}
</div>
</div>
In sql environment, you can create form in simple template.
```sql
%spark.sql
select * from ${table=defaultTableName} where text like '%${search}%'
```
To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/intro.html).
See [Zeppelin-Context](../usage/other_features/zeppelin_context.html) for more details.
## Matplotlib Integration (pyspark)
Both the `python` and `pyspark` interpreters have built-in support for inline visualization using `matplotlib`,
@ -518,3 +367,4 @@ This is to make the server communicate with KDC.
> **NOTE:** If you do not have permission to access for the above spark-defaults.conf file, optionally, you can add the above lines to the Spark Interpreter setting through the Interpreter tab in the Zeppelin UI.
4. That's it. Play with Zeppelin!

230
docs/usage/other_features/zeppelin_context.md Normal file
View file

@ -0,0 +1,230 @@
---
layout: page
title: "Zeppelin-Context"
description: "The Zeppelin-Context is a system-wide container for a variety of user-specific settings and parameters that are accessible across notebooks, cells, and interpreters."
group: usage/other_features
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
{% include JB/setup %}
# Zeppelin-Context
<div id="toc"></div>
The zeppelin-context is a system-wide container for common utility functions and
user-specific data. It implements functions for data input, data display, etc. that are
often needed but are not uniformly available in all interpreters.
Its single per-user instance is accessible across all of the user's notebooks and cells,
enabling data exchange between cells - even in different notebooks.
But the way in which the zeppelin-context is used, and the functionality available differs
depending on whether or not the associated interpreter is based on a programming language.
Details of how the zeppelin-context is used for different purposes and in different
environments is described below.
## Usage in Programming Language Cells
In many programming-language based interpreters (e.g. Spark, Python, R) the zeppelin-context is available
as a predefined variable `z` that can be used by directly invoking its methods.
The methods available on the `z` object are described below.
Interpreters based on programming languages like spark.dep, beam, etc. also provide the
predefined variable `z`.
### Exploring Spark DataFrames
The zeppelin-context provides a `show` method, which, using Zeppelin's `table` feature,
can be used to nicely display a Spark DataFrame:
```
df = spark.read.csv('/path/to/csv')
z.show(df)
```
### Object Exchange
`ZeppelinContext` extends map and it's shared between Scala and Python environment.
So you can put some objects from Scala and read it from Python, vice versa.
<div class="codetabs">
<div data-lang="scala" markdown="1">
{% highlight scala %}
// Put object from scala
%spark
val myObject = ...
z.put("objName", myObject)
// Exchanging data frames
myScalaDataFrame = ...
z.put("myScalaDataFrame", myScalaDataFrame)
val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame]
{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
{% highlight python %}
# Get object from python
%spark.pyspark
myObject = z.get("objName")
# Exchanging data frames
myPythonDataFrame = ...
z.put("myPythonDataFrame", postsDf._jdf)
myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext)
{% endhighlight %}
</div>
</div>
### Form Creation
`ZeppelinContext` provides functions for creating forms.
In Scala and Python environments, you can create forms programmatically.
<div class="codetabs">
<div data-lang="scala" markdown="1">
{% highlight scala %}
%spark
/* Create text input form */
z.input("formName")
/* Create text input form with default value */
z.input("formName", "defaultValue")
/* Create select form */
z.select("formName", Seq(("option1", "option1DisplayName"),
("option2", "option2DisplayName")))
/* Create select form with default value*/
z.select("formName", "option1", Seq(("option1", "option1DisplayName"),
("option2", "option2DisplayName")))
{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
{% highlight python %}
%spark.pyspark
# Create text input form
z.input("formName")
# Create text input form with default value
z.input("formName", "defaultValue")
# Create select form
z.select("formName", [("option1", "option1DisplayName"),
("option2", "option2DisplayName")])
# Create select form with default value
z.select("formName", [("option1", "option1DisplayName"),
("option2", "option2DisplayName")], "option1")
{% endhighlight %}
</div>
</div>
In sql environment, you can create form in simple template.
```sql
%spark.sql
select * from ${table=defaultTableName} where text like '%${search}%'
```
To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/intro.html).
### Interpreter-Specific Functions
Some interpreters use a subclass of `BaseZepplinContext` augmented with interpreter-specific
functions. For example the dependency loader (%spark.dep) has `addRepo()`, `load()`, and other
related methods that implement its functionality. Interpreter-specific functions are
described within each interpreter's documentation.
## Usage in Non-Programming Cells
Cells with a non-programming interpreter (see table below) use a pattern match-and-replace
approach to use two of zeppelin-context's features: dynamic forms and
object interpolation.
| Non-Programming Interpreters |
|-------------------------------------------------------------------|
|spark.sql, bigquery, cassandra, elasticsearch, file, hbase, ignite, jdbc, kylin, livy, markdown, neo4j, pig, python, shell, zengine |
Dynamic forms are available in all non-programming interpreters,
but object interpolation is only available in a small (but growing) list of interpreters.
Both these zeppelin-context features are described below.
### Dynamic Forms
Patterns of the form ${ ... } are used to dynamically create additional HTML elements
for requesting user input (that replaces the corresponding pattern in the paragraph text).
Currently only [text](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/text),
[select](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) with
[options](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/option), and
[checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) are supported.
Dynamic forms are described in detail here: [Dynamic Form](../usage/dynamic_form/intro.html).
### Object Interpolation
Some interpreters can interpolate object values from `z` into the paragraph text by using the
`{variable-name}` syntax. The value of any object previously `put` into `z` can be
interpolated into a paragraph text by using such a pattern containing the object's name.
The following example shows one use of this facility:
####In Scala cell:
```
z.put("minAge", 35)
```
####In later SQL cell:
```
%sql select * from members where age >= {minAge}
```
The interpolation of a `{var-name}` pattern is performed only when `z` contains an object with the specified name.
But the pattern is left unchanged if the named object does not exist in `z`.
Further, all `{var-name}` patterns within the paragraph text must must be translatable for any interpolation to occur --
translation of only some of the patterns in a paragraph text is never done.
In some situations, it is necessary to use { and } characters in a paragraph text without invoking the
object interpolation mechanism. For these cases an escaping mechanism is available --
doubled braces {{ and }} should be used. The following example shows the use of {{ and }} for passing a
regular expression containing just { and } into the paragraph text.
```
%sql select * from members where name rlike '[aeiou]{{3}}'
```
To summarize, patterns of the form `{var-name}` within the paragraph text will be interpolated only if a predefined
object of the specified name exists. Additionally, all such patterns within the paragraph text should also
be translatable for any interpolation to occur. Patterns of the form `{{any-text}}` are translated into `{any-text}`.
These translations are performed only when all occurrences of `{`, `}`, `{{`, and `}}` in the paragraph text conform
to one of the two forms described above. Paragraph text containing `{` and/or `}` characters used in any other way
(than `{var-name}` and `{{any-text}}`) is used as-is without any changes.
No error is flagged in any case. This behavior is identical to the implementation of a similar feature in
Jupyter's shell invocation using the `!` magic command.
This feature is disabled by default, and must be explicitly turned on for each interpreter independently
by setting the value of an interpreter-specific property to `true`.
Consult the _Configuration_ section of each interpreter's documentation
to find out if object interpolation is implemented, and the name of the parameter that must be set to `true` to
enable the feature. The name of the parameter used to enable this feature it is different for each interpreter.
For example, the SparkSQL and Shell interpreters use the parameter names `zeppelin.spark.sql.interpolation` and
`zeppelin.shell.interpolation` respectively.
At present only the SparkSQL and Shell interpreters support object interpolation.