mirror of
https://github.com/apache/zeppelin
synced 2026-05-24 09:38:26 +00:00
Merge remote-tracking branch 'origin/master' into ZEPPELIN-946
# Conflicts: # docs/security/shiroauthentication.md
This commit is contained in:
Prabhjyot Singh
commit
3149417c99
68 changed files with 996 additions and 1351 deletions
2
.github/PULL_REQUEST_TEMPLATE
vendored
2
.github/PULL_REQUEST_TEMPLATE
vendored
|
|
@ -1,6 +1,6 @@
|
|||
### What is this PR for?
|
||||
A few sentences describing the overall goals of the pull request's commits.
|
||||
First time? Check out the contributing guide - https://github.com/apache/incubator-zeppelin/blob/master/CONTRIBUTING.md
|
||||
First time? Check out the contributing guide - https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md
|
||||
|
||||
|
||||
### What type of PR is it?
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# How to contribute
|
||||
|
||||
**Zeppelin** is [Apache2 License](https://github.com/apache/incubator-zeppelin/blob/master/CONTRIBUTING.md) Software.
|
||||
**Zeppelin** is [Apache2 License](https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md) Software.
|
||||
Contributing to Zeppelin (Source code, Documents, Image, Website) means you agree to the Apache2 License.
|
||||
|
||||
1. Make sure your issue is not already in the [Jira issue tracker](https://issues.apache.org/jira/browse/ZEPPELIN)
|
||||
|
|
@ -17,7 +17,7 @@ In order to make the review process easier, please follow this template when mak
|
|||
```
|
||||
### What is this PR for?
|
||||
A few sentences describing the overall goals of the pull request's commits.
|
||||
First time? Check out the contributing guide - https://github.com/apache/incubator-zeppelin/blob/master/CONTRIBUTING.md
|
||||
First time? Check out the contributing guide - https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md
|
||||
|
||||
### What type of PR is it?
|
||||
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
|
||||
|
|
@ -58,7 +58,7 @@ You can also test and review a particular Pull Request. Here are two useful ways
|
|||
* Another way is using [github/hub](https://github.com/github/hub).
|
||||
|
||||
```
|
||||
hub checkout https://github.com/apache/incubator-zeppelin/pull/[# of PR]
|
||||
hub checkout https://github.com/apache/zeppelin/pull/[# of PR]
|
||||
```
|
||||
|
||||
The above two methods will help you test and review Pull Requests.
|
||||
|
|
@ -91,8 +91,8 @@ Here are some things you will need to build and test Zeppelin.
|
|||
|
||||
### Software Configuration Management (SCM)
|
||||
|
||||
Zeppelin uses Git for its SCM system. `http://git.apache.org/incubator-zeppelin.git` you'll need git client installed in your development machine.
|
||||
For write access, `https://git-wip-us.apache.org/repos/asf/incubator-zeppelin.git`
|
||||
Zeppelin uses Git for its SCM system. `http://git.apache.org/zeppelin.git` you'll need git client installed in your development machine.
|
||||
For write access, `https://git-wip-us.apache.org/repos/asf/zeppelin.git`
|
||||
|
||||
### Integrated Development Environment (IDE)
|
||||
|
||||
|
|
@ -121,7 +121,7 @@ The top-level pom.xml describes the basic project structure. Currently Zeppelin
|
|||
|
||||
### Web Project Contribution Guidelines
|
||||
If you plan on making a contribution to Zeppelin's WebApplication,
|
||||
please check [its own contribution guidelines](https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-web/CONTRIBUTING.md)
|
||||
please check [its own contribution guidelines](https://github.com/apache/zeppelin/blob/master/zeppelin-web/CONTRIBUTING.md)
|
||||
|
||||
### Code convention
|
||||
We are following Google Code style:
|
||||
|
|
@ -138,31 +138,31 @@ To build the code, install
|
|||
* Apache Maven
|
||||
|
||||
## Getting the source code
|
||||
First of all, you need the Zeppelin source code. The official location for Zeppelin is [http://git.apache.org/incubator-zeppelin.git](http://git.apache.org/incubator-zeppelin.git).
|
||||
First of all, you need the Zeppelin source code. The official location for Zeppelin is [http://git.apache.org/zeppelin.git](http://git.apache.org/zeppelin.git).
|
||||
|
||||
### git access
|
||||
|
||||
Get the source code on your development machine using git.
|
||||
|
||||
```
|
||||
git clone git://git.apache.org/incubator-zeppelin.git zeppelin
|
||||
git clone git://git.apache.org/zeppelin.git zeppelin
|
||||
```
|
||||
|
||||
You may also want to develop against a specific branch. For example, for branch-0.5.6
|
||||
|
||||
```
|
||||
git clone -b branch-0.5.6 git://git.apache.org/incubator-zeppelin.git zeppelin
|
||||
git clone -b branch-0.5.6 git://git.apache.org/zeppelin.git zeppelin
|
||||
```
|
||||
|
||||
or with write access
|
||||
|
||||
```
|
||||
git clone https://git-wip-us.apache.org/repos/asf/incubator-zeppelin.git
|
||||
git clone https://git-wip-us.apache.org/repos/asf/zeppelin.git
|
||||
```
|
||||
|
||||
### Fork repository
|
||||
|
||||
If you want not only build Zeppelin but also make change, then you need fork Zeppelin github mirror repository (https://github.com/apache/incubator-zeppelin) and make pull request.
|
||||
If you want not only build Zeppelin but also make change, then you need fork Zeppelin github mirror repository (https://github.com/apache/zeppelin) and make pull request.
|
||||
|
||||
|
||||
## Build
|
||||
|
|
@ -193,11 +193,11 @@ Zeppelin has 3 types of tests:
|
|||
2. Integration Tests: The integration tests run after all modules are build. The integration tests launch an instance of Zeppelin server. ZeppelinRestApiTest is an example integration test.
|
||||
3. GUI integration tests: These tests validate the Zeppelin UI elements. These tests require a running Zeppelin server and launches a web browser to validate Notebook UI elements like Notes and their execution. See ZeppelinIT as an example.
|
||||
|
||||
Currently the GUI integration tests are not run in the Maven and are only run in the CI environment when the pull request is submitted to github. Make sure to watch the [CI results] (https://travis-ci.org/apache/incubator-zeppelin/pull_requests) for your pull request.
|
||||
Currently the GUI integration tests are not run in the Maven and are only run in the CI environment when the pull request is submitted to github. Make sure to watch the [CI results] (https://travis-ci.org/apache/zeppelin/pull_requests) for your pull request.
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
Zeppelin uses Travis for CI. In the project root there is .travis.yml that configures CI and [publishes CI results] (https://travis-ci.org/apache/incubator-zeppelin/builds)
|
||||
Zeppelin uses Travis for CI. In the project root there is .travis.yml that configures CI and [publishes CI results] (https://travis-ci.org/apache/zeppelin/builds)
|
||||
|
||||
|
||||
## Run Zeppelin server in development mode
|
||||
|
|
@ -225,6 +225,6 @@ You can find issues for [beginner](https://issues.apache.org/jira/browse/ZEPPELI
|
|||
## Stay involved
|
||||
Everyone is welcome to join our mailing list:
|
||||
|
||||
* [users@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-users/) is for usage questions, help, and announcements [ [subscribe](mailto:users-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:users-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-users/) ]
|
||||
* [dev@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-users/) is for people who want to contribute code to Zeppelin.[ [subscribe](mailto:dev-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:dev-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-dev/) ]
|
||||
* [commits@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-commits/) is for commit messages and patches to Zeppelin. [ [subscribe](mailto:commits-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:commits-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-commits/) ]
|
||||
* [users@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/zeppelin-users/) is for usage questions, help, and announcements [ [subscribe](mailto:users-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:users-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/zeppelin-users/) ]
|
||||
* [dev@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/zeppelin-users/) is for people who want to contribute code to Zeppelin.[ [subscribe](mailto:dev-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:dev-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/zeppelin-dev/) ]
|
||||
* [commits@zeppelin.apache.org](http://mail-archives.apache.org/mod_mbox/zeppelin-commits/) is for commit messages and patches to Zeppelin. [ [subscribe](mailto:commits-subscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20subscribe), [unsubscribe](mailto:commits-unsubscribe@zeppelin.apache.org?subject=send%20this%20email%20to%20unsubscribe), [archive](http://mail-archives.apache.org/mod_mbox/zeppelin-commits/) ]
|
||||
|
|
|
|||
2
NOTICE
2
NOTICE
|
|
@ -1,4 +1,4 @@
|
|||
Apache Zeppelin (incubating)
|
||||
Apache Zeppelin
|
||||
Copyright 2015 - 2016 The Apache Software Foundation
|
||||
|
||||
This product includes software developed at
|
||||
|
|
|
|||
11
README.md
11
README.md
|
|
@ -2,10 +2,10 @@
|
|||
|
||||
**Documentation:** [User Guide](http://zeppelin.apache.org/docs/latest/index.html)<br/>
|
||||
**Mailing Lists:** [User and Dev mailing list](http://zeppelin.apache.org/community.html)<br/>
|
||||
**Continuous Integration:** [](https://travis-ci.org/apache/incubator-zeppelin) <br/>
|
||||
**Contributing:** [Contribution Guide](https://github.com/apache/incubator-zeppelin/blob/master/CONTRIBUTING.md)<br/>
|
||||
**Continuous Integration:** [](https://travis-ci.org/apache/zeppelin) <br/>
|
||||
**Contributing:** [Contribution Guide](https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md)<br/>
|
||||
**Issue Tracker:** [Jira](https://issues.apache.org/jira/browse/ZEPPELIN)<br/>
|
||||
**License:** [Apache 2.0](https://github.com/apache/incubator-zeppelin/blob/master/LICENSE)
|
||||
**License:** [Apache 2.0](https://github.com/apache/zeppelin/blob/master/LICENSE)
|
||||
|
||||
|
||||
**Zeppelin**, a web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more.
|
||||
|
|
@ -162,8 +162,7 @@ enable 3rd party vendor repository (cloudera)
|
|||
##### `-Pmapr[version]` (optional)
|
||||
|
||||
For the MapR Hadoop Distribution, these profiles will handle the Hadoop version. As MapR allows different versions of Spark to be installed, you should specify which version of Spark is installed on the cluster by adding a Spark profile (`-Pspark-1.2`, `-Pspark-1.3`, etc.) as needed.
|
||||
For Hive, check the hive/pom.xml and adjust the version installed as well. The correct Maven
|
||||
artifacts can be found for every version of MapR at http://doc.mapr.com
|
||||
The correct Maven artifacts can be found for every version of MapR at http://doc.mapr.com
|
||||
|
||||
Available profiles are
|
||||
|
||||
|
|
@ -288,4 +287,4 @@ mvn verify
|
|||
mvn verify -P using-packaged-distr
|
||||
```
|
||||
|
||||
[](https://github.com/igrigorik/ga-beacon)
|
||||
[](https://github.com/igrigorik/ga-beacon)
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ limitations under the License.
|
|||
|
||||
# Shiro Authentication
|
||||
To connect to Zeppelin, users will be asked to enter their credentials. Once logged, a user has access to all notes including other users notes.
|
||||
This a a first step toward full security as implemented by this pull request (https://github.com/apache/incubator-zeppelin/pull/53).
|
||||
This a a first step toward full security as implemented by this pull request (https://github.com/apache/zeppelin/pull/53).
|
||||
|
||||
# Security setup
|
||||
1. Secure the HTTP channel: Comment the line "/** = anon" and uncomment the line "/** = authcBasic" in the file conf/shiro.ini. Read more about he shiro.ini file format at the following URL http://shiro.apache.org/configuration.html#Configuration-INISections.
|
||||
|
|
|
|||
|
|
@ -30,12 +30,12 @@
|
|||
<artifactId>zeppelin-cassandra</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
<version>0.6.0-SNAPSHOT</version>
|
||||
<name>Zeppelin: Cassandra</name>
|
||||
<name>Zeppelin: Apache Cassandra interpreter</name>
|
||||
<description>Zeppelin cassandra support</description>
|
||||
<url>http://zeppelin.apache.org</url>
|
||||
|
||||
<properties>
|
||||
<cassandra.driver.version>3.0.0-rc1</cassandra.driver.version>
|
||||
<cassandra.driver.version>3.0.1</cassandra.driver.version>
|
||||
<snappy.version>1.0.5.4</snappy.version>
|
||||
<lz4.version>1.3.0</lz4.version>
|
||||
<scala.version>2.10.4</scala.version>
|
||||
|
|
|
|||
|
|
@ -110,7 +110,7 @@ public class CassandraInterpreter extends Interpreter {
|
|||
public static final String DEFAULT_PORT = "9042";
|
||||
public static final String DEFAULT_CLUSTER = "Test Cluster";
|
||||
public static final String DEFAULT_KEYSPACE = "system";
|
||||
public static final String DEFAULT_PROTOCOL_VERSION = "3";
|
||||
public static final String DEFAULT_PROTOCOL_VERSION = "4";
|
||||
public static final String DEFAULT_COMPRESSION = "NONE";
|
||||
public static final String DEFAULT_CREDENTIAL = "none";
|
||||
public static final String DEFAULT_POLICY = "DEFAULT";
|
||||
|
|
@ -159,7 +159,7 @@ public class CassandraInterpreter extends Interpreter {
|
|||
"IP address). Default = localhost. Ex: '192.168.0.12,node2,node3'")
|
||||
.add(CASSANDRA_PORT, DEFAULT_PORT, "Cassandra native port. Default = 9042")
|
||||
.add(CASSANDRA_PROTOCOL_VERSION, DEFAULT_PROTOCOL_VERSION,
|
||||
"Cassandra protocol version. Default = 3")
|
||||
"Cassandra protocol version. Default = 4")
|
||||
.add(CASSANDRA_CLUSTER_NAME, DEFAULT_CLUSTER, "Cassandra cluster name. " +
|
||||
"Default = 'Test Cluster'")
|
||||
.add(CASSANDRA_KEYSPACE_NAME, DEFAULT_KEYSPACE, "Cassandra keyspace name. " +
|
||||
|
|
@ -311,7 +311,7 @@ public class CassandraInterpreter extends Interpreter {
|
|||
|
||||
@Override
|
||||
public FormType getFormType() {
|
||||
return FormType.NATIVE;
|
||||
return FormType.SIMPLE;
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
|
|||
|
|
@ -103,12 +103,12 @@
|
|||
<ul class="dropdown-menu">
|
||||
<li>
|
||||
<a role="button">
|
||||
<span class="text-info">Version <strong>2.0</strong></span>
|
||||
<span class="text-info">Version <strong>3.0</strong></span>
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a role="button">
|
||||
<span class="text-info">Java Driver Version <strong>3.0.0-rc1</strong></span>
|
||||
<span class="text-info">Java Driver Version <strong>3.0.1</strong></span>
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
|
|
@ -215,6 +215,14 @@
|
|||
<tr><th>Cassandra version</th><th>Documentation</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>3.x</strong></td>
|
||||
<td>
|
||||
<a href="http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html" target="_blank">
|
||||
http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>2.2</strong></td>
|
||||
<td>
|
||||
|
|
@ -464,6 +472,11 @@
|
|||
<td><strong>@fetchSize=<em>int value</em></strong></td>
|
||||
<td>Apply the given fetch size to all queries in the paragraph</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Request Timeout</td>
|
||||
<td><strong>@requestTimeOut=<em>int value</em></strong></td>
|
||||
<td>Apply the given request timeout <strong>in millisecs</strong> to all queries in the paragraph</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<br/>
|
||||
|
|
@ -506,6 +519,10 @@
|
|||
<td>Fetch Size</td>
|
||||
<td>Any integer value</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Request Timeout</td>
|
||||
<td>Any integer value</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<br/>
|
||||
|
|
@ -526,19 +543,19 @@
|
|||
);
|
||||
TRUNCATE spark_demo.ts;
|
||||
|
||||
# Timestamp in the past
|
||||
// Timestamp in the past
|
||||
@timestamp=10
|
||||
|
||||
# Force timestamp directly in the first insert
|
||||
// Force timestamp directly in the first insert
|
||||
INSERT INTO spark_demo.ts(key,value) VALUES(1,'first insert') USING TIMESTAMP 100;
|
||||
|
||||
# Select some data to make the clock turn
|
||||
// Select some data to make the clock turn
|
||||
SELECT * FROM spark_demo.albums LIMIT 100;
|
||||
|
||||
# Now insert using the timestamp parameter set at the beginning(10)
|
||||
// Now insert using the timestamp parameter set at the beginning(10)
|
||||
INSERT INTO spark_demo.ts(key,value) VALUES(1,'second insert');
|
||||
|
||||
# Check for the result. You should see 'first insert'
|
||||
// Check for the result. You should see 'first insert'
|
||||
SELECT value FROM spark_demo.ts WHERE key=1;
|
||||
</pre>
|
||||
</div>
|
||||
|
|
@ -599,13 +616,13 @@
|
|||
<div class="col-md-10 col-md-offset-1">
|
||||
<pre>
|
||||
|
||||
@prepare[statement_name]=...
|
||||
@prepare[statement-name]=...
|
||||
|
||||
@bind[statement_name]=’text’, 1223, ’2015-07-30 12:00:01’, null, true, [‘list_item1’, ’list_item2’]
|
||||
@bind[statement-name]=’text’, 1223, ’2015-07-30 12:00:01’, null, true, [‘list_item1’, ’list_item2’]
|
||||
|
||||
@bind[statement_name_with_no_bound_value]
|
||||
@bind[statement-name-with-no-bound-value]
|
||||
|
||||
@remove_prepare[statement_name]
|
||||
@remove_prepare[statement-name]
|
||||
|
||||
</pre>
|
||||
</div>
|
||||
|
|
@ -615,10 +632,10 @@
|
|||
<h3>II @prepare</h3>
|
||||
<br/>
|
||||
<p>
|
||||
You can use the syntax "<strong>@prepare[statement_name]=SELECT ...</strong>" to create a prepared statement.
|
||||
The <em>statement_name</em> is mandatory because the interpreter prepares the given statement with the
|
||||
You can use the syntax "<strong>@prepare[statement-name]=SELECT ...</strong>" to create a prepared statement.
|
||||
The <em>statement-name</em> is mandatory because the interpreter prepares the given statement with the
|
||||
Java driver and saves the generated prepared statement in an internal map, using the provided
|
||||
<em>statement_name</em> as search key.
|
||||
<em>statement-name</em> as search key.
|
||||
<br/><br/>
|
||||
<div class="alert alert-info">
|
||||
Please note that this internal prepared statement map is shared with <strong>all notebooks</strong>
|
||||
|
|
@ -626,7 +643,7 @@
|
|||
</div>
|
||||
<br/>
|
||||
<div class="alert alert-warning">
|
||||
If the interpreter encounters many @prepare for the <strong>same statement_name</strong> (key),
|
||||
If the interpreter encounters many @prepare for the <strong>same statement-name</strong> (key),
|
||||
only the <strong>first</strong> statement will be taken into account.
|
||||
</div>
|
||||
<br/>
|
||||
|
|
@ -645,7 +662,7 @@
|
|||
<br/>
|
||||
|
||||
For the above example, the prepared statement is <strong>"SELECT * FROM spark_demo.albums LIMIT ?"</strong>.
|
||||
<em>"SELECT * FROM spark_demo.artists LIMIT ?"</em> is ignored because an entry already exists in the
|
||||
<strong>"SELECT * FROM spark_demo.artists LIMIT ?"</strong> is ignored because an entry already exists in the
|
||||
prepared statements map with the key <strong>select</strong>.
|
||||
<br/><br/>
|
||||
In the context of Zeppelin, a notebook can be scheduled to be executed at regular interval,
|
||||
|
|
@ -712,7 +729,7 @@
|
|||
<h3>IV @remove_prepare</h3>
|
||||
<br/>
|
||||
<p>
|
||||
To avoid for a prepared statement to stay forever in the prepared statement map, you can use the <strong>@remove_prepare[statement_name]</strong> syntax
|
||||
To avoid for a prepared statement to stay forever in the prepared statement map, you can use the <strong>@remove_prepare[statement-name]</strong> syntax
|
||||
to remove it. Removing a non-existing prepared statement yields no error.
|
||||
</p>
|
||||
</div>
|
||||
|
|
@ -735,16 +752,22 @@
|
|||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p>
|
||||
Instead of hard-coding your CQL queries, it is possible to use the mustache syntax (<strong>{{ }}</strong>)
|
||||
to inject simple value or multiple choices forms.
|
||||
Instead of hard-coding your CQL queries, it is possible to use <strong>
|
||||
<a href="http://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/manual/dynamicform.html" target="_blank">Zeppelin dynamic form</a>
|
||||
</strong> syntax to inject simple value or multiple choices forms.
|
||||
|
||||
The legacy mustache syntax ( <strong>{{ }}</strong> ) to bind input text and select form is still supported but is deprecated and will be removed in future releases.
|
||||
|
||||
<br/><br/>
|
||||
|
||||
<h6> -- Legacy syntax -- </h6>
|
||||
The syntax for simple parameter is: <strong>{{input_Label=default value}}</strong>.
|
||||
The default value is mandatory because the first time the paragraph is executed,
|
||||
we launch the CQL query before rendering the form so at least one value should be provided.
|
||||
<br/><br/>
|
||||
The syntax for multiple choices parameter is: <strong>{{input_Label=value1 | value2 | … | valueN }}</strong>.
|
||||
By default the first choice is used for CQL query the first time the paragraph is executed.
|
||||
<h6> -- End legacy syntax -- </h6>
|
||||
<br/><br/>
|
||||
Example:
|
||||
<br/>
|
||||
|
|
@ -755,8 +778,8 @@
|
|||
#Secondary index on performer style
|
||||
SELECT name, country, performer
|
||||
FROM spark_demo.performers
|
||||
WHERE name='{{performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}}'
|
||||
AND styles CONTAINS '{{style=Rock}}';
|
||||
WHERE name='\${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}'
|
||||
AND styles CONTAINS '\${style=Rock}';
|
||||
|
||||
</pre>
|
||||
</div>
|
||||
|
|
@ -766,13 +789,13 @@
|
|||
In the above example, the first CQL query will be executed for <em>performer='Sheryl Crow'</em>
|
||||
AND <em>style='Rock'</em>. For subsequent queries, you can change the value directly using the form.
|
||||
Please note that we enclosed the {{ }} block between simple quotes (') because Cassandra expects a String here.
|
||||
We could have also use the <strong>{{style='Rock'}}</strong> syntax but this time, the value
|
||||
We could have also use the <strong>\${style='Rock'}</strong> syntax but this time, the value
|
||||
displayed on the form is <em>'Rock'</em> and not <em>Rock</em>.
|
||||
|
||||
<br/><br/>
|
||||
<div class="alert alert-info">
|
||||
It is also possible to use dynamic forms for <strong>prepared statements</strong>: <br/>
|
||||
<strong>@bind[select]=='{{performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}}', '{{style=Rock}}'</strong>
|
||||
<strong>@bind[select]=='\${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}', '\${style=Rock}'</strong>
|
||||
</div>
|
||||
</pre>
|
||||
</p>
|
||||
|
|
@ -892,7 +915,7 @@
|
|||
</tr>
|
||||
<tr>
|
||||
<td>cassandra.protocol.version</td>
|
||||
<td><strong>3</strong></td>
|
||||
<td><strong>4</strong></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>cassandra.query.default.consistency</td>
|
||||
|
|
@ -947,20 +970,35 @@
|
|||
<div id="${sharedStatesId}" class="panel-collapse collapse" role="tabpanel">
|
||||
<div class="panel-body">
|
||||
It is possible to execute many paragraphs in parallel. However, at the back-end side, we’re still using synchronous queries. <em>Asynchronous execution</em> is only possible when it is possible to return a <strong>Future</strong> value in the <strong>InterpreterResult</strong>. It may be an interesting proposal for the <strong>Zeppelin</strong> project.
|
||||
<br/>
|
||||
Another caveat is that the same <strong>com.datastax.driver.core.Session</strong> object is used for <strong>all</strong> notebooks and paragraphs. Consequently, if you use the <em>USE keyspace name;</em> statement to log into a keyspace, it will change the keyspace for <strong>all current users</strong> of the Cassandra interpreter because we only create 1 <strong>com.datastax.driver.core.Session</strong> object per instance of <strong>Cassandra</strong> interpreter.
|
||||
<br/>
|
||||
The same remark does apply to the <strong>prepared statement hash map</strong>, it is shared by <strong>all users</strong> using the same instance of <strong>Cassandra</strong> interpreter.
|
||||
<br/><br/>
|
||||
Recently, <strong>Zeppelin</strong> allows you to choose the level of isolation for your interpreters (see
|
||||
<strong><a href="http://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/manual/interpreters.html" target="_blank">Interpreter Binding Mode</a></strong> ).
|
||||
<br/><br/>
|
||||
Long story short, you have 3 available bindings:
|
||||
|
||||
<ul>
|
||||
<li><strong>shared</strong> : <em>same JVM</em> and <em>same Interpreter instance</em> for all notes</li>
|
||||
<li><strong>scoped</strong> : <em>same JVM</em> but <em>different Interpreter instances</em>, one for each note</li>
|
||||
<li><strong>isolated</strong> : <em>different JVM</em> running a <em>single Interpreter instance</em>, one JVM for each note</li>
|
||||
</ul>
|
||||
|
||||
<br/>
|
||||
Until <strong>Zeppelin</strong> offers a real multi-users separation, there is a work-around to segregate user environment and states: <em>create different Cassandra interpreter instances</em>
|
||||
<br/>
|
||||
<ol>
|
||||
<li>First go to the <strong>Interpreter</strong> menu and click on the <strong>Create</strong> button</li>
|
||||
<li>In the interpreter creation form, put <strong>cass-instance2</strong> as <strong>Name</strong> and select the <strong>cassandra</strong> in the interpreter drop-down list</li>
|
||||
<li>Click on <strong>Save</strong> to create the new interpreter instance. Now you should be able to see it in the interpreter list</li>
|
||||
<li>Go back to your notebook and click on the <strong>Gear</strong> icon to configure interpreter bindings. You should be able to see and select the <strong>cass-instance2</strong> interpreter instance in the available interpreter list instead of the standard <strong>cassandra</strong> instance</li>
|
||||
</ol>
|
||||
</div>
|
||||
Using the <strong>shared</strong> binding, the same <code>com.datastax.driver.core.Session</code> object is used for all notes and paragraphs.
|
||||
Consequently, if you use the <strong>USE keyspace name;</strong> statement to log into a keyspace,
|
||||
it will change the keyspace for all current users of the Cassandra interpreter because we only create 1
|
||||
<code>com.datastax.driver.core.Session</code> object per instance of Cassandra interpreter.
|
||||
|
||||
<br/><br/>
|
||||
The same remark does apply to the <strong>prepared statement hash map</strong>, it is shared by all users using the same instance of Cassandra interpreter.
|
||||
<br/><br/>
|
||||
When using <strong>scoped</strong> binding, in the <em>same JVM</em> <strong>Zeppelin</strong> will create multiple instances of the Cassandra interpreter,
|
||||
thus multiple <code>com.datastax.driver.core.Session</code> objects.
|
||||
<strong>Beware of resource and memory usage using this binding !</strong>
|
||||
<br/><br/>
|
||||
The <strong>isolated</strong> mode is the most extreme and will create as many JVM/<code>com.datastax.driver.core.Session</code> object as there are distinct notes.
|
||||
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
|
@ -974,6 +1012,20 @@
|
|||
</div>
|
||||
<div id="${changelogId}" class="panel-collapse collapse" role="tabpanel">
|
||||
<div class="panel-body">
|
||||
<strong>3.0</strong> :
|
||||
<br/>
|
||||
<ul>
|
||||
<li>Update documentation</li>
|
||||
<li>Update interactive documentation</li>
|
||||
<li>Add support for binary protocol <strong>V4</strong></li>
|
||||
<li>Implement new <code>@requestTimeOut</code> runtime option</li>
|
||||
<li>Upgrade Java driver version to <strong>3.0.1</strong></li>
|
||||
<li>Allow interpreter to add dynamic forms programmatically when using FormType.SIMPLE</li>
|
||||
<li>Allow dynamic form using default Zeppelin syntax</li>
|
||||
<li>Fixing typo on FallThroughPolicy</li>
|
||||
<li>Look for data in AngularObjectRegistry before creating dynamic form</li>
|
||||
<li>Add missing support for <code>ALTER</code> statements</li>
|
||||
</ul>
|
||||
<strong>2.0</strong> :
|
||||
<br/>
|
||||
<ul>
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ import com.datastax.driver.core._
|
|||
import com.datastax.driver.core.exceptions.DriverException
|
||||
import com.datastax.driver.core.policies.{LoggingRetryPolicy, FallthroughRetryPolicy, DowngradingConsistencyRetryPolicy, Policies}
|
||||
import org.apache.zeppelin.cassandra.TextBlockHierarchy._
|
||||
import org.apache.zeppelin.display.AngularObjectRegistry
|
||||
import org.apache.zeppelin.display.Input.ParamOption
|
||||
import org.apache.zeppelin.interpreter.InterpreterResult.Code
|
||||
import org.apache.zeppelin.interpreter.{InterpreterException, InterpreterResult, InterpreterContext}
|
||||
|
|
@ -41,17 +42,20 @@ import scala.collection.mutable.ArrayBuffer
|
|||
|
||||
/**
|
||||
* Value object to store runtime query parameters
|
||||
* @param consistency consistency level
|
||||
*
|
||||
* @param consistency consistency level
|
||||
* @param serialConsistency serial consistency level
|
||||
* @param timestamp timestamp
|
||||
* @param retryPolicy retry policy
|
||||
* @param fetchSize query fetch size
|
||||
* @param requestTimeOut request time out in millisecs
|
||||
*/
|
||||
case class CassandraQueryOptions(consistency: Option[ConsistencyLevel],
|
||||
serialConsistency:Option[ConsistencyLevel],
|
||||
timestamp: Option[Long],
|
||||
retryPolicy: Option[RetryPolicy],
|
||||
fetchSize: Option[Int])
|
||||
fetchSize: Option[Int],
|
||||
requestTimeOut: Option[Int])
|
||||
|
||||
/**
|
||||
* Singleton object to store constants
|
||||
|
|
@ -71,7 +75,7 @@ object InterpreterLogic {
|
|||
val fallThroughRetryPolicy = FallthroughRetryPolicy.INSTANCE
|
||||
val loggingDefaultRetryPolicy = new LoggingRetryPolicy(defaultRetryPolicy)
|
||||
val loggingDownGradingRetryPolicy = new LoggingRetryPolicy(downgradingConsistencyRetryPolicy)
|
||||
val loggingFallThrougRetryPolicy = new LoggingRetryPolicy(fallThroughRetryPolicy)
|
||||
val loggingFallThroughRetryPolicy = new LoggingRetryPolicy(fallThroughRetryPolicy)
|
||||
|
||||
val preparedStatements : mutable.Map[String,PreparedStatement] = new ConcurrentHashMap[String,PreparedStatement]().asScala
|
||||
|
||||
|
|
@ -273,7 +277,13 @@ class InterpreterLogic(val session: Session) {
|
|||
.flatMap(x => Option(x.value))
|
||||
.headOption
|
||||
|
||||
CassandraQueryOptions(consistency,serialConsistency, timestamp, retryPolicy, fetchSize)
|
||||
val requestTimeOut: Option[Int] = parameters
|
||||
.filter(_.paramType == RequestTimeOutParam)
|
||||
.map(_.getParam[RequestTimeOut])
|
||||
.flatMap(x => Option(x.value))
|
||||
.headOption
|
||||
|
||||
CassandraQueryOptions(consistency,serialConsistency, timestamp, retryPolicy, fetchSize, requestTimeOut)
|
||||
}
|
||||
|
||||
def generateSimpleStatement(st: SimpleStm, options: CassandraQueryOptions,context: InterpreterContext): SimpleStatement = {
|
||||
|
|
@ -305,19 +315,38 @@ class InterpreterLogic(val session: Session) {
|
|||
|
||||
def maybeExtractVariables(statement: String, context: InterpreterContext): String = {
|
||||
|
||||
def findInAngularRepository(variable: String): Option[AnyRef] = {
|
||||
val registry = context.getAngularObjectRegistry
|
||||
val noteId = context.getNoteId
|
||||
val paragraphId = context.getParagraphId
|
||||
val paragraphScoped: Option[AnyRef] = Option(registry.get(variable, noteId, paragraphId)).map[AnyRef](_.get())
|
||||
|
||||
paragraphScoped
|
||||
}
|
||||
|
||||
def extractVariableAndDefaultValue(statement: String, exp: String):String = {
|
||||
exp match {
|
||||
case MULTIPLE_CHOICES_VARIABLE_DEFINITION_PATTERN(variable,choices) => {
|
||||
case MULTIPLE_CHOICES_VARIABLE_DEFINITION_PATTERN(variable, choices) => {
|
||||
val escapedExp: String = exp.replaceAll( """\{""", """\\{""").replaceAll( """\}""", """\\}""").replaceAll("""\|""","""\\|""")
|
||||
val listChoices:List[String] = choices.trim.split(CHOICES_SEPARATOR).toList
|
||||
val paramOptions= listChoices.map(choice => new ParamOption(choice, choice))
|
||||
val selected = context.getGui.select(variable, listChoices.head, paramOptions.toArray)
|
||||
statement.replaceAll(escapedExp,selected.toString)
|
||||
findInAngularRepository(variable) match {
|
||||
case Some(value) => statement.replaceAll(escapedExp,value.toString)
|
||||
case None => {
|
||||
val listChoices:List[String] = choices.trim.split(CHOICES_SEPARATOR).toList
|
||||
val paramOptions= listChoices.map(choice => new ParamOption(choice, choice))
|
||||
val selected = context.getGui.select(variable, listChoices.head, paramOptions.toArray)
|
||||
statement.replaceAll(escapedExp,selected.toString)
|
||||
}
|
||||
}
|
||||
}
|
||||
case SIMPLE_VARIABLE_DEFINITION_PATTERN(variable,defaultVal) => {
|
||||
val escapedExp: String = exp.replaceAll( """\{""", """\\{""").replaceAll( """\}""", """\\}""")
|
||||
val value = context.getGui.input(variable,defaultVal)
|
||||
statement.replaceAll(escapedExp,value.toString)
|
||||
findInAngularRepository(variable) match {
|
||||
case Some(value) => statement.replaceAll(escapedExp,value.toString)
|
||||
case None => {
|
||||
val value = context.getGui.input(variable,defaultVal)
|
||||
statement.replaceAll(escapedExp,value.toString)
|
||||
}
|
||||
}
|
||||
}
|
||||
case _ => throw new ParsingException(s"Invalid bound variable definition for '$exp' in '$statement'. It should be of form 'variable=defaultValue' or 'variable=value1|value2|...|valueN'")
|
||||
}
|
||||
|
|
@ -336,10 +365,11 @@ class InterpreterLogic(val session: Session) {
|
|||
case FallThroughRetryPolicy => statement.setRetryPolicy(fallThroughRetryPolicy)
|
||||
case LoggingDefaultRetryPolicy => statement.setRetryPolicy(loggingDefaultRetryPolicy)
|
||||
case LoggingDowngradingRetryPolicy => statement.setRetryPolicy(loggingDownGradingRetryPolicy)
|
||||
case LoggingFallThroughRetryPolicy => statement.setRetryPolicy(loggingFallThrougRetryPolicy)
|
||||
case LoggingFallThroughRetryPolicy => statement.setRetryPolicy(loggingFallThroughRetryPolicy)
|
||||
case _ => throw new InterpreterException(s"""Unknown retry policy ${options.retryPolicy.getOrElse("???")}""")
|
||||
}
|
||||
options.fetchSize.foreach(statement.setFetchSize(_))
|
||||
options.requestTimeOut.foreach(statement.setReadTimeoutMillis(_))
|
||||
}
|
||||
|
||||
private def createBoundStatement(codecRegistry: CodecRegistry, name: String, ps: PreparedStatement, rawBoundValues: String): BoundStatement = {
|
||||
|
|
|
|||
|
|
@ -207,6 +207,16 @@ class JavaDriverConfig {
|
|||
DEFAULT_MAX_REQUEST_PER_CONNECTION_LOCAL = "1024"
|
||||
DEFAULT_MAX_REQUEST_PER_CONNECTION_REMOTE = "256"
|
||||
return ProtocolVersion.V3
|
||||
case "4" =>
|
||||
DEFAULT_MAX_CONNECTION_PER_HOST_LOCAL = "1"
|
||||
DEFAULT_MAX_CONNECTION_PER_HOST_REMOTE = "1"
|
||||
DEFAULT_CORE_CONNECTION_PER_HOST_LOCAL = "1"
|
||||
DEFAULT_CORE_CONNECTION_PER_HOST_REMOTE = "1"
|
||||
DEFAULT_NEW_CONNECTION_THRESHOLD_LOCAL = "800"
|
||||
DEFAULT_NEW_CONNECTION_THRESHOLD_REMOTE = "200"
|
||||
DEFAULT_MAX_REQUEST_PER_CONNECTION_LOCAL = "1024"
|
||||
DEFAULT_MAX_REQUEST_PER_CONNECTION_REMOTE = "256"
|
||||
return ProtocolVersion.V4
|
||||
case _ =>
|
||||
DEFAULT_MAX_CONNECTION_PER_HOST_LOCAL = "1"
|
||||
DEFAULT_MAX_CONNECTION_PER_HOST_REMOTE = "1"
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ object ParagraphParser {
|
|||
LOGGING_DEFAULT_RETRY, LOGGING_DOWNGRADING_RETRY, LOGGING_FALLTHROUGH_RETRY)
|
||||
.mkString("""^\s*@retryPolicy\s*=\s*(""", "|" , """)\s*$""").r
|
||||
val FETCHSIZE_PATTERN = """^\s*@fetchSize\s*=\s*([0-9]+)\s*$""".r
|
||||
val REQUEST_TIMEOUT_PATTERN = """^\s*@requestTimeOut\s*=\s*([0-9]+)\s*$""".r
|
||||
|
||||
val SIMPLE_STATEMENT_PATTERN = """([^;]+;)""".r
|
||||
val PREPARE_STATEMENT_PATTERN = """^\s*@prepare\[([^]]+)\]\s*=\s*([^;]+)$""".r
|
||||
|
|
@ -69,7 +70,7 @@ object ParagraphParser {
|
|||
val UDF_PATTERN = """(?is)\s*(CREATE(?:\s+OR REPLACE)?\s+FUNCTION(?:\s+IF\s+NOT\s+EXISTS)?.+?(?:\s+|\n|\r|\f)AS(?:\s+|\n|\r|\f)(?:'|\$\$).+?(?:'|\$\$)\s*;)""".r
|
||||
|
||||
val GENERIC_STATEMENT_PREFIX =
|
||||
"""(?is)\s*(?:INSERT|UPDATE|DELETE|SELECT|CREATE|UPDATE|
|
||||
"""(?is)\s*(?:INSERT|UPDATE|DELETE|SELECT|CREATE|ALTER|
|
||||
|DROP|GRANT|REVOKE|TRUNCATE|LIST|USE)\s+""".r
|
||||
|
||||
val VALID_IDENTIFIER = "[a-z][a-z0-9_]*"
|
||||
|
|
@ -146,6 +147,7 @@ class ParagraphParser extends RegexParsers{
|
|||
def timestamp: Parser[Timestamp] = """\s*@timestamp.+""".r ^^ {case x => extractTimestamp(x.trim)}
|
||||
def retryPolicy: Parser[RetryPolicy] = """\s*@retryPolicy.+""".r ^^ {case x => extractRetryPolicy(x.trim)}
|
||||
def fetchSize: Parser[FetchSize] = """\s*@fetchSize.+""".r ^^ {case x => extractFetchSize(x.trim)}
|
||||
def requestTimeOut: Parser[RequestTimeOut] = """\s*@requestTimeOut.+""".r ^^ {case x => extractRequestTimeOut(x.trim)}
|
||||
|
||||
//Statements
|
||||
def createFunctionStatement: Parser[SimpleStm] = UDF_PATTERN ^^{case x => extractUdfStatement(x.trim)}
|
||||
|
|
@ -188,7 +190,7 @@ class ParagraphParser extends RegexParsers{
|
|||
case begin ~ cqls ~ end => BatchStm(extractBatchType(begin),cqls)}
|
||||
|
||||
def queries:Parser[List[AnyBlock]] = rep(singleLineComment | multiLineComment | consistency | serialConsistency |
|
||||
timestamp | retryPolicy | fetchSize | removePrepare | prepare | bind | batch | describeCluster |
|
||||
timestamp | retryPolicy | fetchSize | requestTimeOut | removePrepare | prepare | bind | batch | describeCluster |
|
||||
describeKeyspace | describeKeyspaces |
|
||||
describeTable | describeTables |
|
||||
describeType | describeTypes |
|
||||
|
|
@ -244,6 +246,14 @@ class ParagraphParser extends RegexParsers{
|
|||
}
|
||||
}
|
||||
|
||||
def extractRequestTimeOut(text: String): RequestTimeOut = {
|
||||
text match {
|
||||
case REQUEST_TIMEOUT_PATTERN(requestTimeOut) => RequestTimeOut(requestTimeOut.trim.toInt)
|
||||
case _ => throw new InterpreterException(s"Invalid syntax for @requestTimeOut. " +
|
||||
s"It should comply to the pattern ${REQUEST_TIMEOUT_PATTERN.toString}")
|
||||
}
|
||||
}
|
||||
|
||||
def extractSimpleStatement(text: String): SimpleStm = {
|
||||
text match {
|
||||
case SIMPLE_STATEMENT_PATTERN(statement) => SimpleStm(statement)
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ object TextBlockHierarchy {
|
|||
object TimestampParam extends ParameterType
|
||||
object RetryPolicyParam extends ParameterType
|
||||
object FetchSizeParam extends ParameterType
|
||||
object RequestTimeOutParam extends ParameterType
|
||||
|
||||
|
||||
abstract class QueryParameters(val paramType: ParameterType) extends AnyBlock(ParameterBlock) {
|
||||
|
|
@ -60,6 +61,8 @@ object TextBlockHierarchy {
|
|||
|
||||
case class FetchSize(value: Int) extends QueryParameters(FetchSizeParam)
|
||||
|
||||
case class RequestTimeOut(value: Int) extends QueryParameters(RequestTimeOutParam)
|
||||
|
||||
abstract class RetryPolicy extends QueryParameters(RetryPolicyParam)
|
||||
|
||||
object DefaultRetryPolicy extends RetryPolicy
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ import com.datastax.driver.core.Session;
|
|||
|
||||
import info.archinnov.achilles.embedded.CassandraEmbeddedServerBuilder;
|
||||
|
||||
import org.apache.zeppelin.display.AngularObjectRegistry;
|
||||
import org.apache.zeppelin.interpreter.Interpreter;
|
||||
import org.apache.zeppelin.interpreter.InterpreterContext;
|
||||
import org.apache.zeppelin.interpreter.InterpreterResult;
|
||||
|
|
@ -45,7 +46,6 @@ import org.mockito.Answers;
|
|||
import org.mockito.Mock;
|
||||
import org.mockito.runners.MockitoJUnitRunner;
|
||||
|
||||
import java.io.BufferedInputStream;
|
||||
import java.io.BufferedReader;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
|
|
@ -63,7 +63,7 @@ public class CassandraInterpreterTest {
|
|||
.withScript("prepare_data.cql")
|
||||
.withProtocolVersion(ProtocolVersion.V3)
|
||||
.buildNativeSessionOnly();
|
||||
// public static Session session = null;
|
||||
|
||||
private static CassandraInterpreter interpreter;
|
||||
|
||||
@Mock(answer = Answers.RETURNS_DEEP_STUBS)
|
||||
|
|
@ -73,7 +73,7 @@ public class CassandraInterpreterTest {
|
|||
public static void setUp() {
|
||||
Properties properties = new Properties();
|
||||
final Cluster cluster = session.getCluster();
|
||||
// final Cluster cluster = null;
|
||||
|
||||
properties.setProperty(CASSANDRA_CLUSTER_NAME, cluster.getClusterName());
|
||||
properties.setProperty(CASSANDRA_COMPRESSION_PROTOCOL, "NONE");
|
||||
properties.setProperty(CASSANDRA_CREDENTIALS_USERNAME, "none");
|
||||
|
|
@ -289,6 +289,19 @@ public class CassandraInterpreterTest {
|
|||
assertThat(actual.code()).isEqualTo(Code.SUCCESS);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void should_execute_statement_with_request_timeout() throws Exception {
|
||||
//Given
|
||||
String statement = "@requestTimeOut=10000000\n" +
|
||||
"SELECT * FROM zeppelin.artists;";
|
||||
|
||||
//When
|
||||
final InterpreterResult actual = interpreter.interpret(statement, intrContext);
|
||||
|
||||
//Then
|
||||
assertThat(actual.code()).isEqualTo(Code.SUCCESS);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void should_execute_prepared_and_bound_statements() throws Exception {
|
||||
//Given
|
||||
|
|
@ -354,6 +367,8 @@ public class CassandraInterpreterTest {
|
|||
@Test
|
||||
public void should_extract_variable_from_statement() throws Exception {
|
||||
//Given
|
||||
AngularObjectRegistry angularObjectRegistry = new AngularObjectRegistry("cassandra", null);
|
||||
when(intrContext.getAngularObjectRegistry()).thenReturn(angularObjectRegistry);
|
||||
when(intrContext.getGui().input("login", "hsue")).thenReturn("hsue");
|
||||
when(intrContext.getGui().input("age", "27")).thenReturn("27");
|
||||
|
||||
|
|
|
|||
|
|
@ -31,6 +31,8 @@ import com.datastax.driver.core.ConsistencyLevel;
|
|||
import com.datastax.driver.core.Session;
|
||||
import com.datastax.driver.core.SimpleStatement;
|
||||
import com.datastax.driver.core.Statement;
|
||||
|
||||
import org.apache.zeppelin.display.AngularObjectRegistry;
|
||||
import org.apache.zeppelin.display.GUI;
|
||||
import org.apache.zeppelin.display.Input.ParamOption;
|
||||
import org.apache.zeppelin.interpreter.InterpreterContext;
|
||||
|
|
@ -101,6 +103,8 @@ public class InterpreterLogicTest {
|
|||
@Test
|
||||
public void should_extract_variable_and_default_value() throws Exception {
|
||||
//Given
|
||||
AngularObjectRegistry angularObjectRegistry = new AngularObjectRegistry("cassandra", null);
|
||||
when(intrContext.getAngularObjectRegistry()).thenReturn(angularObjectRegistry);
|
||||
when(intrContext.getGui().input("table", "zeppelin.demo")).thenReturn("zeppelin.demo");
|
||||
when(intrContext.getGui().input("id", "'John'")).thenReturn("'John'");
|
||||
|
||||
|
|
@ -114,6 +118,8 @@ public class InterpreterLogicTest {
|
|||
@Test
|
||||
public void should_extract_variable_and_choices() throws Exception {
|
||||
//Given
|
||||
AngularObjectRegistry angularObjectRegistry = new AngularObjectRegistry("cassandra", null);
|
||||
when(intrContext.getAngularObjectRegistry()).thenReturn(angularObjectRegistry);
|
||||
when(intrContext.getGui().select(eq("name"), eq("'Paul'"), optionsCaptor.capture())).thenReturn("'Jack'");
|
||||
|
||||
//When
|
||||
|
|
@ -141,6 +147,23 @@ public class InterpreterLogicTest {
|
|||
assertThat(actual).isEqualTo("SELECT * FROM zeppelin.demo");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void should_extract_variable_from_angular_object_registry() throws Exception {
|
||||
//Given
|
||||
AngularObjectRegistry angularObjectRegistry = new AngularObjectRegistry("cassandra", null);
|
||||
angularObjectRegistry.add("id", "from_angular_registry", "noteId", "paragraphId");
|
||||
when(intrContext.getAngularObjectRegistry()).thenReturn(angularObjectRegistry);
|
||||
when(intrContext.getNoteId()).thenReturn("noteId");
|
||||
when(intrContext.getParagraphId()).thenReturn("paragraphId");
|
||||
|
||||
//When
|
||||
final String actual = helper.maybeExtractVariables("SELECT * FROM zeppelin.demo WHERE id='{{id=John}}'", intrContext);
|
||||
|
||||
//Then
|
||||
assertThat(actual).isEqualTo("SELECT * FROM zeppelin.demo WHERE id='from_angular_registry'");
|
||||
verify(intrContext, never()).getGui();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void should_error_if_incorrect_variable_definition() throws Exception {
|
||||
//Given
|
||||
|
|
@ -203,6 +226,18 @@ public class InterpreterLogicTest {
|
|||
assertThat(actual.retryPolicy().get()).isSameAs(DowngradingRetryPolicy$.MODULE$);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void should_extract_request_timeout_option() throws Exception {
|
||||
//Given
|
||||
List<QueryParameters> options = Arrays.<QueryParameters>asList(new RequestTimeOut(100));
|
||||
|
||||
//When
|
||||
final CassandraQueryOptions actual = helper.extractQueryOptions(toScalaList(options));
|
||||
|
||||
//Then
|
||||
assertThat(actual.requestTimeOut().get()).isEqualTo(100);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void should_generate_simple_statement() throws Exception {
|
||||
//Given
|
||||
|
|
@ -211,6 +246,7 @@ public class InterpreterLogicTest {
|
|||
Option.<ConsistencyLevel>empty(),
|
||||
Option.empty(),
|
||||
Option.<RetryPolicy>empty(),
|
||||
Option.empty(),
|
||||
Option.empty());
|
||||
|
||||
//When
|
||||
|
|
@ -232,6 +268,7 @@ public class InterpreterLogicTest {
|
|||
Option.<ConsistencyLevel>empty(),
|
||||
Option.empty(),
|
||||
Option.<RetryPolicy>empty(),
|
||||
Option.empty(),
|
||||
Option.empty());
|
||||
|
||||
//When
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -182,6 +182,12 @@ class ParagraphParserTest extends FlatSpec
|
|||
parsed should matchPattern { case parser.Success(FetchSize(100), _) =>}
|
||||
}
|
||||
|
||||
"Parser" should "parse request timeout" in {
|
||||
val query:String ="@requestTimeOut=100"
|
||||
val parsed = parser.parseAll(parser.requestTimeOut, query)
|
||||
parsed should matchPattern { case parser.Success(RequestTimeOut(100), _) =>}
|
||||
}
|
||||
|
||||
"Parser" should "fails parsing invalid fetch size" in {
|
||||
val query:String =""" @fetchSize=TEST""".stripMargin
|
||||
val ex = intercept[InterpreterException] {
|
||||
|
|
@ -944,4 +950,15 @@ class ParagraphParserTest extends FlatSpec
|
|||
case parser.Success(List(SimpleStm(query)), _) =>
|
||||
}
|
||||
}
|
||||
|
||||
"Parser" should "parse ALTER KEYSPACE" in {
|
||||
val query = "ALTER KEYSPACE toto WITH replication = " +
|
||||
"{'class': 'SimpleStrategy', 'replication_factor': 1};"
|
||||
|
||||
val parsed = parser.parseAll(parser.queries, query)
|
||||
|
||||
parsed should matchPattern {
|
||||
case parser.Success(List(SimpleStm(query)), _) =>
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -178,7 +178,7 @@
|
|||
|
||||
<property>
|
||||
<name>zeppelin.interpreters</name>
|
||||
<value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,,org.apache.zeppelin.python.PythonInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter</value>
|
||||
<value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,,org.apache.zeppelin.python.PythonInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter</value>
|
||||
<description>Comma separated interpreter configurations. First interpreter become a default</description>
|
||||
</property>
|
||||
|
||||
|
|
|
|||
|
|
@ -23,8 +23,6 @@
|
|||
# Here's some helpful documents for the release
|
||||
# http://www.apache.org/dev/release.html
|
||||
# http://www.apache.org/dev/release-publishing
|
||||
# http://incubator.apache.org/guides/releasemanagement.html
|
||||
# http://incubator.apache.org/guides/release.html
|
||||
# http://www.apache.org/dev/release-signing.html
|
||||
# http://www.apache.org/dev/publishing-maven-artifacts.html
|
||||
|
||||
|
|
@ -38,7 +36,7 @@ fi
|
|||
|
||||
|
||||
if [[ -z "${WORKING_DIR}" ]]; then
|
||||
WORKING_DIR=/tmp/incubator-zeppelin-release
|
||||
WORKING_DIR=/tmp/zeppelin-release
|
||||
fi
|
||||
|
||||
if [[ -z "${GPG_PASSPHRASE}" ]]; then
|
||||
|
|
@ -49,7 +47,7 @@ fi
|
|||
|
||||
if [[ $# -ne 2 ]]; then
|
||||
echo "usage) $0 [Release name] [Branch or Tag]"
|
||||
echo " ex. $0 0.5.0-incubating branch-0.5"
|
||||
echo " ex. $0 0.6.0 branch-0.6"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
|
@ -66,7 +64,7 @@ mkdir ${WORKING_DIR}
|
|||
|
||||
echo "Cloning the source and packaging"
|
||||
# clone source
|
||||
git clone -b ${BRANCH} git@github.com:apache/incubator-zeppelin.git ${WORKING_DIR}/zeppelin
|
||||
git clone -b ${BRANCH} git@github.com:apache/zeppelin.git ${WORKING_DIR}/zeppelin
|
||||
if [[ $? -ne 0 ]]; then
|
||||
echo "Can not clone source repository"
|
||||
exit 1
|
||||
|
|
|
|||
|
|
@ -48,8 +48,8 @@ JIRA_USERNAME = os.environ.get("JIRA_USERNAME", "moon")
|
|||
# ASF JIRA password
|
||||
JIRA_PASSWORD = os.environ.get("JIRA_PASSWORD", "00000")
|
||||
|
||||
GITHUB_BASE = "https://github.com/apache/incubator-zeppelin/pull"
|
||||
GITHUB_API_BASE = "https://api.github.com/repos/apache/incubator-zeppelin"
|
||||
GITHUB_BASE = "https://github.com/apache/zeppelin/pull"
|
||||
GITHUB_API_BASE = "https://api.github.com/repos/apache/zeppelin"
|
||||
JIRA_BASE = "https://issues.apache.org/jira/browse"
|
||||
JIRA_API_BASE = "https://issues.apache.org/jira"
|
||||
# Prefix added to temporary branches
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ if len(sys.argv) == 1:
|
|||
|
||||
|
||||
pr=sys.argv[1]
|
||||
githubApi="https://api.github.com/repos/apache/incubator-zeppelin"
|
||||
githubApi="https://api.github.com/repos/apache/zeppelin"
|
||||
|
||||
prInfo = json.load(urllib.urlopen(githubApi + "/pulls/" + pr))
|
||||
if "message" in prInfo and prInfo["message"] == "Not Found":
|
||||
|
|
|
|||
|
|
@ -44,7 +44,7 @@ should be updated
|
|||
|
||||
2. checkout ASF repo
|
||||
```
|
||||
svn co https://svn.apache.org/repos/asf/incubator/zeppelin asf-zeppelin
|
||||
svn co https://svn.apache.org/repos/asf/zeppelin asf-zeppelin
|
||||
```
|
||||
3. copy `zeppelin/docs/_site` to `asf-zeppelin/site/docs/[VERSION]`
|
||||
4. ```svn commit```
|
||||
|
|
|
|||
|
|
@ -93,7 +93,6 @@
|
|||
<li><a href="{{BASE_PATH}}/rest-api/rest-configuration.html">Configuration API</a></li>
|
||||
<li role="separator" class="divider"></li>
|
||||
<!-- li><span><b>Security</b><span></li -->
|
||||
<li><a href="{{BASE_PATH}}/security/overview.html">Security Overview</a></li>
|
||||
<li><a href="{{BASE_PATH}}/security/authentication.html">Authentication for NGINX</a></li>
|
||||
<li><a href="{{BASE_PATH}}/security/shiroauthentication.html">Shiro Authentication</a></li>
|
||||
<li><a href="{{BASE_PATH}}/security/notebook_authorization.html">Notebook Authorization</a></li>
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 6.4 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 131 KiB |
BIN
docs/assets/themes/zeppelin/img/docs-img/permission_setting.png
Normal file
BIN
docs/assets/themes/zeppelin/img/docs-img/permission_setting.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 188 KiB |
|
|
@ -30,24 +30,24 @@ To build the code, install
|
|||
* Apache Maven
|
||||
|
||||
## Getting the source code
|
||||
First of all, you need Zeppelin source code. The official location of Zeppelin is [http://git.apache.org/incubator-zeppelin.git](http://git.apache.org/incubator-zeppelin.git).
|
||||
First of all, you need Zeppelin source code. The official location of Zeppelin is [http://git.apache.org/zeppelin.git](http://git.apache.org/zeppelin.git).
|
||||
|
||||
### git access
|
||||
|
||||
Get the source code on your development machine using git.
|
||||
|
||||
```
|
||||
git clone git://git.apache.org/incubator-zeppelin.git zeppelin
|
||||
git clone git://git.apache.org/zeppelin.git zeppelin
|
||||
```
|
||||
|
||||
You may also want to develop against a specific branch. For example, for branch-0.5.6
|
||||
|
||||
```
|
||||
git clone -b branch-0.5.6 git://git.apache.org/incubator-zeppelin.git zeppelin
|
||||
git clone -b branch-0.5.6 git://git.apache.org/zeppelin.git zeppelin
|
||||
```
|
||||
|
||||
#### Fork repository
|
||||
If you want not only build Zeppelin but also make any changes, then you need fork [Zeppelin github mirror repository](https://github.com/apache/incubator-zeppelin) and make a pull request.
|
||||
If you want not only build Zeppelin but also make any changes, then you need fork [Zeppelin github mirror repository](https://github.com/apache/zeppelin) and make a pull request.
|
||||
|
||||
###Build
|
||||
|
||||
|
|
|
|||
|
|
@ -17,22 +17,22 @@ Any contribution to Zeppelin (Source code, Documents, Image, Website) means you
|
|||
#### Getting the source code
|
||||
Website is hosted in 'master' branch under `/docs/` dir.
|
||||
|
||||
First of all, you need the website source code. The official location of mirror for Zeppelin is [http://git.apache.org/incubator-zeppelin.git](http://git.apache.org/incubator-zeppelin.git).
|
||||
First of all, you need the website source code. The official location of mirror for Zeppelin is [http://git.apache.org/zeppelin.git](http://git.apache.org/zeppelin.git).
|
||||
|
||||
Get the source code on your development machine using git.
|
||||
|
||||
```
|
||||
git clone git://git.apache.org/incubator-zeppelin.git
|
||||
git clone git://git.apache.org/zeppelin.git
|
||||
cd docs
|
||||
```
|
||||
|
||||
#### Build
|
||||
|
||||
To build, you'll need to install some prerequisites. Please check 'Build documentation' section in [docs/README.md](https://github.com/apache/incubator-zeppelin/blob/master/docs/README.md#build-documentation).
|
||||
To build, you'll need to install some prerequisites. Please check 'Build documentation' section in [docs/README.md](https://github.com/apache/zeppelin/blob/master/docs/README.md#build-documentation).
|
||||
|
||||
#### Run website in development mode
|
||||
|
||||
While you're modifying website, you'll want to see preview of it. Please check 'Run website' section in [docs/README.md](https://github.com/apache/incubator-zeppelin/blob/master/docs/README.md#run-website).
|
||||
While you're modifying website, you'll want to see preview of it. Please check 'Run website' section in [docs/README.md](https://github.com/apache/zeppelin/blob/master/docs/README.md#run-website).
|
||||
|
||||
You'll be able to access it on [http://localhost:4000](http://localhost:4000) with your web browser.
|
||||
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ In 'Separate Interpreter for each note' mode, new Interpreter instance will be c
|
|||
|
||||
### Make your own Interpreter
|
||||
|
||||
Creating a new interpreter is quite simple. Just extend [org.apache.zeppelin.interpreter](https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java) abstract class and implement some methods.
|
||||
Creating a new interpreter is quite simple. Just extend [org.apache.zeppelin.interpreter](https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java) abstract class and implement some methods.
|
||||
You can include `org.apache.zeppelin:zeppelin-interpreter:[VERSION]` artifact in your build system. And you should your jars under your interpreter directory with specific directory name. Zeppelin server reads interpreter directories recursively and initializes interpreters including your own interpreter.
|
||||
|
||||
There are three locations where you can store your interpreter group, name and other information. Zeppelin server tries to find the location below. Next, Zeppelin tries to find `interpareter-setting.json` in your interpreter jar.
|
||||
|
|
@ -126,14 +126,14 @@ To configure your interpreter you need to follow these steps:
|
|||
</property>
|
||||
```
|
||||
|
||||
2. Add your interpreter to the [default configuration](https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java#L397) which is used when there is no `zeppelin-site.xml`.
|
||||
2. Add your interpreter to the [default configuration](https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java#L397) which is used when there is no `zeppelin-site.xml`.
|
||||
|
||||
3. Start Zeppelin by running `./bin/zeppelin-daemon.sh start`.
|
||||
|
||||
4. In the interpreter page, click the `+Create` button and configure your interpreter properties.
|
||||
Now you are done and ready to use your interpreter.
|
||||
|
||||
Note that the interpreters released with zeppelin have a [default configuration](https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java#L397) which is used when there is no `conf/zeppelin-site.xml`.
|
||||
Note that the interpreters released with zeppelin have a [default configuration](https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java#L397) which is used when there is no `conf/zeppelin-site.xml`.
|
||||
|
||||
### Use your interpreter
|
||||
|
||||
|
|
@ -196,10 +196,10 @@ codes for myintp2
|
|||
|
||||
Checkout some interpreters released with Zeppelin by default.
|
||||
|
||||
- [spark](https://github.com/apache/incubator-zeppelin/tree/master/spark)
|
||||
- [markdown](https://github.com/apache/incubator-zeppelin/tree/master/markdown)
|
||||
- [shell](https://github.com/apache/incubator-zeppelin/tree/master/shell)
|
||||
- [hive](https://github.com/apache/incubator-zeppelin/tree/master/hive)
|
||||
- [spark](https://github.com/apache/zeppelin/tree/master/spark)
|
||||
- [markdown](https://github.com/apache/zeppelin/tree/master/markdown)
|
||||
- [shell](https://github.com/apache/zeppelin/tree/master/shell)
|
||||
- [jdbc](https://github.com/apache/zeppelin/tree/master/jdbc)
|
||||
|
||||
### Contributing a new Interpreter to Zeppelin releases
|
||||
|
||||
|
|
@ -207,9 +207,9 @@ We welcome contribution to a new interpreter. Please follow these few steps:
|
|||
|
||||
- First, check out the general contribution guide [here](./howtocontributewebsite.html).
|
||||
- Follow the steps in "Make your own Interpreter" section above.
|
||||
- Add your interpreter as in the "Configure your interpreter" section above; also add it to the example template [zeppelin-site.xml.template](https://github.com/apache/incubator-zeppelin/blob/master/conf/zeppelin-site.xml.template).
|
||||
- Add your interpreter as in the "Configure your interpreter" section above; also add it to the example template [zeppelin-site.xml.template](https://github.com/apache/zeppelin/blob/master/conf/zeppelin-site.xml.template).
|
||||
- Add tests! They are run by Travis for all changes and it is important that they are self-contained.
|
||||
- Include your interpreter as a module in [`pom.xml`](https://github.com/apache/incubator-zeppelin/blob/master/pom.xml).
|
||||
- Add documentation on how to use your interpreter under `docs/interpreter/`. Follow the Markdown style as this [example](https://github.com/apache/incubator-zeppelin/blob/master/docs/interpreter/elasticsearch.md). Make sure you list config settings and provide working examples on using your interpreter in code boxes in Markdown. Link to images as appropriate (images should go to `docs/assets/themes/zeppelin/img/docs-img/`). And add a link to your documentation in the navigation menu (`docs/_includes/themes/zeppelin/_navigation.html`).
|
||||
- Most importantly, ensure licenses of the transitive closure of all dependencies are list in [license file](https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-distribution/src/bin_license/LICENSE).
|
||||
- Commit your changes and open a Pull Request on the project [Mirror on GitHub](https://github.com/apache/incubator-zeppelin); check to make sure Travis CI build is passing.
|
||||
- Include your interpreter as a module in [`pom.xml`](https://github.com/apache/zeppelin/blob/master/pom.xml).
|
||||
- Add documentation on how to use your interpreter under `docs/interpreter/`. Follow the Markdown style as this [example](https://github.com/apache/zeppelin/blob/master/docs/interpreter/elasticsearch.md). Make sure you list config settings and provide working examples on using your interpreter in code boxes in Markdown. Link to images as appropriate (images should go to `docs/assets/themes/zeppelin/img/docs-img/`). And add a link to your documentation in the navigation menu (`docs/_includes/themes/zeppelin/_navigation.html`).
|
||||
- Most importantly, ensure licenses of the transitive closure of all dependencies are list in [license file](https://github.com/apache/zeppelin/blob/master/zeppelin-distribution/src/bin_license/LICENSE).
|
||||
- Commit your changes and open a Pull Request on the project [Mirror on GitHub](https://github.com/apache/zeppelin); check to make sure Travis CI build is passing.
|
||||
|
|
|
|||
|
|
@ -41,7 +41,7 @@ limitations under the License.
|
|||
### Multiple language backend
|
||||
|
||||
Zeppelin interpreter concept allows any language/data-processing-backend to be plugged into Zeppelin.
|
||||
Currently Zeppelin supports many interpreters such as Scala(with Apache Spark), Python(with Apache Spark), SparkSQL, Hive, Markdown and Shell.
|
||||
Currently Zeppelin supports many interpreters such as Scala(with Apache Spark), Python(with Apache Spark), SparkSQL, JDBC, Markdown and Shell.
|
||||
|
||||
<img class="img-responsive" src="/assets/themes/zeppelin/img/screenshots/multiple_language_backend.png" />
|
||||
|
||||
|
|
@ -116,7 +116,7 @@ If you want to learn more about this feature, please visit [this page](./manual/
|
|||
<br />
|
||||
### 100% Opensource
|
||||
|
||||
Apache Zeppelin is Apache2 Licensed software. Please check out the [source repository](http://git.apache.org/incubator-zeppelin.git) and [How to contribute](./development/howtocontribute.html)
|
||||
Apache Zeppelin is Apache2 Licensed software. Please check out the [source repository](http://git.apache.org/zeppelin.git) and [How to contribute](./development/howtocontribute.html)
|
||||
|
||||
Zeppelin has a very active development community.
|
||||
Join the [Mailing list](https://zeppelin.apache.org/community.html) and report issues on our [Issue tracker](https://issues.apache.org/jira/browse/ZEPPELIN).
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ You can also build Zeppelin from the source.
|
|||
* Maven(3.1.x or higher)
|
||||
* Node.js Package Manager
|
||||
|
||||
If you don't have requirements prepared, please check instructions in [README.md](https://github.com/apache/incubator-zeppelin/blob/master/README.md) for the details.
|
||||
If you don't have requirements prepared, please check instructions in [README.md](https://github.com/apache/zeppelin/blob/master/README.md) for the details.
|
||||
|
||||
<a name="zeppelin-configuration"> </a>
|
||||
## Zeppelin Configuration
|
||||
|
|
@ -232,7 +232,7 @@ You can configure Zeppelin with both **environment variables** in `conf/zeppelin
|
|||
<td>ZEPPELIN_INTERPRETERS</td>
|
||||
<td>zeppelin.interpreters</td>
|
||||
<description></description>
|
||||
<td>org.apache.zeppelin.spark.SparkInterpreter,<br />org.apache.zeppelin.spark.PySparkInterpreter,<br />org.apache.zeppelin.spark.SparkSqlInterpreter,<br />org.apache.zeppelin.spark.DepInterpreter,<br />org.apache.zeppelin.markdown.Markdown,<br />org.apache.zeppelin.shell.ShellInterpreter,<br />org.apache.zeppelin.hive.HiveInterpreter<br />
|
||||
<td>org.apache.zeppelin.spark.SparkInterpreter,<br />org.apache.zeppelin.spark.PySparkInterpreter,<br />org.apache.zeppelin.spark.SparkSqlInterpreter,<br />org.apache.zeppelin.spark.DepInterpreter,<br />org.apache.zeppelin.markdown.Markdown,<br />org.apache.zeppelin.shell.ShellInterpreter,<br />
|
||||
...
|
||||
</td>
|
||||
<td>Comma separated interpreter configurations [Class] <br /> The first interpreter will be a default value. <br /> It means only the first interpreter in this list can be available without <code>%interpreter_name</code> annotation in Zeppelin notebook paragraph. </td>
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ limitations under the License.
|
|||
{% include JB/setup %}
|
||||
|
||||
## Introduction
|
||||
This page describes how to pre-configure a bare metal node, configure Zeppelin and connect it to existing YARN cluster running Hortonworks flavour of Hadoop. It also describes steps to configure Spark & Hive interpreter of Zeppelin.
|
||||
This page describes how to pre-configure a bare metal node, configure Zeppelin and connect it to existing YARN cluster running Hortonworks flavour of Hadoop. It also describes steps to configure Spark interpreter of Zeppelin.
|
||||
|
||||
## Prepare Node
|
||||
|
||||
|
|
@ -53,7 +53,7 @@ Its assumed in the rest of the document that zeppelin user is indeed created and
|
|||
It's assumed that the node has CentOS 6.x installed on it. Although any version of Linux distribution should work fine.
|
||||
|
||||
#### Hadoop client
|
||||
Zeppelin can work with multiple versions & distributions of Hadoop. A complete list is available [here](https://github.com/apache/incubator-zeppelin#build). This document assumes Hadoop 2.7.x client libraries including configuration files are installed on Zeppelin node. It also assumes /etc/hadoop/conf contains various Hadoop configuration files. The location of Hadoop configuration files may vary, hence use appropriate location.
|
||||
Zeppelin can work with multiple versions & distributions of Hadoop. A complete list is available [here](https://github.com/apache/zeppelin#build). This document assumes Hadoop 2.7.x client libraries including configuration files are installed on Zeppelin node. It also assumes /etc/hadoop/conf contains various Hadoop configuration files. The location of Hadoop configuration files may vary, hence use appropriate location.
|
||||
|
||||
```bash
|
||||
hadoop version
|
||||
|
|
@ -67,7 +67,7 @@ This command was run using /usr/hdp/2.3.1.0-2574/hadoop/lib/hadoop-common-2.7.1.
|
|||
|
||||
#### Spark
|
||||
Spark is supported out of the box and to take advantage of this, you need to Download appropriate version of Spark binary packages from [Spark Download page](http://spark.apache.org/downloads.html) and unzip it.
|
||||
Zeppelin can work with multiple versions of Spark. A complete list is available [here](https://github.com/apache/incubator-zeppelin#build).
|
||||
Zeppelin can work with multiple versions of Spark. A complete list is available [here](https://github.com/apache/zeppelin#build).
|
||||
This document assumes Spark 1.6.0 is installed at /usr/lib/spark.
|
||||
> Note: Spark should be installed on the same node as Zeppelin.
|
||||
|
||||
|
|
@ -75,15 +75,15 @@ This document assumes Spark 1.6.0 is installed at /usr/lib/spark.
|
|||
|
||||
#### Zeppelin
|
||||
|
||||
Checkout source code from [git://git.apache.org/incubator-zeppelin.git](https://github.com/apache/incubator-zeppelin.git) or download binary package from [Download page](https://zeppelin.apache.org/download.html).
|
||||
Checkout source code from [git://git.apache.org/zeppelin.git](https://github.com/apache/zeppelin.git) or download binary package from [Download page](https://zeppelin.apache.org/download.html).
|
||||
You can refer [Install](install.html) page for the details.
|
||||
This document assumes that Zeppelin is located under `/home/zeppelin/incubator-zeppelin`.
|
||||
This document assumes that Zeppelin is located under `/home/zeppelin/zeppelin`.
|
||||
|
||||
## Zeppelin Configuration
|
||||
Zeppelin configuration needs to be modified to connect to YARN cluster. Create a copy of zeppelin environment shell script.
|
||||
|
||||
```bash
|
||||
cp /home/zeppelin/incubator-zeppelin/conf/zeppelin-env.sh.template /home/zeppelin/incubator-zeppelin/conf/zeppelin-env.sh
|
||||
cp /home/zeppelin/zeppelin/conf/zeppelin-env.sh.template /home/zeppelin/zeppelin/conf/zeppelin-env.sh
|
||||
```
|
||||
|
||||
Set the following properties
|
||||
|
|
@ -106,7 +106,7 @@ hdp-select status hadoop-client | sed 's/hadoop-client - \(.*\)/\1/'
|
|||
### Start Zeppelin
|
||||
|
||||
```
|
||||
cd /home/zeppelin/incubator-zeppelin
|
||||
cd /home/zeppelin/zeppelin
|
||||
bin/zeppelin-daemon.sh start
|
||||
```
|
||||
After successful start, visit http://[zeppelin-server-host-name]:8080 with your web browser.
|
||||
|
|
@ -118,16 +118,12 @@ bin/zeppelin-daemon.sh stop
|
|||
```
|
||||
|
||||
## Interpreter
|
||||
Zeppelin provides various distributed processing frameworks to process data that ranges from Spark, Hive, Tajo, Ignite and Lens to name a few. This document describes to configure Hive & Spark interpreters.
|
||||
Zeppelin provides various distributed processing frameworks to process data that ranges from Spark, JDBC, Tajo, Ignite and Lens to name a few. This document describes to configure JDBC & Spark interpreters.
|
||||
|
||||
### Hive
|
||||
Zeppelin supports Hive interpreter and hence copy hive-site.xml that should be present at /etc/hive/conf to the configuration folder of Zeppelin. Once Zeppelin is built it will have conf folder under /home/zeppelin/incubator-zeppelin.
|
||||
Zeppelin supports Hive through JDBC interpreter. You might need the information to use Hive and can find in your hive-site.xml
|
||||
|
||||
```bash
|
||||
cp /etc/hive/conf/hive-site.xml /home/zeppelin/incubator-zeppelin/conf
|
||||
```
|
||||
|
||||
Once Zeppelin server has started successfully, visit http://[zeppelin-server-host-name]:8080 with your web browser. Click on Interpreter tab next to Notebook dropdown. Look for Hive configurations and set them appropriately. By default hive.hiveserver2.url will be pointing to localhost and hive.hiveserver2.password/hive.hiveserver2.user are set to hive/hive. Set them as per Hive installation on YARN cluster.
|
||||
Once Zeppelin server has started successfully, visit http://[zeppelin-server-host-name]:8080 with your web browser. Click on Interpreter tab next to Notebook dropdown. Look for Hive configurations and set them appropriately. Set them as per Hive installation on YARN cluster.
|
||||
Click on Save button. Once these configurations are updated, Zeppelin will prompt you to restart the interpreter. Accept the prompt and the interpreter will reload the configurations.
|
||||
|
||||
### Spark
|
||||
|
|
@ -165,7 +161,7 @@ Zeppelin does not emit any kind of error messages on web interface when notebook
|
|||
|
||||
```bash
|
||||
[zeppelin@zeppelin-3529 logs]$ pwd
|
||||
/home/zeppelin/incubator-zeppelin/logs
|
||||
/home/zeppelin/zeppelin/logs
|
||||
[zeppelin@zeppelin-3529 logs]$ ls -l
|
||||
total 844
|
||||
-rw-rw-r-- 1 zeppelin zeppelin 14648 Aug 3 14:45 zeppelin-interpreter-hive-zeppelin-zeppelin-3529.log
|
||||
|
|
|
|||
|
|
@ -100,6 +100,7 @@ Each statement should be separated by a semi-colon ( **;** ) except the special
|
|||
6. @timestamp
|
||||
7. @retryPolicy
|
||||
8. @fetchSize
|
||||
9. @requestTimeOut
|
||||
|
||||
Multi-line statements as well as multiple statements on the same line are also supported as long as they are separated by a semi-colon. Ex:
|
||||
|
||||
|
|
@ -145,6 +146,15 @@ The complete list of all CQL statements and versions can be found below:
|
|||
<th>Cassandra Version</th>
|
||||
<th>Documentation Link</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>3.x</strong></td>
|
||||
<td>
|
||||
<a target="_blank"
|
||||
href="http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html">
|
||||
http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>2.2</strong></td>
|
||||
<td>
|
||||
|
|
@ -333,6 +343,11 @@ Below is the list of all parameters:
|
|||
<td><strong>@fetchSize=<em>integer value</em></strong></td>
|
||||
<td>Apply the given fetch size to all queries in the paragraph</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td nowrap>Request Time Out</td>
|
||||
<td><strong>@requestTimeOut=<em>integer value</em></strong></td>
|
||||
<td>Apply the given request timeout <strong>in millisecs</strong> to all queries in the paragraph</td>
|
||||
</tr>
|
||||
</table>
|
||||
</center>
|
||||
|
||||
|
|
@ -379,19 +394,19 @@ CREATE TABLE IF NOT EXISTS spark_demo.ts(
|
|||
);
|
||||
TRUNCATE spark_demo.ts;
|
||||
|
||||
# Timestamp in the past
|
||||
// Timestamp in the past
|
||||
@timestamp=10
|
||||
|
||||
# Force timestamp directly in the first insert
|
||||
// Force timestamp directly in the first insert
|
||||
INSERT INTO spark_demo.ts(key,value) VALUES(1,'first insert') USING TIMESTAMP 100;
|
||||
|
||||
# Select some data to make the clock turn
|
||||
// Select some data to make the clock turn
|
||||
SELECT * FROM spark_demo.albums LIMIT 100;
|
||||
|
||||
# Now insert using the timestamp parameter set at the beginning(10)
|
||||
// Now insert using the timestamp parameter set at the beginning(10)
|
||||
INSERT INTO spark_demo.ts(key,value) VALUES(1,'second insert');
|
||||
|
||||
# Check for the result. You should see 'first insert'
|
||||
// Check for the result. You should see 'first insert'
|
||||
SELECT value FROM spark_demo.ts WHERE key=1;
|
||||
```
|
||||
|
||||
|
|
@ -415,25 +430,25 @@ This interpreter provides 3 commands to handle prepared and bound statements:
|
|||
Example:
|
||||
|
||||
```
|
||||
@prepare[statement_name]=...
|
||||
@prepare[statement-name]=...
|
||||
|
||||
@bind[statement_name]=’text’, 1223, ’2015-07-30 12:00:01’, null, true, [‘list_item1’, ’list_item2’]
|
||||
@bind[statement-name]=’text’, 1223, ’2015-07-30 12:00:01’, null, true, [‘list_item1’, ’list_item2’]
|
||||
|
||||
@bind[statement_name_with_no_bound_value]
|
||||
@bind[statement-name-with-no-bound-value]
|
||||
|
||||
@remove_prepare[statement_name]
|
||||
@remove_prepare[statement-name]
|
||||
```
|
||||
|
||||
#### @prepare
|
||||
|
||||
You can use the syntax _"@prepare[statement_name]=SELECT ..."_ to create a prepared statement.
|
||||
The _statement_name_ is **mandatory** because the interpreter prepares the given statement with the Java driver and
|
||||
saves the generated prepared statement in an **internal hash map**, using the provided _statement_name_ as search key.
|
||||
You can use the syntax _"@prepare[statement-name]=SELECT..."_ to create a prepared statement.
|
||||
The _statement-name_ is **mandatory** because the interpreter prepares the given statement with the Java driver and
|
||||
saves the generated prepared statement in an **internal hash map**, using the provided _statement-name_ as search key.
|
||||
|
||||
> Please note that this internal prepared statement map is shared with **all notebooks** and **all paragraphs** because
|
||||
there is only one instance of the interpreter for Cassandra
|
||||
|
||||
> If the interpreter encounters **many** @prepare for the **same _statement_name_ (key)**, only the **first** statement will be taken into account.
|
||||
> If the interpreter encounters **many** @prepare for the **same _statement-name_ (key)**, only the **first** statement will be taken into account.
|
||||
|
||||
Example:
|
||||
|
||||
|
|
@ -443,8 +458,8 @@ Example:
|
|||
@prepare[select]=SELECT * FROM spark_demo.artists LIMIT ?
|
||||
```
|
||||
|
||||
For the above example, the prepared statement is _SELECT * FROM spark_demo.albums LIMIT ?_.
|
||||
_SELECT * FROM spark_demo.artists LIMIT ?_ is ignored because an entry already exists in the prepared statements map with the key select.
|
||||
For the above example, the prepared statement is `SELECT * FROM spark_demo.albums LIMIT ?`.
|
||||
`SELECT * FROM spark_demo.artists LIMIT ? is ignored because an entry already exists in the prepared statements map with the key select.
|
||||
|
||||
In the context of **Zeppelin**, a notebook can be scheduled to be executed at regular interval,
|
||||
thus it is necessary to **avoid re-preparing many time the same statement (considered an anti-pattern)**.
|
||||
|
|
@ -483,18 +498,22 @@ Bound values are not mandatory for the **@bind** statement. However if you provi
|
|||
#### @remove_prepare
|
||||
|
||||
To avoid for a prepared statement to stay forever in the prepared statement map, you can use the
|
||||
**@remove_prepare[statement_name]** syntax to remove it.
|
||||
**@remove_prepare[statement-name]** syntax to remove it.
|
||||
Removing a non-existing prepared statement yields no error.
|
||||
|
||||
## Using Dynamic Forms
|
||||
|
||||
Instead of hard-coding your CQL queries, it is possible to use the mustache syntax ( **\{\{ \}\}** ) to inject simple value or multiple choices forms.
|
||||
Instead of hard-coding your CQL queries, it is possible to use **[Zeppelin dynamic form]** syntax to inject simple value or multiple choices forms.
|
||||
|
||||
The syntax for simple parameter is: **\{\{input_Label=default value\}\}**. The default value is mandatory because the first time the paragraph is executed,
|
||||
we launch the CQL query before rendering the form so at least one value should be provided.
|
||||
The legacy mustache syntax ( **\{\{ \}\}** ) to bind input text and select form is still supported but is deprecated and will be removed in future releases.
|
||||
|
||||
> **Legacy**
|
||||
> The syntax for simple parameter is: **\{\{input_Label=default value\}\}**. The default value is mandatory because the first time the paragraph is executed,
|
||||
> we launch the CQL query before rendering the form so at least one value should be provided.
|
||||
>
|
||||
> The syntax for multiple choices parameter is: **\{\{input_Label=value1 | value2 | … | valueN \}\}**. By default the first choice is used for CQL query
|
||||
> the first time the paragraph is executed.
|
||||
|
||||
The syntax for multiple choices parameter is: **\{\{input_Label=value1 | value2 | … | valueN \}\}**. By default the first choice is used for CQL query
|
||||
the first time the paragraph is executed.
|
||||
|
||||
Example:
|
||||
|
||||
|
|
@ -502,22 +521,22 @@ Example:
|
|||
#Secondary index on performer style
|
||||
SELECT name, country, performer
|
||||
FROM spark_demo.performers
|
||||
WHERE name='{{performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}}'
|
||||
AND styles CONTAINS '{{style=Rock}}';
|
||||
WHERE name='${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}'
|
||||
AND styles CONTAINS '${style=Rock}';
|
||||
{% endraw %}
|
||||
|
||||
|
||||
In the above example, the first CQL query will be executed for _performer='Sheryl Crow' AND style='Rock'_.
|
||||
For subsequent queries, you can change the value directly using the form.
|
||||
|
||||
> Please note that we enclosed the **\{\{ \}\}** block between simple quotes ( **'** ) because Cassandra expects a String here.
|
||||
> We could have also use the **\{\{style='Rock'\}\}** syntax but this time, the value displayed on the form is **_'Rock'_** and not **_Rock_**.
|
||||
> Please note that we enclosed the **$\{ \}** block between simple quotes ( **'** ) because Cassandra expects a String here.
|
||||
> We could have also use the **$\{style='Rock'\}** syntax but this time, the value displayed on the form is **_'Rock'_** and not **_Rock_**.
|
||||
|
||||
It is also possible to use dynamic forms for **prepared statements**:
|
||||
|
||||
{% raw %}
|
||||
|
||||
@bind[select]=='{{performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}}', '{{style=Rock}}'
|
||||
@bind[select]=='${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}', '${style=Rock}'
|
||||
|
||||
{% endraw %}
|
||||
|
||||
|
|
@ -527,39 +546,26 @@ It is possible to execute many paragraphs in parallel. However, at the back-end
|
|||
_Asynchronous execution_ is only possible when it is possible to return a `Future` value in the `InterpreterResult`.
|
||||
It may be an interesting proposal for the **Zeppelin** project.
|
||||
|
||||
Another caveat is that the same `com.datastax.driver.core.Session` object is used for **all** notebooks and paragraphs.
|
||||
Recently, **Zeppelin** allows you to choose the level of isolation for your interpreters (see **[Interpreter Binding Mode]** ).
|
||||
|
||||
Long story short, you have 3 available bindings:
|
||||
|
||||
- **shared** : _same JVM_ and _same Interpreter instance_ for all notes
|
||||
- **scoped** : _same JVM_ but _different Interpreter instances_, one for each note
|
||||
- **isolated**: _different JVM_ running a _single Interpreter instance_, one JVM for each note
|
||||
|
||||
Using the **shared** binding, the same `com.datastax.driver.core.Session` object is used for **all** notes and paragraphs.
|
||||
Consequently, if you use the **USE _keyspace name_;** statement to log into a keyspace, it will change the keyspace for
|
||||
**all current users** of the **Cassandra** interpreter because we only create 1 `com.datastax.driver.core.Session` object
|
||||
per instance of **Cassandra** interpreter.
|
||||
|
||||
The same remark does apply to the **prepared statement hash map**, it is shared by **all users** using the same instance of **Cassandra** interpreter.
|
||||
|
||||
Until **Zeppelin** offers a real multi-users separation, there is a work-around to segregate user environment and states:
|
||||
_create different **Cassandra** interpreter instances_
|
||||
When using **scoped** binding, in the _same JVM_ **Zeppelin** will create multiple instances of the Cassandra interpreter, thus
|
||||
multiple `com.datastax.driver.core.Session` objects. **Beware of resource and memory usage using this binding !**
|
||||
|
||||
For this, first go to the **Interpreter** menu and click on the **Create** button
|
||||
<center>
|
||||
|
||||
</center>
|
||||
The **isolated** mode is the most extreme and will create as many JVM/`com.datastax.driver.core.Session` object as there are distinct notes.
|
||||
|
||||
In the interpreter creation form, put **cass-instance2** as **Name** and select the **cassandra**
|
||||
in the interpreter drop-down list
|
||||
<center>
|
||||
|
||||
</center>
|
||||
|
||||
Click on **Save** to create the new interpreter instance. Now you should be able to see it in the interpreter list.
|
||||
<center>
|
||||
|
||||
</center>
|
||||
|
||||
Go back to your notebook and click on the **Gear** icon to configure interpreter bindings.
|
||||
You should be able to see and select the **cass-instance2** interpreter instance in the available
|
||||
interpreter list instead of the standard **cassandra** instance.
|
||||
|
||||
<center>
|
||||
|
||||
</center>
|
||||
|
||||
## Interpreter Configuration
|
||||
|
||||
|
|
@ -694,7 +700,7 @@ Below are the configuration parameters and their default value.
|
|||
<tr>
|
||||
<td>cassandra.protocol.version</td>
|
||||
<td>Cassandra binary protocol version</td>
|
||||
<td>3</td>
|
||||
<td>4</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>cassandra.query.default.consistency</td>
|
||||
|
|
@ -771,12 +777,28 @@ Below are the configuration parameters and their default value.
|
|||
|
||||
## Change Log
|
||||
|
||||
**3.0** _(Zeppelin {{ site.ZEPPELIN_VERSION }})_ :
|
||||
|
||||
* Update documentation
|
||||
* Update interactive documentation
|
||||
* Add support for binary protocol **V4**
|
||||
* Implement new `@requestTimeOut` runtime option
|
||||
* Upgrade Java driver version to **3.0.1**
|
||||
* Allow interpreter to add dynamic forms programmatically when using FormType.SIMPLE
|
||||
* Allow dynamic form using default Zeppelin syntax
|
||||
* Fixing typo on FallThroughPolicy
|
||||
* Look for data in AngularObjectRegistry before creating dynamic form
|
||||
* Add missing support for `ALTER` statements
|
||||
|
||||
|
||||
**2.0** _(Zeppelin {{ site.ZEPPELIN_VERSION }})_ :
|
||||
|
||||
* Update help menu and add changelog
|
||||
* Add Support for **User Defined Functions**, **User Defined Aggregates** and **Materialized Views**
|
||||
* Upgrade Java driver version to **3.0.0-rc1**
|
||||
|
||||
**1.0** _(Zeppelin 0.5.5-incubating)_ :
|
||||
|
||||
* Initial version
|
||||
|
||||
## Bugs & Contacts
|
||||
|
|
@ -789,5 +811,7 @@ Below are the configuration parameters and their default value.
|
|||
[standard CQL syntax]: http://docs.datastax.com/en/cql/3.1/cql/cql_using/use_collections_c.html
|
||||
[Tuple CQL syntax]: http://docs.datastax.com/en/cql/3.1/cql/cql_reference/tupleType.html
|
||||
[UDT CQL syntax]: http://docs.datastax.com/en/cql/3.1/cql/cql_using/cqlUseUDT.html
|
||||
[Zeppelin dynamic form]: http://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/manual/dynamicform.html
|
||||
[Interpreter Binding Mode]: http://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/manual/interpreters.html
|
||||
[JIRA]: https://issues.apache.org/jira/browse/ZEPPELIN-382?jql=project%20%3D%20ZEPPELIN
|
||||
[@doanduyhai]: https://twitter.com/doanduyhai
|
||||
|
|
|
|||
|
|
@ -9,6 +9,51 @@ group: manual
|
|||
## Hive Interpreter for Apache Zeppelin
|
||||
The [Apache Hive](https://hive.apache.org/) ™ data warehouse software facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL.
|
||||
|
||||
## Important Notice
|
||||
Hive Interpreter will be deprecated and merged into JDBC Interpreter. You can use Hive Interpreter by using JDBC Interpreter with same functionality. See the example below of settings and dependencies.
|
||||
|
||||
### Properties
|
||||
<table class="table-configuration">
|
||||
<tr>
|
||||
<th>Property</th>
|
||||
<th>Value</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>hive.driver</td>
|
||||
<td>org.apache.hive.jdbc.HiveDriver</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>hive.url</td>
|
||||
<td>jdbc:hive2://localhost:10000</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>hive.user</td>
|
||||
<td>hiveUser</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>hive.password</td>
|
||||
<td>hivePassword</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
### Dependencies
|
||||
<table class="table-configuration">
|
||||
<tr>
|
||||
<th>Artifact</th>
|
||||
<th>Exclude</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>org.apache.hive:hive-jdbc:0.14.0</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>org.apache.hadoop:hadoop-common:2.6.0</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
----
|
||||
|
||||
### Configuration
|
||||
<table class="table-configuration">
|
||||
<tr>
|
||||
|
|
|
|||
|
|
@ -195,6 +195,47 @@ To develop this functionality use this [method](http://docs.oracle.com/javase/7/
|
|||
</tr>
|
||||
</table>
|
||||
|
||||
### Examples
|
||||
#### Hive
|
||||
##### Properties
|
||||
<table class="table-configuration">
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Value</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>hive.driver</td>
|
||||
<td>org.apache.hive.jdbc.HiveDriver</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>hive.url</td>
|
||||
<td>jdbc:hive2://localhost:10000</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>hive.user</td>
|
||||
<td>hive_user</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>hive.password</td>
|
||||
<td>hive_password</td>
|
||||
</tr>
|
||||
</table>
|
||||
##### Dependencies
|
||||
<table class="table-configuration">
|
||||
<tr>
|
||||
<th>Artifact</th>
|
||||
<th>Excludes</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>org.apache.hive:hive-jdbc:0.14.0</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>org.apache.hadoop:hadoop-common:2.6.0</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
### How to use
|
||||
|
||||
#### Reference in paragraph
|
||||
|
|
|
|||
|
|
@ -28,10 +28,49 @@ In a notebook, to enable the **Scalding** interpreter, click on the **Gear** ico
|
|||
</center>
|
||||
|
||||
### Configuring the Interpreter
|
||||
Zeppelin comes with a pre-configured Scalding interpreter in local mode, so you do not need to install anything.
|
||||
|
||||
Scalding interpreter runs in two modes:
|
||||
|
||||
* local
|
||||
* hdfs
|
||||
|
||||
In the local mode, you can access files on the local server and scalding transformation are done locally.
|
||||
|
||||
In hdfs mode you can access files in HDFS and scalding transformation are run as hadoop map-reduce jobs.
|
||||
|
||||
Zeppelin comes with a pre-configured Scalding interpreter in local mode.
|
||||
|
||||
To run the scalding interpreter in the hdfs mode you have to do the following:
|
||||
|
||||
**Set the classpath with ZEPPELIN\_CLASSPATH\_OVERRIDES**
|
||||
|
||||
In conf/zeppelin_env.sh, you have to set
|
||||
ZEPPELIN_CLASSPATH_OVERRIDES to the contents of 'hadoop classpath'
|
||||
and directories with custom jar files you need for your scalding commands.
|
||||
|
||||
**Set arguments to the scalding repl**
|
||||
|
||||
The default arguments are: "--local --repl"
|
||||
|
||||
For hdfs mode you need to add: "--hdfs --repl"
|
||||
|
||||
If you want to add custom jars, you need to add:
|
||||
"-libjars directory/*:directory/*"
|
||||
|
||||
For reducer estimation, you need to add something like:
|
||||
"-Dscalding.reducer.estimator.classes=com.twitter.scalding.reducer_estimation.InputSizeReducerEstimator"
|
||||
|
||||
**Set max.open.instances**
|
||||
|
||||
If you want to control the maximum number of open interpreters, you have to select "scoped" interpreter for note
|
||||
option and set max.open.instances argument.
|
||||
|
||||
### Testing the Interpreter
|
||||
In example, by using the [Alice in Wonderland](https://gist.github.com/johnynek/a47699caa62f4f38a3e2) tutorial, we will count words (of course!), and plot a graph of the top 10 words in the book.
|
||||
|
||||
#### Local mode
|
||||
|
||||
In example, by using the [Alice in Wonderland](https://gist.github.com/johnynek/a47699caa62f4f38a3e2) tutorial,
|
||||
we will count words (of course!), and plot a graph of the top 10 words in the book.
|
||||
|
||||
```
|
||||
%scalding
|
||||
|
|
@ -71,7 +110,44 @@ print("%table " + table)
|
|||
If you click on the icon for the pie chart, you should be able to see a chart like this:
|
||||
|
||||
|
||||
### Current Status & Future Work
|
||||
The current implementation of the Scalding interpreter does not support canceling jobs, or fine-grained progress updates.
|
||||
|
||||
The pre-configured Scalding interpreter only supports Scalding in local mode. Hadoop mode for Scalding is currently unsupported, and will be future work (contributions welcome!).
|
||||
#### HDFS mode
|
||||
|
||||
**Test mode**
|
||||
|
||||
```
|
||||
%scalding
|
||||
mode
|
||||
```
|
||||
This command should print:
|
||||
|
||||
```
|
||||
res4: com.twitter.scalding.Mode = Hdfs(true,Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml)
|
||||
```
|
||||
|
||||
|
||||
**Test HDFS read**
|
||||
|
||||
```
|
||||
val testfile = TypedPipe.from(TextLine("/user/x/testfile"))
|
||||
testfile.dump
|
||||
```
|
||||
|
||||
This command should print the contents of the hdfs file /user/x/testfile.
|
||||
|
||||
**Test map-reduce job**
|
||||
|
||||
```
|
||||
val testfile = TypedPipe.from(TextLine("/user/x/testfile"))
|
||||
val a = testfile.groupAll.size.values
|
||||
a.toList
|
||||
|
||||
```
|
||||
|
||||
This command should create a map reduce job.
|
||||
|
||||
### Future Work
|
||||
* Better user feedback (hadoop url, progress updates)
|
||||
* Ability to cancel jobs
|
||||
* Ability to dynamically load jars without restarting the interpreter
|
||||
* Multiuser scalability (run scalding interpreters on different servers)
|
||||
|
|
|
|||
|
|
@ -56,7 +56,7 @@ http://127.0.0.1:8080/api/interpreter/load/md/markdown
|
|||
|
||||
```
|
||||
{
|
||||
"artifact": "org.apache.zeppelin:zeppelin-markdown:0.6.0-incubating-SNAPSHOT",
|
||||
"artifact": "org.apache.zeppelin:zeppelin-markdown:0.6.0-SNAPSHOT",
|
||||
"className": "org.apache.zeppelin.markdown.Markdown",
|
||||
"repository": {
|
||||
"url": "http://dl.bintray.com/spark-packages/maven",
|
||||
|
|
@ -71,7 +71,7 @@ The meaning of each parameters is:
|
|||
1. **Artifact**
|
||||
- groupId: org.apache.zeppelin
|
||||
- artifactId: zeppelin-markdown
|
||||
- version: 0.6.0-incubating-SNAPSHOT
|
||||
- version: 0.6.0-SNAPSHOT
|
||||
|
||||
2. **Class Name**
|
||||
- Package Name: org.apache.zeppelin
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ limitations under the License.
|
|||
## Interpreters in Zeppelin
|
||||
In this section, we will explain about the role of interpreters, interpreters group and interpreter settings in Zeppelin.
|
||||
The concept of Zeppelin interpreter allows any language/data-processing-backend to be plugged into Zeppelin.
|
||||
Currently, Zeppelin supports many interpreters such as Scala ( with Apache Spark ), Python ( with Apache Spark ), SparkSQL, Hive, Markdown, Shell and so on.
|
||||
Currently, Zeppelin supports many interpreters such as Scala ( with Apache Spark ), Python ( with Apache Spark ), SparkSQL, JDBC, Markdown, Shell and so on.
|
||||
|
||||
## What is Zeppelin interpreter?
|
||||
Zeppelin Interpreter is a plug-in which enables Zeppelin users to use a specific language/data-processing-backend. For example, to use Scala code in Zeppelin, you need `%spark` interpreter.
|
||||
|
|
|
|||
|
|
@ -119,7 +119,7 @@ This instruction based on Ubuntu 14.04 LTS but may work with other OS with few c
|
|||
1. More security consideration
|
||||
|
||||
* Using HTTPS connection with Basic Authentication is highly recommended since basic auth without encryption may expose your important credential information over the network.
|
||||
* Using [Shiro Security feature built-into Zeppelin](https://github.com/apache/incubator-zeppelin/blob/master/SECURITY-README.md) is recommended if you prefer all-in-one solution for authentication but NGINX may provides ad-hoc solution for re-use authentication served by your system's NGINX server or in case of you need to separate authentication from zeppelin server.
|
||||
* Using [Shiro Security feature built-into Zeppelin](https://github.com/apache/zeppelin/blob/master/SECURITY-README.md) is recommended if you prefer all-in-one solution for authentication but NGINX may provides ad-hoc solution for re-use authentication served by your system's NGINX server or in case of you need to separate authentication from zeppelin server.
|
||||
* It is recommended to isolate direct connection to Zeppelin server from public internet or external services to secure your zeppelin instance from unexpected attack or problems caused by public zone.
|
||||
|
||||
### Another option
|
||||
|
|
|
|||
|
|
@ -27,10 +27,10 @@ Interpreter authorization involves permissions like creating an interpreter and
|
|||
|
||||
Data source authorization involves authenticating to the data source like a Mysql database and letting it determine user permissions.
|
||||
|
||||
For the Hive interpreter, we need to maintain per-user connection pools.
|
||||
For the JDBC interpreter, we need to maintain per-user connection pools.
|
||||
The interpret method takes the user string as parameter and executes the jdbc call using a connection in the user's connection pool.
|
||||
|
||||
In case of Presto, we don't need password if the Presto DB server runs backend code using HDFS authorization for the user.
|
||||
For databases like Vertica and Mysql we have to store password information for users.
|
||||
|
||||
The Credentials tab in the navbar allows users to save credentials for data sources which are passed to interpreters.
|
||||
The Credentials tab in the navbar allows users to save credentials for data sources which are passed to interpreters.
|
||||
|
|
|
|||
|
|
@ -17,21 +17,39 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
# Notebook Authorization
|
||||
# Zeppelin Notebook Authorization
|
||||
|
||||
We assume that there is an authentication component that associates a user string and a set of group strings with every NotebookSocket.
|
||||
We assume that there is an **Shiro Authentication** component that associates a user string and a set of group strings with every NotebookSocket.
|
||||
If you don't set the authentication components yet, please check [Shiro authentication for Apache Zeppelin](./shiroauthentication.html) first.
|
||||
|
||||
Each note has the following:
|
||||
* set of owner entities (users or groups)
|
||||
* set of reader entities (users or groups)
|
||||
* set of writer entities (users or groups)
|
||||
## Authorization Setting
|
||||
You can set Zeppelin notebook permissions in each notebooks. Of course only **notebook owners** can change this configuration.
|
||||
Just click **Lock icon** and open the permission setting page in your notebook.
|
||||
|
||||
If a set is empty, it means that any user can perform that operation.
|
||||
As you can see, each Zeppelin notebooks has 3 entities :
|
||||
|
||||
The NotebookServer classifies every Note operation into three categories: read, write, manage.
|
||||
Before executing a Note operation, it checks if the user and the groups associated with the NotebookSocket have permissions. For example, before executing an read
|
||||
operation, it checks if the user and the groups have at least one entity that belongs to the reader entities.
|
||||
* Owners ( users or groups )
|
||||
* Readers ( users or groups )
|
||||
* Writers ( users or groups )
|
||||
|
||||
To initialize and modify note permissions, we provide UI like "Interpreter binding". The user inputs comma separated entities for owners, readers and writers.
|
||||
We execute a rest api call with this information. In the backend we get the user information for the connection and allow the operation if the user and groups
|
||||
associated with the current user have at least one entity that belongs to owner entities for the note.
|
||||
<center><img src="../assets/themes/zeppelin/img/docs-img/permission_setting.png"></center>
|
||||
|
||||
Fill out the each forms with comma seperated **users** and **groups** configured in `conf/shiro.ini` file.
|
||||
If the form is empty (*), it means that any users can perform that operation.
|
||||
|
||||
If someone who doesn't have **read** permission is trying to access the notebook or someone who doesn't have **write** permission is trying to edit the notebook, Zeppelin will ask to login or block the user.
|
||||
|
||||
<center><img src="../assets/themes/zeppelin/img/docs-img/insufficient_privileges.png"></center>
|
||||
|
||||
## How it works
|
||||
In this section, we will explain the detail about how the notebook authorization works in backend side.
|
||||
|
||||
#### NotebookServer
|
||||
The [NotebookServer](https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java) classifies every notebook operations into three categories: **Read**, **Write**, **Manage**.
|
||||
Before executing a notebook operation, it checks if the user and the groups associated with the `NotebookSocket` have permissions.
|
||||
For example, before executing a **Read** operation, it checks if the user and the groups have at least one entity that belongs to the **Reader** entities.
|
||||
|
||||
#### Notebook REST API call
|
||||
Zeppelin executes a [REST API call](https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java) for the notebook permission information.
|
||||
In the backend side, Zeppelin gets the user information for the connection and allows the operation if the users and groups
|
||||
associated with the current user have at least one entity that belongs to owner entities for the notebook.
|
||||
|
|
|
|||
|
|
@ -1,28 +0,0 @@
|
|||
---
|
||||
layout: page
|
||||
title: "Security Overview"
|
||||
description: "Security Overview"
|
||||
group: security
|
||||
---
|
||||
<!--
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
{% include JB/setup %}
|
||||
|
||||
# Security Overview
|
||||
|
||||
There are three aspects to Zeppelin security:
|
||||
|
||||
* Authentication: is the user who they say they are? [More](authentication.html)
|
||||
* Notebook authorization: does the user have permissions to read or write to a note? [More](notebook_authorization.html)
|
||||
* Interpreter and data source authorization: does the user have permissions to perform interpreter operations or access data source objects? [More](interpreter_authorization.html)
|
||||
|
|
@ -101,4 +101,4 @@ group1 = *
|
|||
|
||||
All of above configurations are defined in the `conf/shiro.ini` file.
|
||||
|
||||
> **NOTE :** This documentation is originally from [SECURITY-README.md](https://github.com/apache/incubator-zeppelin/blob/master/SECURITY-README.md).
|
||||
> **NOTE :** This documentation is originally from [SECURITY-README.md](https://github.com/apache/zeppelin/blob/master/SECURITY-README.md).
|
||||
|
|
|
|||
|
|
@ -159,6 +159,7 @@
|
|||
<exclude>**/.project</exclude>
|
||||
<exclude>**/target/**</exclude>
|
||||
<exclude>**/README.md</exclude>
|
||||
<exclude>**/interpreter-setting.json</exclude>
|
||||
<exclude>dependency-reduced-pom.xml</exclude>
|
||||
</excludes>
|
||||
</configuration>
|
||||
|
|
|
|||
|
|
@ -67,19 +67,6 @@ public class FlinkInterpreter extends Interpreter {
|
|||
super(property);
|
||||
}
|
||||
|
||||
static {
|
||||
Interpreter.register(
|
||||
"flink",
|
||||
"flink",
|
||||
FlinkInterpreter.class.getName(),
|
||||
new InterpreterPropertyBuilder()
|
||||
.add("host", "local",
|
||||
"host name of running JobManager. 'local' runs flink in local mode")
|
||||
.add("port", "6123", "port of running JobManager")
|
||||
.build()
|
||||
);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void open() {
|
||||
out = new ByteArrayOutputStream();
|
||||
|
|
|
|||
21
flink/src/main/resources/interpreter-setting.json
Normal file
21
flink/src/main/resources/interpreter-setting.json
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
[
|
||||
{
|
||||
"group": "flink",
|
||||
"name": "flink",
|
||||
"className": "org.apache.zeppelin.flink.FlinkInterpreter",
|
||||
"properties": {
|
||||
"host": {
|
||||
"envName": "host",
|
||||
"propertyName": null,
|
||||
"defaultValue": "local",
|
||||
"description": "host name of running JobManager. 'local' runs flink in local mode."
|
||||
},
|
||||
"port": {
|
||||
"envName": "port",
|
||||
"propertyName": null,
|
||||
"defaultValue": "6123",
|
||||
"description": "port of running JobManager."
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
165
hive/pom.xml
165
hive/pom.xml
|
|
@ -1,165 +0,0 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!--
|
||||
~ Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
~ contributor license agreements. See the NOTICE file distributed with
|
||||
~ this work for additional information regarding copyright ownership.
|
||||
~ The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
~ (the "License"); you may not use this file except in compliance with
|
||||
~ the License. You may obtain a copy of the License at
|
||||
~
|
||||
~ http://www.apache.org/licenses/LICENSE-2.0
|
||||
~
|
||||
~ Unless required by applicable law or agreed to in writing, software
|
||||
~ distributed under the License is distributed on an "AS IS" BASIS,
|
||||
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
~ See the License for the specific language governing permissions and
|
||||
~ limitations under the License.
|
||||
-->
|
||||
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<artifactId>zeppelin</artifactId>
|
||||
<groupId>org.apache.zeppelin</groupId>
|
||||
<version>0.6.0-SNAPSHOT</version>
|
||||
<relativePath>..</relativePath>
|
||||
</parent>
|
||||
|
||||
<groupId>org.apache.zeppelin</groupId>
|
||||
<artifactId>zeppelin-hive</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
<version>0.6.0-SNAPSHOT</version>
|
||||
<name>Zeppelin: Hive interpreter</name>
|
||||
<url>http://www.apache.org</url>
|
||||
<!--
|
||||
<repositories>
|
||||
<repository>
|
||||
<id>mapr-releases</id>
|
||||
<url>http://repository.mapr.com/maven/</url>
|
||||
<snapshots><enabled>false</enabled></snapshots>
|
||||
<releases><enabled>true</enabled></releases>
|
||||
</repository>
|
||||
</repositories>
|
||||
-->
|
||||
<properties>
|
||||
<hive.hive.version>0.14.0</hive.hive.version>
|
||||
<hive.hadoop.version>2.6.0</hive.hadoop.version>
|
||||
<!--<hive.hive.version>1.0.0-mapr-1504</hive.hive.version>
|
||||
<hive.hadoop.version>2.7.0-mapr-1506</hive.hadoop.version> -->
|
||||
</properties>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.apache.zeppelin</groupId>
|
||||
<artifactId>zeppelin-interpreter</artifactId>
|
||||
<version>${project.version}</version>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.commons</groupId>
|
||||
<artifactId>commons-exec</artifactId>
|
||||
<version>1.3</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-api</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-log4j12</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.hive</groupId>
|
||||
<artifactId>hive-jdbc</artifactId>
|
||||
<version>${hive.hive.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-common</artifactId>
|
||||
<version>${hive.hadoop.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.h2database</groupId>
|
||||
<artifactId>h2</artifactId>
|
||||
<version>1.4.190</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-deploy-plugin</artifactId>
|
||||
<version>2.7</version>
|
||||
<configuration>
|
||||
<skip>true</skip>
|
||||
</configuration>
|
||||
</plugin>
|
||||
|
||||
<plugin>
|
||||
<artifactId>maven-enforcer-plugin</artifactId>
|
||||
<version>1.3.1</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>enforce</id>
|
||||
<phase>none</phase>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
|
||||
<plugin>
|
||||
<artifactId>maven-dependency-plugin</artifactId>
|
||||
<version>2.8</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>copy-dependencies</id>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>copy-dependencies</goal>
|
||||
</goals>
|
||||
<configuration>
|
||||
<outputDirectory>${project.build.directory}/../../interpreter/hive</outputDirectory>
|
||||
<overWriteReleases>false</overWriteReleases>
|
||||
<overWriteSnapshots>false</overWriteSnapshots>
|
||||
<overWriteIfNewer>true</overWriteIfNewer>
|
||||
<includeScope>runtime</includeScope>
|
||||
</configuration>
|
||||
</execution>
|
||||
<execution>
|
||||
<id>copy-artifact</id>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>copy</goal>
|
||||
</goals>
|
||||
<configuration>
|
||||
<outputDirectory>${project.build.directory}/../../interpreter/hive</outputDirectory>
|
||||
<overWriteReleases>false</overWriteReleases>
|
||||
<overWriteSnapshots>false</overWriteSnapshots>
|
||||
<overWriteIfNewer>true</overWriteIfNewer>
|
||||
<includeScope>runtime</includeScope>
|
||||
<artifactItems>
|
||||
<artifactItem>
|
||||
<groupId>${project.groupId}</groupId>
|
||||
<artifactId>${project.artifactId}</artifactId>
|
||||
<version>${project.version}</version>
|
||||
<type>${project.packaging}</type>
|
||||
</artifactItem>
|
||||
</artifactItems>
|
||||
</configuration>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
</project>
|
||||
|
|
@ -1,391 +0,0 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.zeppelin.hive;
|
||||
|
||||
import java.sql.Connection;
|
||||
import java.sql.DriverManager;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.ResultSetMetaData;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Statement;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Properties;
|
||||
import java.util.Set;
|
||||
|
||||
import org.apache.zeppelin.interpreter.Interpreter;
|
||||
import org.apache.zeppelin.interpreter.InterpreterContext;
|
||||
import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
|
||||
import org.apache.zeppelin.interpreter.InterpreterResult;
|
||||
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
|
||||
import org.apache.zeppelin.scheduler.Scheduler;
|
||||
import org.apache.zeppelin.scheduler.SchedulerFactory;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import static org.apache.commons.lang.StringUtils.containsIgnoreCase;
|
||||
|
||||
/**
|
||||
* Hive interpreter for Zeppelin.
|
||||
*/
|
||||
public class HiveInterpreter extends Interpreter {
|
||||
Logger logger = LoggerFactory.getLogger(HiveInterpreter.class);
|
||||
|
||||
static final String COMMON_KEY = "common";
|
||||
static final String MAX_LINE_KEY = "max_count";
|
||||
static final String MAX_LINE_DEFAULT = "1000";
|
||||
|
||||
static final String DEFAULT_KEY = "default";
|
||||
static final String DRIVER_KEY = "driver";
|
||||
static final String URL_KEY = "url";
|
||||
static final String USER_KEY = "user";
|
||||
static final String PASSWORD_KEY = "password";
|
||||
static final String DOT = ".";
|
||||
|
||||
static final char TAB = '\t';
|
||||
static final char NEWLINE = '\n';
|
||||
static final String EXPLAIN_PREDICATE = "EXPLAIN ";
|
||||
static final String TABLE_MAGIC_TAG = "%table ";
|
||||
static final String UPDATE_COUNT_HEADER = "Update Count";
|
||||
|
||||
static final String COMMON_MAX_LINE = COMMON_KEY + DOT + MAX_LINE_KEY;
|
||||
|
||||
static final String DEFAULT_DRIVER = DEFAULT_KEY + DOT + DRIVER_KEY;
|
||||
static final String DEFAULT_URL = DEFAULT_KEY + DOT + URL_KEY;
|
||||
static final String DEFAULT_USER = DEFAULT_KEY + DOT + USER_KEY;
|
||||
static final String DEFAULT_PASSWORD = DEFAULT_KEY + DOT + PASSWORD_KEY;
|
||||
|
||||
private final HashMap<String, Properties> propertiesMap;
|
||||
private final Map<String, Statement> paragraphIdStatementMap;
|
||||
|
||||
private final Map<String, ArrayList<Connection>> propertyKeyUnusedConnectionListMap;
|
||||
private final Map<String, Connection> paragraphIdConnectionMap;
|
||||
|
||||
static {
|
||||
Interpreter.register(
|
||||
"hql",
|
||||
"hive",
|
||||
HiveInterpreter.class.getName(),
|
||||
new InterpreterPropertyBuilder()
|
||||
.add(COMMON_MAX_LINE, MAX_LINE_DEFAULT, "Maximum line of results")
|
||||
.add(DEFAULT_DRIVER, "org.apache.hive.jdbc.HiveDriver", "Hive JDBC driver")
|
||||
.add(DEFAULT_URL, "jdbc:hive2://localhost:10000", "The URL for HiveServer2.")
|
||||
.add(DEFAULT_USER, "hive", "The hive user")
|
||||
.add(DEFAULT_PASSWORD, "", "The password for the hive user").build());
|
||||
}
|
||||
|
||||
public HiveInterpreter(Properties property) {
|
||||
super(property);
|
||||
propertiesMap = new HashMap<>();
|
||||
propertyKeyUnusedConnectionListMap = new HashMap<>();
|
||||
paragraphIdStatementMap = new HashMap<>();
|
||||
paragraphIdConnectionMap = new HashMap<>();
|
||||
}
|
||||
|
||||
public HashMap<String, Properties> getPropertiesMap() {
|
||||
return propertiesMap;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void open() {
|
||||
logger.debug("property: {}", property);
|
||||
|
||||
for (String propertyKey : property.stringPropertyNames()) {
|
||||
logger.debug("propertyKey: {}", propertyKey);
|
||||
String[] keyValue = propertyKey.split("\\.", 2);
|
||||
if (2 == keyValue.length) {
|
||||
logger.debug("key: {}, value: {}", keyValue[0], keyValue[1]);
|
||||
Properties prefixProperties;
|
||||
if (propertiesMap.containsKey(keyValue[0])) {
|
||||
prefixProperties = propertiesMap.get(keyValue[0]);
|
||||
} else {
|
||||
prefixProperties = new Properties();
|
||||
propertiesMap.put(keyValue[0], prefixProperties);
|
||||
}
|
||||
prefixProperties.put(keyValue[1], property.getProperty(propertyKey));
|
||||
}
|
||||
}
|
||||
|
||||
Set<String> removeKeySet = new HashSet<>();
|
||||
for (String key : propertiesMap.keySet()) {
|
||||
if (!COMMON_KEY.equals(key)) {
|
||||
Properties properties = propertiesMap.get(key);
|
||||
if (!properties.containsKey(DRIVER_KEY) || !properties.containsKey(URL_KEY)) {
|
||||
logger.error("{} will be ignored. {}.{} and {}.{} is mandatory.",
|
||||
key, DRIVER_KEY, key, key, URL_KEY);
|
||||
removeKeySet.add(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (String key : removeKeySet) {
|
||||
propertiesMap.remove(key);
|
||||
}
|
||||
|
||||
logger.debug("propertiesMap: {}", propertiesMap);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
try {
|
||||
for (List<Connection> connectionList : propertyKeyUnusedConnectionListMap.values()) {
|
||||
for (Connection c : connectionList) {
|
||||
c.close();
|
||||
}
|
||||
}
|
||||
|
||||
for (Statement statement : paragraphIdStatementMap.values()) {
|
||||
statement.close();
|
||||
}
|
||||
paragraphIdStatementMap.clear();
|
||||
|
||||
for (Connection connection : paragraphIdConnectionMap.values()) {
|
||||
connection.close();
|
||||
}
|
||||
paragraphIdConnectionMap.clear();
|
||||
|
||||
} catch (SQLException e) {
|
||||
logger.error("Error while closing...", e);
|
||||
}
|
||||
}
|
||||
|
||||
public Connection getConnection(String propertyKey) throws ClassNotFoundException, SQLException {
|
||||
Connection connection = null;
|
||||
if (propertyKey == null || propertiesMap.get(propertyKey) == null) {
|
||||
return null;
|
||||
}
|
||||
if (propertyKeyUnusedConnectionListMap.containsKey(propertyKey)) {
|
||||
ArrayList<Connection> connectionList = propertyKeyUnusedConnectionListMap.get(propertyKey);
|
||||
if (0 != connectionList.size()) {
|
||||
connection = propertyKeyUnusedConnectionListMap.get(propertyKey).remove(0);
|
||||
if (null != connection && connection.isClosed()) {
|
||||
connection.close();
|
||||
connection = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (null == connection) {
|
||||
Properties properties = propertiesMap.get(propertyKey);
|
||||
Class.forName(properties.getProperty(DRIVER_KEY));
|
||||
String url = properties.getProperty(URL_KEY);
|
||||
String user = properties.getProperty(USER_KEY);
|
||||
String password = properties.getProperty(PASSWORD_KEY);
|
||||
if (null != user && null != password) {
|
||||
connection = DriverManager.getConnection(url, user, password);
|
||||
} else {
|
||||
connection = DriverManager.getConnection(url, properties);
|
||||
}
|
||||
}
|
||||
return connection;
|
||||
}
|
||||
|
||||
public Statement getStatement(String propertyKey, String paragraphId)
|
||||
throws SQLException, ClassNotFoundException {
|
||||
Connection connection;
|
||||
if (paragraphIdConnectionMap.containsKey(paragraphId)) {
|
||||
// Never enter for now.
|
||||
connection = paragraphIdConnectionMap.get(paragraphId);
|
||||
} else {
|
||||
connection = getConnection(propertyKey);
|
||||
}
|
||||
|
||||
if (connection == null) {
|
||||
return null;
|
||||
}
|
||||
|
||||
Statement statement = connection.createStatement();
|
||||
if (isStatementClosed(statement)) {
|
||||
connection = getConnection(propertyKey);
|
||||
statement = connection.createStatement();
|
||||
}
|
||||
paragraphIdConnectionMap.put(paragraphId, connection);
|
||||
paragraphIdStatementMap.put(paragraphId, statement);
|
||||
|
||||
return statement;
|
||||
}
|
||||
|
||||
private boolean isStatementClosed(Statement statement) {
|
||||
try {
|
||||
return statement.isClosed();
|
||||
} catch (Throwable t) {
|
||||
logger.debug("{} doesn't support isClosed method", statement);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
public InterpreterResult executeSql(String propertyKey, String sql,
|
||||
InterpreterContext interpreterContext) {
|
||||
String paragraphId = interpreterContext.getParagraphId();
|
||||
|
||||
try {
|
||||
|
||||
Statement statement = getStatement(propertyKey, paragraphId);
|
||||
|
||||
if (statement == null) {
|
||||
return new InterpreterResult(Code.ERROR, "Prefix not found.");
|
||||
}
|
||||
|
||||
statement.setMaxRows(getMaxResult());
|
||||
|
||||
StringBuilder msg;
|
||||
|
||||
if (containsIgnoreCase(sql, EXPLAIN_PREDICATE)) {
|
||||
msg = new StringBuilder();
|
||||
} else {
|
||||
msg = new StringBuilder(TABLE_MAGIC_TAG);
|
||||
}
|
||||
|
||||
ResultSet resultSet = null;
|
||||
|
||||
try {
|
||||
boolean isResultSetAvailable = statement.execute(sql);
|
||||
|
||||
if (isResultSetAvailable) {
|
||||
resultSet = statement.getResultSet();
|
||||
|
||||
ResultSetMetaData md = resultSet.getMetaData();
|
||||
|
||||
for (int i = 1; i < md.getColumnCount() + 1; i++) {
|
||||
if (i > 1) {
|
||||
msg.append(TAB);
|
||||
}
|
||||
msg.append(md.getColumnName(i));
|
||||
}
|
||||
msg.append(NEWLINE);
|
||||
|
||||
int displayRowCount = 0;
|
||||
while (resultSet.next() && displayRowCount < getMaxResult()) {
|
||||
for (int i = 1; i < md.getColumnCount() + 1; i++) {
|
||||
msg.append(resultSet.getString(i));
|
||||
if (i != md.getColumnCount()) {
|
||||
msg.append(TAB);
|
||||
}
|
||||
}
|
||||
msg.append(NEWLINE);
|
||||
displayRowCount++;
|
||||
}
|
||||
} else {
|
||||
// Response contains either an update count or there are no results.
|
||||
int updateCount = statement.getUpdateCount();
|
||||
msg.append(UPDATE_COUNT_HEADER).append(NEWLINE);
|
||||
msg.append(updateCount).append(NEWLINE);
|
||||
}
|
||||
} finally {
|
||||
try {
|
||||
if (resultSet != null) {
|
||||
resultSet.close();
|
||||
}
|
||||
statement.close();
|
||||
} finally {
|
||||
moveConnectionToUnused(propertyKey, paragraphId);
|
||||
}
|
||||
}
|
||||
|
||||
return new InterpreterResult(Code.SUCCESS, msg.toString());
|
||||
|
||||
} catch (SQLException | ClassNotFoundException ex) {
|
||||
logger.error("Cannot run " + sql, ex);
|
||||
return new InterpreterResult(Code.ERROR, ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private void moveConnectionToUnused(String propertyKey, String paragraphId) {
|
||||
if (paragraphIdConnectionMap.containsKey(paragraphId)) {
|
||||
Connection connection = paragraphIdConnectionMap.remove(paragraphId);
|
||||
if (null != connection) {
|
||||
if (propertyKeyUnusedConnectionListMap.containsKey(propertyKey)) {
|
||||
propertyKeyUnusedConnectionListMap.get(propertyKey).add(connection);
|
||||
} else {
|
||||
ArrayList<Connection> connectionList = new ArrayList<>();
|
||||
connectionList.add(connection);
|
||||
propertyKeyUnusedConnectionListMap.put(propertyKey, connectionList);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public InterpreterResult interpret(String cmd, InterpreterContext contextInterpreter) {
|
||||
String propertyKey = getPropertyKey(cmd);
|
||||
|
||||
if (null != propertyKey && !propertyKey.equals(DEFAULT_KEY)) {
|
||||
cmd = cmd.substring(propertyKey.length() + 2);
|
||||
}
|
||||
|
||||
cmd = cmd.trim();
|
||||
|
||||
logger.info("PropertyKey: {}, SQL command: '{}'", propertyKey, cmd);
|
||||
|
||||
return executeSql(propertyKey, cmd, contextInterpreter);
|
||||
}
|
||||
|
||||
private int getMaxResult() {
|
||||
return Integer.valueOf(
|
||||
propertiesMap.get(COMMON_KEY).getProperty(MAX_LINE_KEY, MAX_LINE_DEFAULT));
|
||||
}
|
||||
|
||||
public String getPropertyKey(String cmd) {
|
||||
boolean firstLineIndex = cmd.startsWith("(");
|
||||
|
||||
if (firstLineIndex) {
|
||||
int configStartIndex = cmd.indexOf("(");
|
||||
int configLastIndex = cmd.indexOf(")");
|
||||
if (configStartIndex != -1 && configLastIndex != -1) {
|
||||
return cmd.substring(configStartIndex + 1, configLastIndex);
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
} else {
|
||||
return DEFAULT_KEY;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void cancel(InterpreterContext context) {
|
||||
String paragraphId = context.getParagraphId();
|
||||
try {
|
||||
paragraphIdStatementMap.get(paragraphId).cancel();
|
||||
} catch (SQLException e) {
|
||||
logger.error("Error while cancelling...", e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public FormType getFormType() {
|
||||
return FormType.SIMPLE;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getProgress(InterpreterContext context) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Scheduler getScheduler() {
|
||||
return SchedulerFactory.singleton().createOrGetParallelScheduler(
|
||||
HiveInterpreter.class.getName() + this.hashCode(), 10);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<String> completion(String buf, int cursor) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.zeppelin.hive;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.sql.Connection;
|
||||
import java.sql.DriverManager;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Statement;
|
||||
import java.util.Properties;
|
||||
|
||||
import org.apache.zeppelin.interpreter.InterpreterContext;
|
||||
import org.apache.zeppelin.interpreter.InterpreterResult;
|
||||
import org.junit.After;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
import static org.junit.Assert.*;
|
||||
import static java.lang.String.format;
|
||||
|
||||
/**
|
||||
* Hive interpreter unit tests
|
||||
*/
|
||||
public class HiveInterpreterTest {
|
||||
static String jdbcConnection;
|
||||
|
||||
private static String getJdbcConnection() throws IOException {
|
||||
if(null == jdbcConnection) {
|
||||
Path tmpDir = Files.createTempDirectory("h2-test-");
|
||||
tmpDir.toFile().deleteOnExit();
|
||||
jdbcConnection = format("jdbc:h2:%s", tmpDir);
|
||||
}
|
||||
return jdbcConnection;
|
||||
}
|
||||
@BeforeClass
|
||||
public static void setUp() throws Exception {
|
||||
|
||||
Class.forName("org.h2.Driver");
|
||||
Connection connection = DriverManager.getConnection(getJdbcConnection());
|
||||
Statement statement = connection.createStatement();
|
||||
statement.execute(
|
||||
"DROP TABLE IF EXISTS test_table; " +
|
||||
"CREATE TABLE test_table(id varchar(255), name varchar(255));");
|
||||
statement.execute(
|
||||
"insert into test_table(id, name) values ('a', 'a_name'),('b', 'b_name');"
|
||||
);
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testForParsePropertyKey() throws IOException {
|
||||
HiveInterpreter t = new HiveInterpreter(new Properties());
|
||||
|
||||
assertEquals(t.getPropertyKey("(fake) select max(cant) from test_table where id >= 2452640"),
|
||||
"fake");
|
||||
|
||||
assertEquals(t.getPropertyKey("() select max(cant) from test_table where id >= 2452640"),
|
||||
"");
|
||||
|
||||
assertEquals(t.getPropertyKey(")fake( select max(cant) from test_table where id >= 2452640"),
|
||||
"default");
|
||||
|
||||
// when you use a %hive(prefix1), prefix1 is the propertyKey as form part of the cmd string
|
||||
assertEquals(t.getPropertyKey("(prefix1)\n select max(cant) from test_table where id >= 2452640"),
|
||||
"prefix1");
|
||||
|
||||
assertEquals(t.getPropertyKey("(prefix2) select max(cant) from test_table where id >= 2452640"),
|
||||
"prefix2");
|
||||
|
||||
// when you use a %hive, prefix is the default
|
||||
assertEquals(t.getPropertyKey("select max(cant) from test_table where id >= 2452640"),
|
||||
"default");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testForMapPrefix() throws SQLException, IOException {
|
||||
Properties properties = new Properties();
|
||||
properties.setProperty("common.max_count", "1000");
|
||||
properties.setProperty("common.max_retry", "3");
|
||||
properties.setProperty("default.driver", "org.h2.Driver");
|
||||
properties.setProperty("default.url", getJdbcConnection());
|
||||
properties.setProperty("default.user", "");
|
||||
properties.setProperty("default.password", "");
|
||||
HiveInterpreter t = new HiveInterpreter(properties);
|
||||
t.open();
|
||||
|
||||
String sqlQuery = "(fake) select * from test_table";
|
||||
|
||||
InterpreterResult interpreterResult = t.interpret(sqlQuery, new InterpreterContext("", "1", "", "", null, null, null, null, null, null, null));
|
||||
|
||||
// if prefix not found return ERROR and Prefix not found.
|
||||
assertEquals(InterpreterResult.Code.ERROR, interpreterResult.code());
|
||||
assertEquals("Prefix not found.", interpreterResult.message());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void readTest() throws IOException {
|
||||
Properties properties = new Properties();
|
||||
properties.setProperty("common.max_count", "1000");
|
||||
properties.setProperty("common.max_retry", "3");
|
||||
properties.setProperty("default.driver", "org.h2.Driver");
|
||||
properties.setProperty("default.url", getJdbcConnection());
|
||||
properties.setProperty("default.user", "");
|
||||
properties.setProperty("default.password", "");
|
||||
HiveInterpreter t = new HiveInterpreter(properties);
|
||||
t.open();
|
||||
|
||||
assertTrue(t.interpret("show databases", new InterpreterContext("", "1", "", "", null, null, null, null, null, null, null)).message().contains("SCHEMA_NAME"));
|
||||
assertEquals("ID\tNAME\na\ta_name\nb\tb_name\n",
|
||||
t.interpret("select * from test_table", new InterpreterContext("", "1", "", "", null, null, null, null, null, null, null)).message());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void readTestWithConfiguration() throws IOException {
|
||||
Properties properties = new Properties();
|
||||
properties.setProperty("common.max_count", "1000");
|
||||
properties.setProperty("common.max_retry", "3");
|
||||
properties.setProperty("default.driver", "wrong.Driver");
|
||||
properties.setProperty("default.url", getJdbcConnection());
|
||||
properties.setProperty("default.user", "");
|
||||
properties.setProperty("default.password", "");
|
||||
properties.setProperty("h2.driver", "org.h2.Driver");
|
||||
properties.setProperty("h2.url", getJdbcConnection());
|
||||
properties.setProperty("h2.user", "");
|
||||
properties.setProperty("h2.password", "");
|
||||
HiveInterpreter t = new HiveInterpreter(properties);
|
||||
t.open();
|
||||
|
||||
assertEquals("ID\tNAME\na\ta_name\nb\tb_name\n",
|
||||
t.interpret("(h2)\n select * from test_table", new InterpreterContext("", "1", "", "", null, null, null, null, null, null, null)).message());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void jdbcRestart() throws IOException, SQLException, ClassNotFoundException {
|
||||
Properties properties = new Properties();
|
||||
properties.setProperty("common.max_count", "1000");
|
||||
properties.setProperty("common.max_retry", "3");
|
||||
properties.setProperty("default.driver", "org.h2.Driver");
|
||||
properties.setProperty("default.url", getJdbcConnection());
|
||||
properties.setProperty("default.user", "");
|
||||
properties.setProperty("default.password", "");
|
||||
HiveInterpreter t = new HiveInterpreter(properties);
|
||||
t.open();
|
||||
|
||||
InterpreterResult interpreterResult =
|
||||
t.interpret("select * from test_table", new InterpreterContext("", "1", "", "", null, null, null, null, null, null, null));
|
||||
assertEquals("ID\tNAME\na\ta_name\nb\tb_name\n", interpreterResult.message());
|
||||
|
||||
t.getConnection("default").close();
|
||||
|
||||
interpreterResult =
|
||||
t.interpret("select * from test_table", new InterpreterContext("", "1", "", "", null, null, null, null, null, null, null));
|
||||
assertEquals("ID\tNAME\na\ta_name\nb\tb_name\n", interpreterResult.message());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void test() throws IOException {
|
||||
Properties properties = new Properties();
|
||||
properties.setProperty("common.max_count", "1000");
|
||||
properties.setProperty("common.max_retry", "3");
|
||||
properties.setProperty("default.driver", "org.h2.Driver");
|
||||
properties.setProperty("default.url", getJdbcConnection());
|
||||
properties.setProperty("default.user", "");
|
||||
properties.setProperty("default.password", "");
|
||||
HiveInterpreter t = new HiveInterpreter(properties);
|
||||
t.open();
|
||||
|
||||
InterpreterContext interpreterContext = new InterpreterContext(null, "a", null, null, null, null, null, null, null, null, null);
|
||||
|
||||
//simple select test
|
||||
InterpreterResult result = t.interpret("select * from test_table", interpreterContext);
|
||||
assertEquals(result.type(), InterpreterResult.Type.TABLE);
|
||||
|
||||
//explain test
|
||||
result = t.interpret("explain select * from test_table", interpreterContext);
|
||||
assertEquals(result.type(), InterpreterResult.Type.TEXT);
|
||||
t.close();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void parseMultiplePropertiesMap() {
|
||||
Properties properties = new Properties();
|
||||
properties.setProperty("common.max_count", "1000");
|
||||
properties.setProperty("common.max_retry", "3");
|
||||
properties.setProperty("default.driver", "defaultDriver");
|
||||
properties.setProperty("default.url", "defaultUri");
|
||||
properties.setProperty("default.user", "defaultUser");
|
||||
HiveInterpreter hi = new HiveInterpreter(properties);
|
||||
hi.open();
|
||||
assertNotNull("propertiesMap is not null", hi.getPropertiesMap());
|
||||
assertNotNull("propertiesMap.get(default) is not null", hi.getPropertiesMap().get("default"));
|
||||
assertTrue("default exists", "defaultDriver".equals(hi.getPropertiesMap().get("default").getProperty("driver")));
|
||||
hi.close();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void ignoreInvalidSettings() {
|
||||
Properties properties = new Properties();
|
||||
properties.setProperty("common.max_count", "1000");
|
||||
properties.setProperty("common.max_retry", "3");
|
||||
properties.setProperty("default.driver", "defaultDriver");
|
||||
properties.setProperty("default.url", "defaultUri");
|
||||
properties.setProperty("default.user", "defaultUser");
|
||||
properties.setProperty("presto.driver", "com.facebook.presto.jdbc.PrestoDriver");
|
||||
HiveInterpreter hi = new HiveInterpreter(properties);
|
||||
hi.open();
|
||||
assertTrue("default exists", hi.getPropertiesMap().containsKey("default"));
|
||||
assertFalse("presto doesn't exists", hi.getPropertiesMap().containsKey("presto"));
|
||||
hi.close();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void getPropertyKey() {
|
||||
HiveInterpreter hi = new HiveInterpreter(new Properties());
|
||||
hi.open();
|
||||
String testCommand = "(default)\nshow tables";
|
||||
assertEquals("get key of default", "default", hi.getPropertyKey(testCommand));
|
||||
testCommand = "(default) show tables";
|
||||
assertEquals("get key of default", "default", hi.getPropertyKey(testCommand));
|
||||
hi.close();
|
||||
}
|
||||
}
|
||||
9
pom.xml
9
pom.xml
|
|
@ -55,9 +55,9 @@
|
|||
</licenses>
|
||||
|
||||
<scm>
|
||||
<url>https://git-wip-us.apache.org/repos/asf/incubator-zeppelin.git</url>
|
||||
<connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-zeppelin.git</connection>
|
||||
<developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-zeppelin.git</developerConnection>
|
||||
<url>https://git-wip-us.apache.org/repos/asf/zeppelin.git</url>
|
||||
<connection>scm:git:https://git-wip-us.apache.org/repos/asf/zeppelin.git</connection>
|
||||
<developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/zeppelin.git</developerConnection>
|
||||
</scm>
|
||||
|
||||
<inceptionYear>2013</inceptionYear>
|
||||
|
|
@ -72,7 +72,6 @@
|
|||
<module>angular</module>
|
||||
<module>shell</module>
|
||||
<module>livy</module>
|
||||
<module>hive</module>
|
||||
<module>hbase</module>
|
||||
<module>phoenix</module>
|
||||
<module>postgresql</module>
|
||||
|
|
@ -362,8 +361,6 @@
|
|||
<resourceBundles>
|
||||
<!-- Will generate META-INF/DEPENDENCIES META-INF/LICENSE META-INF/NOTICE -->
|
||||
<resourceBundle>org.apache.apache.resources:apache-jar-resource-bundle:1.5-SNAPSHOT</resourceBundle>
|
||||
<!-- Will generate META-INF/DISCLAIMER -->
|
||||
<resourceBundle>org.apache.apache.resources:apache-incubator-disclaimer-resource-bundle:1.2-SNAPSHOT</resourceBundle>
|
||||
</resourceBundles>
|
||||
</configuration>
|
||||
</execution>
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
Package: rzeppelin
|
||||
Type: Package
|
||||
Title: Interface from scala to R, based on rscala, for the Apache (Incubation) Zeppelin project
|
||||
Title: Interface from scala to R, based on rscala, for the Apache Zeppelin project
|
||||
Version: 0.1.0
|
||||
Date: 2015-12-01
|
||||
Authors@R: c(person(given="David B.",family="Dahl",role=c("aut","cre"),email="dahl@stat.byu.edu"),
|
||||
|
|
|
|||
|
|
@ -34,9 +34,9 @@
|
|||
<url>http://zeppelin.apache.org</url>
|
||||
|
||||
<properties>
|
||||
<scala.version>2.10.4</scala.version>
|
||||
<hadoop.version>2.3.0</hadoop.version>
|
||||
<scalding.version>0.15.1-RC13</scalding.version>
|
||||
<scala.version>2.11.8</scala.version>
|
||||
<hadoop.version>2.6.0</hadoop.version>
|
||||
<scalding.version>0.16.1-RC1</scalding.version>
|
||||
</properties>
|
||||
|
||||
<repositories>
|
||||
|
|
@ -45,6 +45,11 @@
|
|||
<name>Concurrent Maven Repo</name>
|
||||
<url>http://conjars.org/repo</url>
|
||||
</repository>
|
||||
<repository>
|
||||
<id>twitter</id>
|
||||
<name>Twitter Maven Repo</name>
|
||||
<url>http://maven.twttr.com</url>
|
||||
</repository>
|
||||
</repositories>
|
||||
|
||||
<dependencies>
|
||||
|
|
@ -69,13 +74,43 @@
|
|||
|
||||
<dependency>
|
||||
<groupId>com.twitter</groupId>
|
||||
<artifactId>scalding-core_2.10</artifactId>
|
||||
<artifactId>scalding-core_2.11</artifactId>
|
||||
<version>${scalding.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.twitter</groupId>
|
||||
<artifactId>scalding-repl_2.10</artifactId>
|
||||
<artifactId>scalding-args_2.11</artifactId>
|
||||
<version>${scalding.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.twitter</groupId>
|
||||
<artifactId>scalding-date_2.11</artifactId>
|
||||
<version>${scalding.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.twitter</groupId>
|
||||
<artifactId>scalding-commons_2.11</artifactId>
|
||||
<version>${scalding.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.twitter</groupId>
|
||||
<artifactId>scalding-avro_2.11</artifactId>
|
||||
<version>${scalding.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.twitter</groupId>
|
||||
<artifactId>scalding-parquet_2.11</artifactId>
|
||||
<version>${scalding.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.twitter</groupId>
|
||||
<artifactId>scalding-repl_2.11</artifactId>
|
||||
<version>${scalding.version}</version>
|
||||
</dependency>
|
||||
|
||||
|
|
@ -97,12 +132,12 @@
|
|||
<version>${scala.version}</version>
|
||||
</dependency>
|
||||
|
||||
<!-- Scalding REPL needs org.apache.hadoop.conf.Configuration even in local mode -->
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-common</artifactId>
|
||||
<artifactId>hadoop-client</artifactId>
|
||||
<version>${hadoop.version}</version>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
|
|
|
|||
|
|
@ -17,35 +17,29 @@
|
|||
|
||||
package org.apache.zeppelin.scalding;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.File;
|
||||
import java.io.PrintStream;
|
||||
import java.io.PrintWriter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.LinkedList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Properties;
|
||||
import java.net.URL;
|
||||
import java.net.URLClassLoader;
|
||||
|
||||
import com.twitter.scalding.ScaldingILoop;
|
||||
import org.apache.hadoop.security.UserGroupInformation;
|
||||
import org.apache.zeppelin.interpreter.Interpreter;
|
||||
import org.apache.zeppelin.interpreter.InterpreterContext;
|
||||
import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
|
||||
import org.apache.zeppelin.interpreter.InterpreterResult;
|
||||
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
|
||||
import org.apache.zeppelin.scheduler.Scheduler;
|
||||
import org.apache.zeppelin.scheduler.SchedulerFactory;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import scala.Console;
|
||||
import scala.Some;
|
||||
import scala.None;
|
||||
import scala.tools.nsc.Settings;
|
||||
import scala.tools.nsc.settings.MutableSettings.BooleanSetting;
|
||||
import scala.tools.nsc.settings.MutableSettings.PathSetting;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.PrintStream;
|
||||
import java.io.PrintWriter;
|
||||
import java.security.PrivilegedExceptionAction;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Properties;
|
||||
|
||||
/**
|
||||
* Scalding interpreter for Zeppelin. Based off the Spark interpreter code.
|
||||
|
|
@ -54,16 +48,29 @@ import scala.tools.nsc.settings.MutableSettings.PathSetting;
|
|||
public class ScaldingInterpreter extends Interpreter {
|
||||
Logger logger = LoggerFactory.getLogger(ScaldingInterpreter.class);
|
||||
|
||||
static final String ARGS_STRING = "args.string";
|
||||
static final String ARGS_STRING_DEFAULT = "--local --repl";
|
||||
static final String MAX_OPEN_INSTANCES = "max.open.instances";
|
||||
static final String MAX_OPEN_INSTANCES_DEFAULT = "50";
|
||||
|
||||
public static final List<String> NO_COMPLETION =
|
||||
Collections.unmodifiableList(new ArrayList<String>());
|
||||
|
||||
static {
|
||||
Interpreter.register("scalding", ScaldingInterpreter.class.getName());
|
||||
Interpreter.register(
|
||||
"scalding",
|
||||
"scalding",
|
||||
ScaldingInterpreter.class.getName(),
|
||||
new InterpreterPropertyBuilder()
|
||||
.add(ARGS_STRING, ARGS_STRING_DEFAULT, "Arguments for scalding REPL")
|
||||
.add(MAX_OPEN_INSTANCES, MAX_OPEN_INSTANCES_DEFAULT,
|
||||
"Maximum number of open interpreter instances")
|
||||
.build());
|
||||
}
|
||||
|
||||
static int numOpenInstances = 0;
|
||||
private ScaldingILoop interpreter;
|
||||
private ByteArrayOutputStream out;
|
||||
private Map<String, Object> binder;
|
||||
|
||||
public ScaldingInterpreter(Properties property) {
|
||||
super(property);
|
||||
|
|
@ -72,104 +79,34 @@ public class ScaldingInterpreter extends Interpreter {
|
|||
|
||||
@Override
|
||||
public void open() {
|
||||
URL[] urls = getClassloaderUrls();
|
||||
|
||||
// Very nice discussion about how scala compiler handle classpath
|
||||
// https://groups.google.com/forum/#!topic/scala-user/MlVwo2xCCI0
|
||||
|
||||
/*
|
||||
* > val env = new nsc.Settings(errLogger) > env.usejavacp.value = true > val p = new
|
||||
* Interpreter(env) > p.setContextClassLoader > Alternatively you can set the class path through
|
||||
* nsc.Settings.classpath.
|
||||
*
|
||||
* >> val settings = new Settings() >> settings.usejavacp.value = true >>
|
||||
* settings.classpath.value += File.pathSeparator + >> System.getProperty("java.class.path") >>
|
||||
* val in = new Interpreter(settings) { >> override protected def parentClassLoader =
|
||||
* getClass.getClassLoader >> } >> in.setContextClassLoader()
|
||||
*/
|
||||
Settings settings = new Settings();
|
||||
|
||||
// set classpath for scala compiler
|
||||
PathSetting pathSettings = settings.classpath();
|
||||
String classpath = "";
|
||||
List<File> paths = currentClassPath();
|
||||
for (File f : paths) {
|
||||
if (classpath.length() > 0) {
|
||||
classpath += File.pathSeparator;
|
||||
}
|
||||
classpath += f.getAbsolutePath();
|
||||
numOpenInstances = numOpenInstances + 1;
|
||||
String maxOpenInstancesStr = property.getProperty(MAX_OPEN_INSTANCES,
|
||||
MAX_OPEN_INSTANCES_DEFAULT);
|
||||
int maxOpenInstances = 50;
|
||||
try {
|
||||
maxOpenInstances = Integer.valueOf(maxOpenInstancesStr);
|
||||
} catch (Exception e) {
|
||||
logger.error("Error reading max.open.instances", e);
|
||||
}
|
||||
|
||||
if (urls != null) {
|
||||
for (URL u : urls) {
|
||||
if (classpath.length() > 0) {
|
||||
classpath += File.pathSeparator;
|
||||
}
|
||||
classpath += u.getFile();
|
||||
}
|
||||
logger.info("max.open.instances = {}", maxOpenInstances);
|
||||
if (numOpenInstances > maxOpenInstances) {
|
||||
logger.error("Reached maximum number of open instances");
|
||||
return;
|
||||
}
|
||||
|
||||
pathSettings.v_$eq(classpath);
|
||||
settings.scala$tools$nsc$settings$ScalaSettings$_setter_$classpath_$eq(pathSettings);
|
||||
|
||||
|
||||
// set classloader for scala compiler
|
||||
settings.explicitParentLoader_$eq(new Some<ClassLoader>(Thread.currentThread()
|
||||
.getContextClassLoader()));
|
||||
BooleanSetting b = (BooleanSetting) settings.usejavacp();
|
||||
b.v_$eq(true);
|
||||
settings.scala$tools$nsc$settings$StandardScalaSettings$_setter_$usejavacp_$eq(b);
|
||||
|
||||
/* Scalding interpreter */
|
||||
PrintStream printStream = new PrintStream(out);
|
||||
interpreter = new ScaldingILoop(null, new PrintWriter(out));
|
||||
interpreter.settings_$eq(settings);
|
||||
interpreter.createInterpreter();
|
||||
|
||||
interpreter.intp().
|
||||
interpret("@transient var _binder = new java.util.HashMap[String, Object]()");
|
||||
binder = (Map<String, Object>) getValue("_binder");
|
||||
binder.put("out", printStream);
|
||||
}
|
||||
|
||||
private Object getValue(String name) {
|
||||
Object ret = interpreter.intp().valueOfTerm(name);
|
||||
if (ret instanceof None) {
|
||||
return null;
|
||||
} else if (ret instanceof Some) {
|
||||
return ((Some) ret).get();
|
||||
logger.info("Opening instance {}", numOpenInstances);
|
||||
logger.info("property: {}", property);
|
||||
String argsString = property.getProperty(ARGS_STRING, ARGS_STRING_DEFAULT);
|
||||
String[] args;
|
||||
if (argsString == null) {
|
||||
args = new String[0];
|
||||
} else {
|
||||
return ret;
|
||||
args = argsString.split(" ");
|
||||
}
|
||||
}
|
||||
logger.info("{}", Arrays.toString(args));
|
||||
|
||||
private List<File> currentClassPath() {
|
||||
List<File> paths = classPath(Thread.currentThread().getContextClassLoader());
|
||||
String[] cps = System.getProperty("java.class.path").split(File.pathSeparator);
|
||||
if (cps != null) {
|
||||
for (String cp : cps) {
|
||||
paths.add(new File(cp));
|
||||
}
|
||||
}
|
||||
return paths;
|
||||
}
|
||||
|
||||
private List<File> classPath(ClassLoader cl) {
|
||||
List<File> paths = new LinkedList<File>();
|
||||
if (cl == null) {
|
||||
return paths;
|
||||
}
|
||||
|
||||
if (cl instanceof URLClassLoader) {
|
||||
URLClassLoader ucl = (URLClassLoader) cl;
|
||||
URL[] urls = ucl.getURLs();
|
||||
if (urls != null) {
|
||||
for (URL url : urls) {
|
||||
paths.add(new File(url.getFile()));
|
||||
}
|
||||
}
|
||||
}
|
||||
return paths;
|
||||
PrintWriter printWriter = new PrintWriter(out, true);
|
||||
interpreter = ZeppelinScaldingShell.getRepl(args, printWriter);
|
||||
interpreter.createInterpreter();
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -180,12 +117,49 @@ public class ScaldingInterpreter extends Interpreter {
|
|||
|
||||
@Override
|
||||
public InterpreterResult interpret(String cmd, InterpreterContext contextInterpreter) {
|
||||
logger.info("Running Scalding command '" + cmd + "'");
|
||||
String user = contextInterpreter.getAuthenticationInfo().getUser();
|
||||
logger.info("Running Scalding command: user: {} cmd: '{}'", user, cmd);
|
||||
|
||||
if (interpreter == null) {
|
||||
logger.error(
|
||||
"interpreter == null, open may not have been called because max.open.instances reached");
|
||||
return new InterpreterResult(Code.ERROR,
|
||||
"interpreter == null\n" +
|
||||
"open may not have been called because max.open.instances reached"
|
||||
);
|
||||
}
|
||||
if (cmd == null || cmd.trim().length() == 0) {
|
||||
return new InterpreterResult(Code.SUCCESS);
|
||||
}
|
||||
return interpret(cmd.split("\n"), contextInterpreter);
|
||||
InterpreterResult interpreterResult = new InterpreterResult(Code.ERROR);
|
||||
if (property.getProperty(ARGS_STRING).contains("hdfs")) {
|
||||
UserGroupInformation ugi = null;
|
||||
try {
|
||||
ugi = UserGroupInformation.createProxyUser(user, UserGroupInformation.getLoginUser());
|
||||
} catch (IOException e) {
|
||||
logger.error("Error creating UserGroupInformation", e);
|
||||
return new InterpreterResult(Code.ERROR, e.getMessage());
|
||||
}
|
||||
try {
|
||||
// Make variables final to avoid "local variable is accessed from within inner class;
|
||||
// needs to be declared final" exception in JDK7
|
||||
final String cmd1 = cmd;
|
||||
final InterpreterContext contextInterpreter1 = contextInterpreter;
|
||||
PrivilegedExceptionAction<InterpreterResult> action =
|
||||
new PrivilegedExceptionAction<InterpreterResult>() {
|
||||
public InterpreterResult run() throws Exception {
|
||||
return interpret(cmd1.split("\n"), contextInterpreter1);
|
||||
}
|
||||
};
|
||||
interpreterResult = ugi.doAs(action);
|
||||
} catch (Exception e) {
|
||||
logger.error("Error running command with ugi.doAs", e);
|
||||
return new InterpreterResult(Code.ERROR, e.getMessage());
|
||||
}
|
||||
} else {
|
||||
interpreterResult = interpret(cmd.split("\n"), contextInterpreter);
|
||||
}
|
||||
return interpreterResult;
|
||||
}
|
||||
|
||||
public InterpreterResult interpret(String[] lines, InterpreterContext context) {
|
||||
|
|
@ -205,8 +179,13 @@ public class ScaldingInterpreter extends Interpreter {
|
|||
}
|
||||
linesToRun[lines.length] = "print(\"\")";
|
||||
|
||||
Console.setOut((java.io.PrintStream) binder.get("out"));
|
||||
out.reset();
|
||||
|
||||
// Moving two lines below from open() to this function.
|
||||
// If they are in open output is incomplete.
|
||||
PrintStream printStream = new PrintStream(out, true);
|
||||
Console.setOut(printStream);
|
||||
|
||||
Code r = null;
|
||||
String incomplete = "";
|
||||
boolean inComment = false;
|
||||
|
|
@ -261,7 +240,6 @@ public class ScaldingInterpreter extends Interpreter {
|
|||
incomplete = "";
|
||||
}
|
||||
}
|
||||
|
||||
if (r == Code.INCOMPLETE) {
|
||||
return new InterpreterResult(r, "Incomplete expression");
|
||||
} else {
|
||||
|
|
@ -306,4 +284,5 @@ public class ScaldingInterpreter extends Interpreter {
|
|||
public List<String> completion(String buf, int cursor) {
|
||||
return NO_COMPLETION;
|
||||
}
|
||||
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,111 +0,0 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.zeppelin.scalding;
|
||||
|
||||
import java.io.{BufferedReader, File, FileReader}
|
||||
|
||||
import scala.tools.nsc.GenericRunnerSettings
|
||||
import scala.tools.nsc.interpreter.{ILoop, IR, JPrintWriter}
|
||||
|
||||
|
||||
/**
|
||||
* A class providing Scalding specific commands for inclusion in the Scalding REPL.
|
||||
* This is currently forked from Scalding, but should eventually make it into Scalding itself:
|
||||
* https://github.com/twitter/scalding/blob/develop/scalding-repl/src/main/scala/com/twitter/scalding/ScaldingILoop.scala
|
||||
*/
|
||||
class ScaldingILoop(in0: Option[BufferedReader], out: JPrintWriter)
|
||||
extends ILoop(in0, out) {
|
||||
// def this(in0: BufferedReader, out: JPrintWriter) = this(Some(in0), out)
|
||||
// def this() = this(None, new JPrintWriter(Console.out, true))
|
||||
|
||||
settings = new GenericRunnerSettings({ s => echo(s) })
|
||||
|
||||
override def printWelcome() {
|
||||
val fc = Console.YELLOW
|
||||
val wc = Console.RED
|
||||
def wrapFlames(s: String) = s.replaceAll("[()]+", fc + "$0" + wc)
|
||||
echo(fc +
|
||||
" ( \n" +
|
||||
" )\\ ) ( ( \n" +
|
||||
"(()/( ) )\\ )\\ ) ( ( ( \n" +
|
||||
" /(_)) ( ( /( ((_)(()/( )\\ ( )\\))( \n" +
|
||||
"(_)) )\\ )( )) _ ((_)(( ) )\\ ) (( ))\\ \n".replaceAll("_", wc + "_" + fc) + wc +
|
||||
wrapFlames("/ __|((_) ((_)_ | | _| | (_) _(_(( (_()_) \n") +
|
||||
wrapFlames("\\__ \\/ _| / _` || |/ _` | | || ' \\))/ _` \\ \n") +
|
||||
"|___/\\__| \\__,_||_|\\__,_| |_||_||_| \\__, | \n" +
|
||||
" |___/ ")
|
||||
}
|
||||
|
||||
/**
|
||||
* Commands specific to the Scalding REPL. To define a new command use one of the following
|
||||
* factory methods:
|
||||
* - `LoopCommand.nullary` for commands that take no arguments
|
||||
* - `LoopCommand.cmd` for commands that take one string argument
|
||||
* - `LoopCommand.varargs` for commands that take multiple string arguments
|
||||
*/
|
||||
private val scaldingCommands: List[LoopCommand] = List()
|
||||
|
||||
/**
|
||||
* Change the shell prompt to read scalding>
|
||||
*
|
||||
* @return a prompt string to use for this REPL.
|
||||
*/
|
||||
override def prompt: String = Console.BLUE + "\nscalding> " + Console.RESET
|
||||
|
||||
private[this] def addImports(ids: String*): IR.Result =
|
||||
if (ids.isEmpty) IR.Success
|
||||
else intp.interpret("import " + ids.mkString(", "))
|
||||
|
||||
/**
|
||||
* Search for files with the given name in all directories from current directory
|
||||
* up to root.
|
||||
*/
|
||||
private def findAllUpPath(filename: String): List[File] =
|
||||
Iterator.iterate(System.getProperty("user.dir"))(new File(_).getParent)
|
||||
.takeWhile(_ != "/")
|
||||
.flatMap(new File(_).listFiles.filter(_.toString.endsWith(filename)))
|
||||
.toList
|
||||
|
||||
/**
|
||||
* Gets the list of commands that this REPL supports.
|
||||
*
|
||||
* @return a list of the command supported by this REPL.
|
||||
*/
|
||||
override def commands: List[LoopCommand] = super.commands ++ scaldingCommands
|
||||
|
||||
protected def imports: List[String] = List(
|
||||
"com.twitter.scalding._",
|
||||
"com.twitter.scalding.ReplImplicits._",
|
||||
"com.twitter.scalding.ReplImplicitContext._",
|
||||
"com.twitter.scalding.ReplState._")
|
||||
|
||||
override def createInterpreter() {
|
||||
super.createInterpreter()
|
||||
intp.beQuietDuring {
|
||||
addImports(imports: _*)
|
||||
|
||||
settings match {
|
||||
case s: GenericRunnerSettings =>
|
||||
findAllUpPath(".scalding_repl").reverse.foreach {
|
||||
f => s.loadfiles.appendToValue(f.toString)
|
||||
}
|
||||
case _ => ()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.zeppelin.scalding
|
||||
|
||||
/**
|
||||
* Stores REPL state
|
||||
*/
|
||||
|
||||
import cascading.flow.FlowDef
|
||||
import com.twitter.scalding.BaseReplState
|
||||
import scala.concurrent.{ ExecutionContext => ConcurrentExecutionContext }
|
||||
import scala.concurrent.Future
|
||||
import scala.util.{Failure, Success}
|
||||
|
||||
object ZeppelinReplState extends BaseReplState {
|
||||
override def shell = ZeppelinScaldingShell
|
||||
}
|
||||
|
||||
/**
|
||||
* Implicit FlowDef and Mode, import in the REPL to have the global context implicitly
|
||||
* used everywhere.
|
||||
*/
|
||||
object ZeppelinReplImplicitContext {
|
||||
/** Implicit execution context for using the Execution monad */
|
||||
implicit val executionContext = ConcurrentExecutionContext.global
|
||||
/** Implicit repl state used for ShellPipes */
|
||||
implicit def stateImpl = ZeppelinReplState
|
||||
/** Implicit flowDef for this Scalding shell session. */
|
||||
implicit def flowDefImpl = ZeppelinReplState.flowDef
|
||||
/** Defaults to running in local mode if no mode is specified. */
|
||||
implicit def modeImpl = ZeppelinReplState.mode
|
||||
implicit def configImpl = ZeppelinReplState.config
|
||||
}
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.zeppelin.scalding
|
||||
|
||||
import java.io.BufferedReader
|
||||
import com.twitter.scalding.ScaldingILoop
|
||||
|
||||
import scala.tools.nsc.interpreter._
|
||||
|
||||
/**
|
||||
* TBD
|
||||
*/
|
||||
class ZeppelinScaldingILoop(in: Option[BufferedReader], out: JPrintWriter)
|
||||
extends ScaldingILoop(in, out) {
|
||||
|
||||
override protected def imports = List(
|
||||
"com.twitter.scalding.{ ScaldingILoop => ScaldingScaldingILoop, ScaldingShell => ScaldingScaldingShell, _ }",
|
||||
// ReplImplicits minus fields API parts (esp FieldConversions)
|
||||
"""com.twitter.scalding.ReplImplicits.{
|
||||
iterableToSource,
|
||||
keyedListLikeToShellTypedPipe,
|
||||
typedPipeToShellTypedPipe,
|
||||
valuePipeToShellValuePipe
|
||||
}""",
|
||||
"com.twitter.scalding.ReplImplicits",
|
||||
"org.apache.zeppelin.scalding.ZeppelinReplImplicitContext._",
|
||||
"org.apache.zeppelin.scalding.ZeppelinReplState",
|
||||
"org.apache.zeppelin.scalding.ZeppelinReplState._"
|
||||
)
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.zeppelin.scalding
|
||||
|
||||
import com.twitter.scalding._
|
||||
import com.twitter.scalding.typed.TypedPipe
|
||||
import scala.tools.nsc.{GenericRunnerCommand}
|
||||
import scala.tools.nsc.interpreter._
|
||||
|
||||
/**
|
||||
* TBD
|
||||
*/
|
||||
object ZeppelinScaldingShell extends BaseScaldingShell {
|
||||
|
||||
override def replState = ZeppelinReplState
|
||||
|
||||
def getRepl(args: Array[String], out: JPrintWriter): ScaldingILoop = {
|
||||
|
||||
val argsExpanded = ExpandLibJarsGlobs(args)
|
||||
val ShellArgs(cfg, mode, cmdArgs) = parseModeArgs(argsExpanded)
|
||||
|
||||
// Process command line arguments into a settings object, and use that to start the REPL.
|
||||
// We ignore params we don't care about - hence error function is empty
|
||||
val command = new GenericRunnerCommand(cmdArgs, _ => ())
|
||||
|
||||
// inherit defaults for embedded interpretter (needed for running with SBT)
|
||||
// (TypedPipe chosen arbitrarily, just needs to be something representative)
|
||||
command.settings.embeddedDefaults[TypedPipe[String]]
|
||||
|
||||
// if running from the assembly, need to explicitly tell it to use java classpath
|
||||
if (args.contains("--repl")) command.settings.usejavacp.value = true
|
||||
|
||||
command.settings.classpath.append(System.getProperty("java.class.path"))
|
||||
|
||||
// Force the repl to be synchronous, so all cmds are executed in the same thread
|
||||
command.settings.Yreplsync.value = true
|
||||
|
||||
val repl = new ZeppelinScaldingILoop(None, out)
|
||||
scaldingREPL = Some(repl)
|
||||
replState.mode = mode
|
||||
replState.customConfig = replState.customConfig ++ (mode match {
|
||||
case _: HadoopMode => cfg
|
||||
case _ => Config.empty
|
||||
})
|
||||
|
||||
// if in Hdfs mode, store the mode to enable switching between Local and Hdfs
|
||||
mode match {
|
||||
case m @ Hdfs(_, _) => replState.storedHdfsMode = Some(m)
|
||||
case _ => ()
|
||||
}
|
||||
|
||||
repl.settings = command.settings
|
||||
return repl;
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -57,6 +57,7 @@ public class ScaldingInterpreterTest {
|
|||
|
||||
if (repl == null) {
|
||||
Properties p = new Properties();
|
||||
p.setProperty(ScaldingInterpreter.ARGS_STRING, "--local --repl");
|
||||
|
||||
repl = new ScaldingInterpreter(p);
|
||||
repl.open();
|
||||
|
|
@ -119,7 +120,7 @@ public class ScaldingInterpreterTest {
|
|||
"val salesPipe = TypedPipe.from(salesList)\n" +
|
||||
"val results = salesPipe.map{x => (1, Set(x.state), x.sale)}.\n" +
|
||||
" groupAll.sum.values.map{ case(count, set, sum) => (count, set.size, sum) }\n" +
|
||||
"results.dump",
|
||||
"results.dump",
|
||||
context).code());
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ import org.apache.thrift.TException;
|
|||
import org.apache.zeppelin.display.AngularObject;
|
||||
import org.apache.zeppelin.display.AngularObjectRegistry;
|
||||
import org.apache.zeppelin.display.GUI;
|
||||
import org.apache.zeppelin.display.Input;
|
||||
import org.apache.zeppelin.interpreter.*;
|
||||
import org.apache.zeppelin.interpreter.InterpreterResult.Type;
|
||||
import org.apache.zeppelin.interpreter.thrift.RemoteInterpreterContext;
|
||||
|
|
@ -262,7 +263,8 @@ public class RemoteInterpreter extends Interpreter {
|
|||
|
||||
boolean broken = false;
|
||||
try {
|
||||
GUI settings = context.getGui();
|
||||
|
||||
final GUI currentGUI = context.getGui();
|
||||
RemoteInterpreterResult remoteResult = client.interpret(
|
||||
noteId, className, st, convert(context));
|
||||
|
||||
|
|
@ -272,11 +274,20 @@ public class RemoteInterpreter extends Interpreter {
|
|||
context.getConfig().clear();
|
||||
context.getConfig().putAll(remoteConfig);
|
||||
|
||||
|
||||
if (form == FormType.NATIVE) {
|
||||
GUI remoteGui = gson.fromJson(remoteResult.getGui(), GUI.class);
|
||||
context.getGui().clear();
|
||||
context.getGui().setParams(remoteGui.getParams());
|
||||
context.getGui().setForms(remoteGui.getForms());
|
||||
currentGUI.clear();
|
||||
currentGUI.setParams(remoteGui.getParams());
|
||||
currentGUI.setForms(remoteGui.getForms());
|
||||
} else if (form == FormType.SIMPLE) {
|
||||
final Map<String, Input> currentForms = currentGUI.getForms();
|
||||
final Map<String, Object> currentParams = currentGUI.getParams();
|
||||
final GUI remoteGUI = gson.fromJson(remoteResult.getGui(), GUI.class);
|
||||
final Map<String, Input> remoteForms = remoteGUI.getForms();
|
||||
final Map<String, Object> remoteParams = remoteGUI.getParams();
|
||||
currentForms.putAll(remoteForms);
|
||||
currentParams.putAll(remoteParams);
|
||||
}
|
||||
|
||||
InterpreterResult result = convert(remoteResult);
|
||||
|
|
|
|||
|
|
@ -41,6 +41,6 @@
|
|||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git@github.com:apache/incubator-zeppelin.git"
|
||||
"url": "git@github.com:apache/zeppelin.git"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@
|
|||
|
||||
.configuration table {
|
||||
table-layout: fixed;
|
||||
word-break: break-all;
|
||||
}
|
||||
|
||||
.configuration table tr .configurationPropertyKey {
|
||||
|
|
|
|||
|
|
@ -87,7 +87,7 @@ limitations under the License.
|
|||
target="_blank"><i style="font-size: 15px;" class="fa fa-users"></i> Mailing list</a><br>
|
||||
<a style="text-decoration: none;" href="https://issues.apache.org/jira/browse/ZEPPELIN"
|
||||
target="_blank"><i style="font-size: 15px;" class="fa fa-bug"></i> Issues tracking</a><br>
|
||||
<a style="text-decoration: none;" href="https://github.com/apache/incubator-zeppelin"
|
||||
<a style="text-decoration: none;" href="https://github.com/apache/zeppelin"
|
||||
target="_blank"><i style="font-size: 20px;" class="fa fa-github"></i> Github</a>
|
||||
</div>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -58,6 +58,11 @@
|
|||
list-style-type: none;
|
||||
}
|
||||
|
||||
.interpreter table {
|
||||
table-layout: fixed;
|
||||
word-break: break-all;
|
||||
}
|
||||
|
||||
.interpreter table tr .interpreterPropertyKey {
|
||||
padding : 5px 5px 5px 5px;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12,38 +12,45 @@ See the License for the specific language governing permissions and
|
|||
limitations under the License.
|
||||
-->
|
||||
|
||||
<div id="{{paragraph.id}}_switch"
|
||||
ng-if="paragraph.result.type == 'TABLE' && !asIframe && !viewOnly"
|
||||
class="btn-group"
|
||||
<div ng-if="paragraph.result.type == 'TABLE' && !asIframe && !viewOnly"
|
||||
style='margin-bottom: 10px;'>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('table')}"
|
||||
ng-click="setGraphMode('table', true)" ><i class="fa fa-table"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('multiBarChart')}"
|
||||
ng-click="setGraphMode('multiBarChart', true)"><i class="fa fa-bar-chart"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('pieChart')}"
|
||||
ng-click="setGraphMode('pieChart', true)"><i class="fa fa-pie-chart"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('stackedAreaChart')}"
|
||||
ng-click="setGraphMode('stackedAreaChart', true)"><i class="fa fa-area-chart"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('lineChart') || isGraphMode('lineWithFocusChart')}"
|
||||
ng-click="paragraph.config.graph.lineWithFocus ? setGraphMode('lineWithFocusChart', true) : setGraphMode('lineChart', true)"><i class="fa fa-line-chart"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('scatterChart')}"
|
||||
ng-click="setGraphMode('scatterChart', true)"><i class="cf cf-scatter-chart"></i>
|
||||
</button>
|
||||
<div id="{{paragraph.id}}_switch"
|
||||
class="btn-group">
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('table')}"
|
||||
ng-click="setGraphMode('table', true)" ><i class="fa fa-table"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('multiBarChart')}"
|
||||
ng-click="setGraphMode('multiBarChart', true)"><i class="fa fa-bar-chart"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('pieChart')}"
|
||||
ng-click="setGraphMode('pieChart', true)"><i class="fa fa-pie-chart"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('stackedAreaChart')}"
|
||||
ng-click="setGraphMode('stackedAreaChart', true)"><i class="fa fa-area-chart"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('lineChart') || isGraphMode('lineWithFocusChart')}"
|
||||
ng-click="paragraph.config.graph.lineWithFocus ? setGraphMode('lineWithFocusChart', true) : setGraphMode('lineChart', true)"><i class="fa fa-line-chart"></i>
|
||||
</button>
|
||||
<button type="button" class="btn btn-default btn-sm"
|
||||
ng-class="{'active': isGraphMode('scatterChart')}"
|
||||
ng-click="setGraphMode('scatterChart', true)"><i class="cf cf-scatter-chart"></i>
|
||||
</button>
|
||||
</div>
|
||||
<span>
|
||||
<button type="button" class="btn btn-default btn-sm" style="margin-left:10px"
|
||||
tooltip="Download Data as TSV" tooltip-placement="bottom"
|
||||
ng-click="exportToTSV()"><i class="fa fa-download"></i>
|
||||
</button>
|
||||
</span>
|
||||
<span ng-if="getGraphMode()!='table'"
|
||||
style="margin-left:5px; cursor:pointer; display: inline-block; vertical-align:top; position: relative; line-height:30px;">
|
||||
<a class="btnText" ng-click="toggleGraphOption()">
|
||||
settings <span ng-class="paragraph.config.graph.optionOpen ? 'fa fa-caret-up' : 'fa fa-caret-down'"></span>
|
||||
</a>
|
||||
</span>
|
||||
</div>
|
||||
<span ng-if="getResultType()=='TABLE' && getGraphMode()!='table' && !asIframe && !viewOnly"
|
||||
style="margin-left:10px; cursor:pointer; display: inline-block; vertical-align:top; position: relative; line-height:30px;">
|
||||
<a class="btnText" ng-click="toggleGraphOption()">
|
||||
settings <span ng-class="paragraph.config.graph.optionOpen ? 'fa fa-caret-up' : 'fa fa-caret-down'"></span>
|
||||
</a>
|
||||
</span>
|
||||
|
|
|
|||
|
|
@ -82,9 +82,6 @@ limitations under the License.
|
|||
<li>
|
||||
<a ng-click="goToSingleParagraph()"><span class="icon-share-alt"></span> Link this paragraph</a>
|
||||
</li>
|
||||
<li>
|
||||
<a ng-click="exportToTSV()"><span class="icon-share-alt"></span> Export to TSV</a>
|
||||
</li>
|
||||
<li>
|
||||
<a ng-click="clearParagraphOutput()"><span class="fa fa-eraser"></span> Clear output</a>
|
||||
</li>
|
||||
|
|
|
|||
|
|
@ -79,8 +79,9 @@ angular.module('zeppelinWebApp')
|
|||
var angularObjectRegistry = {};
|
||||
|
||||
var editorModes = {
|
||||
'ace/mode/python': /^%(\w*\.)?pyspark\s*$/,
|
||||
'ace/mode/python': /^%(\w*\.)?(pyspark|python)\s*$/,
|
||||
'ace/mode/scala': /^%(\w*\.)?spark\s*$/,
|
||||
'ace/mode/r': /^%(\w*\.)?(r|sparkr|knitr)\s*$/,
|
||||
'ace/mode/sql': /^%(\w*\.)?\wql/,
|
||||
'ace/mode/markdown': /^%md/,
|
||||
'ace/mode/sh': /^%sh/
|
||||
|
|
@ -1241,20 +1242,21 @@ angular.module('zeppelinWebApp')
|
|||
manualRowResize: true,
|
||||
editor: false,
|
||||
fillHandle: false,
|
||||
fragmentSelection: true,
|
||||
disableVisualSelection: true,
|
||||
cells: function (row, col, prop) {
|
||||
var cellProperties = {};
|
||||
cellProperties.renderer = function(instance, td, row, col, prop, value, cellProperties) {
|
||||
Handsontable.NumericCell.renderer.apply(this, arguments);
|
||||
if (!isNaN(value)) {
|
||||
cellProperties.type = 'numeric';
|
||||
cellProperties.format = '0,0';
|
||||
cellProperties.editor = false;
|
||||
td.style.textAlign = 'left';
|
||||
} else if (value.length > '%html'.length && '%html ' === value.substring(0, '%html '.length)) {
|
||||
td.innerHTML = value.substring('%html'.length);
|
||||
}
|
||||
};
|
||||
cellProperties.renderer = function(instance, td, row, col, prop, value, cellProperties) {
|
||||
if (!isNaN(value)) {
|
||||
cellProperties.format = '0,0.[00000]';
|
||||
td.style.textAlign = 'left';
|
||||
Handsontable.renderers.NumericRenderer.apply(this, arguments);
|
||||
} else if (value.length > '%html'.length && '%html ' === value.substring(0, '%html '.length)) {
|
||||
td.innerHTML = value.substring('%html'.length);
|
||||
} else {
|
||||
Handsontable.renderers.TextRenderer.apply(this, arguments);
|
||||
}
|
||||
};
|
||||
return cellProperties;
|
||||
}
|
||||
});
|
||||
|
|
|
|||
|
|
@ -43,7 +43,9 @@ angular
|
|||
return function(_editor) {
|
||||
function getEditorMode(text) {
|
||||
var editorModes = {
|
||||
'ace/mode/scala': /^%spark/,
|
||||
'ace/mode/scala': /^%(\w*\.)?spark/,
|
||||
'ace/mode/python': /^%(\w*\.)?(pyspark|python)/,
|
||||
'ace/mode/r': /^%(\w*\.)?(r|sparkr|knitr)/,
|
||||
'ace/mode/sql': /^%(\w*\.)?\wql/,
|
||||
'ace/mode/markdown': /^%md/,
|
||||
'ace/mode/sh': /^%sh/
|
||||
|
|
|
|||
Loading…
Reference in a new issue