From 215dd6f36353a1da04c7f5ea339fb653fb12290b Mon Sep 17 00:00:00 2001 From: Luciano Resende Date: Sun, 5 Jun 2016 13:03:20 -0700 Subject: [PATCH 01/10] [ZEPPELIN-956] Download old spark versions direct from archive ### What is this PR for? Older Apache Spark releases seems to have been removed from mirrors, and thus the build scripts needs to be updated to download older releases from the archives. ### What type of PR is it? [Bug Fix] ### What is the Jira issue? [ZEPPELIN-956](https://issues.apache.org/jira/browse/ZEPPELIN-956) ### How should this be tested? Existing build tests Author: Luciano Resende Closes #967 from lresende/download and squashes the following commits: 4fcbf7b [Luciano Resende] [ZEPPELIN-956] Download old spark versions direct from archive --- testing/downloadSpark.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/testing/downloadSpark.sh b/testing/downloadSpark.sh index 004e1dd483..0575284203 100755 --- a/testing/downloadSpark.sh +++ b/testing/downloadSpark.sh @@ -76,15 +76,17 @@ if [[ ! -d "${SPARK_HOME}" ]]; then echo "${SPARK_CACHE} does not have ${SPARK_ARCHIVE} downloading ..." # download archive if not cached - if [[ "${SPARK_VER_RANGE}" == "<=1.2" ]]; then - # spark 1.1.x and spark 1.2.x can be downloaded from archive + if [[ "${SPARK_VERSION}" = "1.1.1" || "${SPARK_VERSION}" = "1.2.2" || "${SPARK_VERSION}" = "1.3.1" || "${SPARK_VERSION}" = "1.4.1" ]]; then + echo "${SPARK_VERSION} being downloaded from archives" + # spark old versions are only available only on the archives (prior to 1.5.2) STARTTIME=`date +%s` #timeout -s KILL "${MAX_DOWNLOAD_TIME_SEC}" wget "http://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_ARCHIVE}.tgz" download_with_retry "http://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_ARCHIVE}.tgz" ENDTIME=`date +%s` DOWNLOADTIME="$((ENDTIME-STARTTIME))" else - # spark 1.3.x and later can be downloaded from mirror + echo "${SPARK_VERSION} being downloaded from mirror" + # spark 1.5.2 and up and later can be downloaded from mirror # get download address from mirror MIRROR_INFO=$(curl -s "http://www.apache.org/dyn/closer.cgi/spark/spark-${SPARK_VERSION}/${SPARK_ARCHIVE}.tgz?asjson=1") From 64085f59c90eb27fa2f3e628f6ab60123adb5744 Mon Sep 17 00:00:00 2001 From: Alexander Bezzubov Date: Mon, 6 Jun 2016 06:56:41 +0200 Subject: [PATCH 02/10] Update external links to project code style guides ### What is this PR for? Update external links to project code style guides in CONTRIBUTING.md as old ones give 404 now ### What type of PR is it? Bug Fix ### How should this be tested? Lins should be clickable and return 200 ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Alexander Bezzubov Closes #968 from bzz/fix/update-styleguid-links and squashes the following commits: f5f8155 [Alexander Bezzubov] Update external links to project code styleguides --- CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 30c76f7de1..ed2849eaaf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -125,8 +125,8 @@ please check [its own contribution guidelines](https://github.com/apache/incubat ### Code convention We are following Google Code style: -* [Java style](http://google-styleguide.googlecode.com/svn/trunk/javaguide.html) -* [Shell style](https://google-styleguide.googlecode.com/svn/trunk/shell.xml) +* [Java style](https://google.github.io/styleguide/javaguide.html) +* [Shell style](https://google.github.io/styleguide/shell.xml) Check style report location are in `${submodule}/target/site/checkstyle.html` Test coverage report location are in `${submodule}/target/site/cobertura/index.html` From 36b3b8d1560669d1d156a93319ebde63922a1f26 Mon Sep 17 00:00:00 2001 From: Prabhjyot Singh Date: Sat, 4 Jun 2016 10:47:06 +0530 Subject: [PATCH 03/10] Move zeppelin version from home page to about section ### What is this PR for? Move zeppelin version that is shown home page to new about section. ### What type of PR is it? [Improvement] ### Todos N/A ### What is the Jira issue? [ZEPPELIN-943](https://issues.apache.org/jira/browse/ZEPPELIN-943) ### How should this be tested? Please refer the gif below. ### Screenshots (if appropriate) Before: ![screen shot 2016-06-02 at 3 43 54 pm](https://cloud.githubusercontent.com/assets/674497/15741904/7420d11e-28d9-11e6-98b7-e34c9f83a7e3.png) After: Without authentication ![gif1](https://cloud.githubusercontent.com/assets/674497/15741910/787609e6-28d9-11e6-9d86-71457afdfc8b.gif) With authentication ![gif2](https://cloud.githubusercontent.com/assets/674497/15741911/78b7e050-28d9-11e6-83b3-570d3d16db31.gif) ### Questions: * Does the licenses files need update? N/A * Is there breaking changes for older versions? N/A * Does this needs documentation? N/A Author: Prabhjyot Singh Closes #951 from prabhjyotsingh/ZEPPELIN-943 and squashes the following commits: 70aaff4 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-943 e714e66 [Prabhjyot Singh] implemeting feedback 5ed77c9 [Prabhjyot Singh] Move zeppelin version from home page to about section --- zeppelin-web/src/app/home/home.controller.js | 11 --- zeppelin-web/src/app/home/home.css | 27 ++++++- zeppelin-web/src/app/home/home.html | 2 +- .../components/navbar/navbar.controller.js | 15 ++++ .../src/components/navbar/navbar.html | 81 ++++++++++++++++--- 5 files changed, 110 insertions(+), 26 deletions(-) diff --git a/zeppelin-web/src/app/home/home.controller.js b/zeppelin-web/src/app/home/home.controller.js index 589370ce95..ddd0f805b4 100644 --- a/zeppelin-web/src/app/home/home.controller.js +++ b/zeppelin-web/src/app/home/home.controller.js @@ -28,19 +28,8 @@ angular.module('zeppelinWebApp').controller('HomeCtrl', function($scope, noteboo $scope.isReloading = false; - var getZeppelinVersion = function() { - $http.get(baseUrlSrv.getRestApiBase() +'/version'). - success(function (data, status, headers, config) { - $scope.zeppelinVersion = data.body; - }). - error(function(data, status, headers, config) { - console.log('Error %o %o', status, data.message); - }); - }; - var initHome = function() { websocketMsgSrv.getHomeNotebook(); - getZeppelinVersion(); }; initHome(); diff --git a/zeppelin-web/src/app/home/home.css b/zeppelin-web/src/app/home/home.css index c37febc526..02b4d04fc1 100644 --- a/zeppelin-web/src/app/home/home.css +++ b/zeppelin-web/src/app/home/home.css @@ -272,9 +272,10 @@ a.navbar-brand:hover { } } -.server-status { +i.server-status { font-size: 12px; - margin-top: 8px; + top: -2px; + position: relative; } .server-connected { @@ -756,3 +757,25 @@ This part should be removed when new version of bootstrap handles this issue. .bootstrap-dialog.type-primary .modal-header { background: #3071a9; } + +.about{ + height: 200px; + padding-top: 20px; + padding-left: 20px; +} + +.about .logo { + width: 20%; + float: left; +} + +.about .content{ + float: left; + padding-left: 10px; + top: -30px; + position: relative; +} + +.about .logo img{ + width: 100%; +} diff --git a/zeppelin-web/src/app/home/home.html b/zeppelin-web/src/app/home/home.html index 2684e5ba5c..e774e41b08 100644 --- a/zeppelin-web/src/app/home/home.html +++ b/zeppelin-web/src/app/home/home.html @@ -37,7 +37,7 @@ limitations under the License.

- Welcome to Zeppelin! ({{zeppelinVersion}}) + Welcome to Zeppelin!

Zeppelin is web-based notebook that enables interactive data analytics.
You can make beautiful data-driven, interactive, collaborative document with SQL, code and even more!
diff --git a/zeppelin-web/src/components/navbar/navbar.controller.js b/zeppelin-web/src/components/navbar/navbar.controller.js index 50c769f1c3..e750162752 100644 --- a/zeppelin-web/src/components/navbar/navbar.controller.js +++ b/zeppelin-web/src/components/navbar/navbar.controller.js @@ -55,6 +55,9 @@ angular.module('zeppelinWebApp').controller('NavCtrl', function($scope, $rootSco $rootScope.truncatedUsername = $rootScope.ticket.principal.substr(0, MAX_USERNAME_LENGTH) + '..'; } } + if (_.isEmpty($rootScope.truncatedUsername)) { + $rootScope.truncatedUsername = 'Connected'; + } }; $scope.$on('loginSuccess', function(event, param) { @@ -100,9 +103,21 @@ angular.module('zeppelinWebApp').controller('NavCtrl', function($scope, $rootSco } }; + function getZeppelinVersion() { + console.log('version'); + $http.get(baseUrlSrv.getRestApiBase() + '/version').success( + function(data, status, headers, config) { + $rootScope.zeppelinVersion = data.body; + }).error( + function(data, status, headers, config) { + console.log('Error %o %o', status, data.message); + }); + } + vm.loadNotes = loadNotes; vm.isActive = isActive; + getZeppelinVersion(); vm.loadNotes(); $scope.checkUsername(); diff --git a/zeppelin-web/src/components/navbar/navbar.html b/zeppelin-web/src/components/navbar/navbar.html index 5a8e0cd333..71868f2bfe 100644 --- a/zeppelin-web/src/components/navbar/navbar.html +++ b/zeppelin-web/src/components/navbar/navbar.html @@ -61,11 +61,11 @@ limitations under the License. -
+ From 486ed2375ffc0d2986d43d54e62370e823d390d5 Mon Sep 17 00:00:00 2001 From: AhyoungRyu Date: Sat, 4 Jun 2016 20:44:26 -0700 Subject: [PATCH 04/10] [ZEPPELIN-923] Apply new mechanism to ShellInterpreter ### What is this PR for? This PR applies the new interpreter registration mechanism to Shell interpreter. ### What type of PR is it? Improvement ### Todos * [x] - Remove static property definition code lines in `shellInterpreter.java` and add `interpreter-setting.json` under the `shell/src/main/resources` ### What is the Jira issue? [ZEPPELIN-923](https://issues.apache.org/jira/browse/ZEPPELIN-923) ### How should this be tested? 1. apply patch 2. rm -r `interpreter/sh` and `conf/interpreter.json` 3. build source 4. bin/zeppelin-daemon.sh start 5. Run some shell command ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: AhyoungRyu Closes #956 from AhyoungRyu/ZEPPELIN-923 and squashes the following commits: 3df8c7d [AhyoungRyu] ZEPPELIN-923: exclude interpreter-setting.json be3e22e [AhyoungRyu] ZEPPELIN-923: Change defaultValue 5000 -> 60000 b70d650 [AhyoungRyu] ZEPPELIN-923: Apply new interpreter mechanism to Shell interpreter --- pom.xml | 1 + .../apache/zeppelin/shell/ShellInterpreter.java | 15 --------------- shell/src/main/resources/interpreter-setting.json | 15 +++++++++++++++ 3 files changed, 16 insertions(+), 15 deletions(-) create mode 100644 shell/src/main/resources/interpreter-setting.json diff --git a/pom.xml b/pom.xml index 8799ff687a..970f6cbe45 100755 --- a/pom.xml +++ b/pom.xml @@ -475,6 +475,7 @@ conf/zeppelin-env.sh spark-*-bin*/** .spark-dist/** + **/interpreter-setting.json docs/assets/themes/zeppelin/bootstrap/** diff --git a/shell/src/main/java/org/apache/zeppelin/shell/ShellInterpreter.java b/shell/src/main/java/org/apache/zeppelin/shell/ShellInterpreter.java index e3c4d0c0f3..1331225265 100644 --- a/shell/src/main/java/org/apache/zeppelin/shell/ShellInterpreter.java +++ b/shell/src/main/java/org/apache/zeppelin/shell/ShellInterpreter.java @@ -45,27 +45,12 @@ public class ShellInterpreter extends Interpreter { Logger logger = LoggerFactory.getLogger(ShellInterpreter.class); private static final String EXECUTOR_KEY = "executor"; public static final String SHELL_COMMAND_TIMEOUT = "shell.command.timeout.millisecs"; - public static final String DEFAULT_COMMAND_TIMEOUT = "600000"; int commandTimeOut; private static final boolean isWindows = System .getProperty("os.name") .startsWith("Windows"); final String shell = isWindows ? "cmd /c" : "bash -c"; - static { - Interpreter.register( - "sh", - "sh", - ShellInterpreter.class.getName(), - new InterpreterPropertyBuilder() - .add( - SHELL_COMMAND_TIMEOUT, - DEFAULT_COMMAND_TIMEOUT, - "Shell command time out in millisecs. Default = 600000") - .build() - ); - } - public ShellInterpreter(Properties property) { super(property); } diff --git a/shell/src/main/resources/interpreter-setting.json b/shell/src/main/resources/interpreter-setting.json new file mode 100644 index 0000000000..5e9a051a2e --- /dev/null +++ b/shell/src/main/resources/interpreter-setting.json @@ -0,0 +1,15 @@ +[ + { + "group": "sh", + "name": "sh", + "className": "org.apache.zeppelin.shell.ShellInterpreter", + "properties": { + "shell.command.timeout.millisecs": { + "envName": "SHELL_COMMAND_TIMEOUT", + "propertyName": "shell.command.timeout.millisecs", + "defaultValue": "60000", + "description": "Shell command time out in millisecs. Default = 60000" + } + } + } +] \ No newline at end of file From de5f55a3c6460a7941baa14fa10b8fc7c5da76e4 Mon Sep 17 00:00:00 2001 From: Ravi Ranjan Date: Mon, 6 Jun 2016 10:47:07 +0530 Subject: [PATCH 05/10] Auto-suggestion feature for notebook permissions ### What is this PR for? This PR will provide user-friendly way to give notebook permissions. It will show suggestions on the basis of usernames matching the input characters. It will communicate with Shiro to fetch usernames from the configured realms and select usernames matching the input character and shows it to the user as suggestions. Currently, it works with shiro.ini file, LDAP, JDBC. In future we can extend it to work with other realms too. ### What type of PR is it? Improvement ### Todos ### What is the Jira issue? [ZEPPELIN-933](https://issues.apache.org/jira/browse/ZEPPELIN-933) ### How should this be tested? 1. Authenticate Zeppelin with any of the realms mention above(LDAP, JDBC,ini). 2. In notebook permission section, try giving permissions to some notebook and press some initial characters . It should show matching users as suggestions and you should be able to select users from the suggestions. 3. You should be able to give usernames separated by comma and finally save it. ### Screenshots (if appropriate) ![screen shot 2016-06-03 at 5 56 01 pm](https://cloud.githubusercontent.com/assets/7026661/15778542/8a208492-29b4-11e6-8bf9-3a31ce90223a.png) ### Questions: * Does the licenses files need update?no * Is there breaking changes for older versions?no * Does this needs documentation?maybe Author: Ravi Ranjan Author: Prabhjyot Singh Closes #937 from ravicodder/auto-suggestion and squashes the following commits: 4b4c826 [Ravi Ranjan] Merge branch 'master' of https://github.com/apache/incubator-zeppelin into auto-suggestion b4ef169 [Ravi Ranjan] use width in palce of margin-right 30bdae4 [Ravi Ranjan] Add log and make UI look better b7333d6 [Ravi Ranjan] reduce the size of autosugesstion and add sepreator 95a420e [Ravi Ranjan] change codeing style 140af69 [Ravi Ranjan] Merge branch 'master' of https://github.com/apache/incubator-zeppelin into auto-suggestion 6627019 [Ravi Ranjan] Merge branch 'master' of https://github.com/apache/incubator-zeppelin into auto-suggestion cdc50b3 [Ravi Ranjan] Modified notbook.html , include alert on save, modify condition to identify the realm 80ace5f [Prabhjyot Singh] aa ef3658c [Ravi Ranjan] add GetUserList.java file 060f27a [Ravi Ranjan] add auto-suggest option --- .../org/apache/zeppelin/rest/GetUserList.java | 153 +++++++++++ .../apache/zeppelin/rest/SecurityRestApi.java | 62 ++++- .../src/app/notebook/notebook.controller.js | 256 ++++++++++++++++-- zeppelin-web/src/app/notebook/notebook.css | 63 +++++ zeppelin-web/src/app/notebook/notebook.html | 46 +++- 5 files changed, 552 insertions(+), 28 deletions(-) create mode 100644 zeppelin-server/src/main/java/org/apache/zeppelin/rest/GetUserList.java diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/GetUserList.java b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/GetUserList.java new file mode 100644 index 0000000000..c603fe977f --- /dev/null +++ b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/GetUserList.java @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zeppelin.rest; + +import org.apache.commons.lang3.reflect.FieldUtils; +import org.apache.shiro.realm.jdbc.JdbcRealm; +import org.apache.shiro.realm.ldap.JndiLdapContextFactory; +import org.apache.shiro.realm.ldap.JndiLdapRealm; +import org.apache.shiro.realm.text.IniRealm; +import org.apache.shiro.util.JdbcUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.naming.NamingEnumeration; +import javax.naming.directory.Attributes; +import javax.naming.directory.SearchControls; +import javax.naming.directory.SearchResult; +import javax.naming.ldap.LdapContext; +import javax.sql.DataSource; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; +import java.util.Map; + +/** + * This is class which help fetching users from different realms. + * getUserList() function is overloaded and according to the realm passed to the function it + * extracts users from its respective realm + */ +public class GetUserList { + + private static final Logger LOG = LoggerFactory.getLogger(GetUserList.class); + + /** + * function to extract users from shiro.ini + */ + public List getUserList(IniRealm r) { + List userList = new ArrayList<>(); + Map getIniUser = r.getIni().get("users"); + Iterator it = getIniUser.entrySet().iterator(); + while (it.hasNext()) { + Map.Entry pair = (Map.Entry) it.next(); + userList.add(pair.getKey().toString()); + } + return userList; + } + + /** + * function to extract users from LDAP + */ + public List getUserList(JndiLdapRealm r) { + List userList = new ArrayList<>(); + String userDnTemplate = r.getUserDnTemplate(); + String userDn[] = userDnTemplate.split(",", 2); + String userDnPrefix = userDn[0].split("=")[0]; + String userDnSuffix = userDn[1]; + JndiLdapContextFactory CF = (JndiLdapContextFactory) r.getContextFactory(); + try { + LdapContext ctx = CF.getSystemLdapContext(); + SearchControls constraints = new SearchControls(); + constraints.setSearchScope(SearchControls.SUBTREE_SCOPE); + String[] attrIDs = {userDnPrefix}; + constraints.setReturningAttributes(attrIDs); + NamingEnumeration result = ctx.search(userDnSuffix, "(objectclass=*)", constraints); + while (result.hasMore()) { + Attributes attrs = ((SearchResult) result.next()).getAttributes(); + if (attrs.get(userDnPrefix) != null) { + String currentUser = attrs.get(userDnPrefix).toString(); + userList.add(currentUser.split(":")[1]); + } + } + } catch (Exception e) { + LOG.error("Error retrieving User list from Ldap Realm", e); + } + return userList; + } + + /** + * function to extract users from JDBCs + */ + public List getUserList(JdbcRealm obj) { + List userlist = new ArrayList<>(); + PreparedStatement ps = null; + ResultSet rs = null; + DataSource dataSource = null; + String authQuery = ""; + String retval[]; + String tablename = ""; + String username = ""; + String userquery = ""; + try { + dataSource = (DataSource) FieldUtils.readField(obj, "dataSource", true); + authQuery = (String) FieldUtils.readField(obj, "DEFAULT_AUTHENTICATION_QUERY", true); + LOG.info(authQuery); + String authQueryLowerCase = authQuery.toLowerCase(); + retval = authQueryLowerCase.split("from", 2); + if (retval.length >= 2) { + retval = retval[1].split("with|where", 2); + tablename = retval[0]; + retval = retval[1].split("where", 2); + if (retval.length >= 2) + retval = retval[1].split("=", 2); + else + retval = retval[0].split("=", 2); + username = retval[0]; + } + + if (username.equals("") || tablename.equals("")){ + return userlist; + } + + userquery = "select " + username + " from " + tablename; + + } catch (IllegalAccessException e) { + LOG.error("Error while accessing dataSource for JDBC Realm", e); + return null; + } + + try { + Connection con = dataSource.getConnection(); + ps = con.prepareStatement(userquery); + rs = ps.executeQuery(); + while (rs.next()) { + userlist.add(rs.getString(1)); + } + } catch (Exception e) { + LOG.error("Error retrieving User list from JDBC Realm", e); + } finally { + JdbcUtils.closeResultSet(rs); + JdbcUtils.closeStatement(ps); + } + return userlist; + } + +} diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java index 342cb00c6a..e3449567c8 100644 --- a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java +++ b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java @@ -17,6 +17,13 @@ package org.apache.zeppelin.rest; + +import org.apache.shiro.realm.Realm; +import org.apache.shiro.realm.jdbc.JdbcRealm; +import org.apache.shiro.realm.ldap.JndiLdapRealm; +import org.apache.shiro.realm.text.IniRealm; +import org.apache.shiro.util.ThreadContext; +import org.apache.shiro.web.mgt.DefaultWebSecurityManager; import org.apache.zeppelin.annotation.ZeppelinApi; import org.apache.zeppelin.conf.ZeppelinConfiguration; import org.apache.zeppelin.server.JsonResponse; @@ -27,11 +34,10 @@ import org.slf4j.LoggerFactory; import javax.ws.rs.GET; import javax.ws.rs.Path; +import javax.ws.rs.PathParam; import javax.ws.rs.Produces; import javax.ws.rs.core.Response; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Map; +import java.util.*; /** * Zeppelin security rest api endpoint. @@ -81,4 +87,54 @@ public class SecurityRestApi { LOG.warn(response.toString()); return response.build(); } + + /** + * Get userlist + * Returns list of all user from available realms + * + * @return 200 response + */ + @GET + @Path("userlist/{searchText}") + public Response getUserList(@PathParam("searchText") String searchText) { + + List usersList = new ArrayList<>(); + try { + GetUserList getUserListObj = new GetUserList(); + DefaultWebSecurityManager defaultWebSecurityManager; + String key = ThreadContext.SECURITY_MANAGER_KEY; + defaultWebSecurityManager = (DefaultWebSecurityManager) ThreadContext.get(key); + Collection realms = defaultWebSecurityManager.getRealms(); + List realmsList = new ArrayList(realms); + for (int i = 0; i < realmsList.size(); i++) { + String name = ((Realm) realmsList.get(i)).getName(); + if (name.equals("iniRealm")) { + usersList.addAll(getUserListObj.getUserList((IniRealm) realmsList.get(i))); + } else if (name.equals("ldapRealm")) { + usersList.addAll(getUserListObj.getUserList((JndiLdapRealm) realmsList.get(i))); + } else if (name.equals("jdbcRealm")) { + usersList.addAll(getUserListObj.getUserList((JdbcRealm) realmsList.get(i))); + } + } + + } catch (Exception e) { + LOG.error("Exception in retrieving Users from realms ", e); + } + List autoSuggestList = new ArrayList<>(); + Collections.sort(usersList); + int maxLength = 0; + for (int i = 0; i < usersList.size(); i++) { + String userLowerCase = usersList.get(i).toLowerCase(); + String searchTextLowerCase = searchText.toLowerCase(); + if (userLowerCase.indexOf(searchTextLowerCase) != -1) { + maxLength++; + autoSuggestList.add(usersList.get(i)); + } + if (maxLength == 5) { + break; + } + } + return new JsonResponse<>(Response.Status.OK, "", autoSuggestList).build(); + } + } diff --git a/zeppelin-web/src/app/notebook/notebook.controller.js b/zeppelin-web/src/app/notebook/notebook.controller.js index 4149311c3b..32c659f33e 100644 --- a/zeppelin-web/src/app/notebook/notebook.controller.js +++ b/zeppelin-web/src/app/notebook/notebook.controller.js @@ -43,6 +43,18 @@ angular.module('zeppelinWebApp').controller('NotebookCtrl', var connectedOnce = false; + // user auto complete related + $scope.suggestions = []; + $scope.selectIndex = -1; + var selectedUser = ''; + var selectedUserIndex = 0; + var previousSelectedList = []; + var previousSelectedListOwners = []; + var previousSelectedListReaders = []; + var previousSelectedListWriters = []; + var searchText = []; + $scope.role = ''; + $scope.$on('setConnectedStatus', function(event, param) { if(connectedOnce && param){ initNotebook(); @@ -683,36 +695,57 @@ angular.module('zeppelinWebApp').controller('NotebookCtrl', } }; - $scope.savePermissions = function() { - $http.put(baseUrlSrv.getRestApiBase() + '/notebook/' +$scope.note.id + '/permissions', + function convertPermissionsToArray() { + if (!angular.isArray($scope.permissions.owners)) { + $scope.permissions.owners = $scope.permissions.owners.split(','); + } + if (!angular.isArray($scope.permissions.readers)) { + $scope.permissions.readers = $scope.permissions.readers.split(','); + } + if (!angular.isArray($scope.permissions.writers)) { + $scope.permissions.writers = $scope.permissions.writers.split(','); + } + } + + $scope.savePermissions = function () { + convertPermissionsToArray(); + $http.put(baseUrlSrv.getRestApiBase() + '/notebook/' + $scope.note.id + '/permissions', $scope.permissions, {withCredentials: true}). - success(function(data, status, headers, config) { - console.log('Note permissions %o saved', $scope.permissions); - $scope.showPermissions = false; - }). - error(function(data, status, headers, config) { - console.log('Error %o %o', status, data.message); - BootstrapDialog.show({ + success(function (data, status, headers, config) { + console.log('Note permissions %o saved', $scope.permissions); + BootstrapDialog.alert({ closable: true, - title: 'Insufficient privileges', + title: 'Permissions Saved Successfully!!!', + message: 'Owners : ' + $scope.permissions.owners + '\n\n' + 'Readers : ' + $scope.permissions.readers + '\n\n' + 'Writers : ' + $scope.permissions.writers + }); + $scope.showPermissions = false; + }). + error(function (data, status, headers, config) { + console.log('Error %o %o', status, data.message); + BootstrapDialog.show({ + closable: true, + title: 'Insufficient privileges', message: data.message, - buttons: [{ + buttons: [ + { label: 'Login', - action: function(dialog) { - dialog.close(); - angular.element('#loginModal').modal({ - show: 'true' - }); + action: function (dialog) { + dialog.close(); + angular.element('#loginModal').modal({ + show: 'true' + }); } - }, { + }, + { label: 'Cancel', - action: function(dialog){ - dialog.close(); + action: function (dialog) { + dialog.close(); } - }] + } + ] + }); }); - }); - }; + }; $scope.togglePermissions = function() { if ($scope.showPermissions) { @@ -739,4 +772,183 @@ angular.module('zeppelinWebApp').controller('NotebookCtrl', } }; + function checkPreviousRole(role) { + var i = 0; + if (role !== $scope.role) { + if ($scope.role === 'owners') { + previousSelectedListOwners = []; + for (i = 0; i < previousSelectedList.length; i++) { + previousSelectedListOwners[i] = previousSelectedList[i]; + } + } + if ($scope.role === 'readers') { + previousSelectedListReaders = []; + for (i = 0; i < previousSelectedList.length; i++) { + previousSelectedListReaders[i] = previousSelectedList[i]; + } + } + if ($scope.role === 'writers') { + previousSelectedListWriters = []; + for (i = 0; i < previousSelectedList.length; i++) { + previousSelectedListWriters[i] = previousSelectedList[i]; + } + } + + $scope.role = role; + previousSelectedList = []; + if (role === 'owners') { + for (i = 0; i < previousSelectedListOwners.length; i++) { + previousSelectedList[i] = previousSelectedListOwners[i]; + } + } + if (role === 'readers') { + for (i = 0; i < previousSelectedListReaders.length; i++) { + previousSelectedList[i] = previousSelectedListReaders[i]; + } + } + if (role === 'writers') { + for (i = 0; i < previousSelectedListWriters.length; i++) { + previousSelectedList[i] = previousSelectedListWriters[i]; + } + } + } + } + + + function convertToArray(role) { + if (role === 'owners') { + searchText = $scope.permissions.owners.split(','); + } + else if (role === 'readers') { + searchText = $scope.permissions.readers.split(','); + } + else if (role === 'writers') { + searchText = $scope.permissions.writers.split(','); + } + for (var i = 0; i < searchText.length; i++) { + searchText[i] = searchText[i].trim(); + } + } + + + function convertToString(role) { + if (role === 'owners') { + $scope.permissions.owners = searchText.join(); + } + else if (role === 'readers') { + $scope.permissions.readers = searchText.join(); + } + else if (role === 'writers') { + $scope.permissions.writers = searchText.join(); + } + } + + function getSuggestions (searchQuery) { + $scope.suggestions =[]; + $http.get(baseUrlSrv.getRestApiBase() + '/security/userlist/' + searchQuery ).then(function + (response) { + var userlist = angular.fromJson(response.data).body; + for (var k in userlist) { + $scope.suggestions.push(userlist[k]); + } + }); + } + + function updatePreviousList() { + for (var i = 0; i < searchText.length; i++) { + previousSelectedList[i] = searchText[i]; + } + } + + + var getChangedIndex = function() { + if (previousSelectedList.length === 0) { + selectedUserIndex = searchText.length - 1; + } + else { + for (var i = 0; i < searchText.length; i++) { + if (previousSelectedList[i] !== searchText[i]) { + selectedUserIndex = i; + previousSelectedList = []; + break; + } + } + } + updatePreviousList(); + }; + + // function to find suggestion list on change + $scope.search = function(role) { + convertToArray(role); + checkPreviousRole(role); + getChangedIndex(); + $scope.selectIndex = -1; + $scope.suggestions = []; + selectedUser = searchText[selectedUserIndex]; + if(selectedUser !== ''){ + getSuggestions(selectedUser); + } + else + { + $scope.suggestions = []; + } + }; + + + var checkIfSelected = function() { + if (($scope.suggestions.length === 0) && ($scope.selectIndex < 0 || $scope.selectIndex >= $scope.suggestions.length) || ( $scope.suggestions.length !== 0 && ( $scope.selectIndex < 0 || $scope.selectIndex >= $scope.suggestions.length ))) { + searchText[selectedUserIndex] = selectedUser; + $scope.suggestions = []; + return true; + } + else { + return false; + } + }; + + + $scope.checkKeyDown = function(event, role) { + if (event.keyCode === 40) { + event.preventDefault(); + if ($scope.selectIndex + 1 !== $scope.suggestions.length) { + $scope.selectIndex++; + } + } + else if (event.keyCode === 38) { + event.preventDefault(); + + if ($scope.selectIndex - 1 !== -1) { + $scope.selectIndex--; + + } + } + else if (event.keyCode === 13) { + event.preventDefault(); + if (!checkIfSelected()) { + selectedUser = $scope.suggestions[$scope.selectIndex]; + searchText[selectedUserIndex] = $scope.suggestions[$scope.selectIndex]; + updatePreviousList(); + convertToString(role); + $scope.suggestions = []; + } + } + }; + + $scope.checkKeyUp = function(event) { + if (event.keyCode !== 8 || event.keyCode !== 46) { + if (searchText[selectedUserIndex] === '') { + $scope.suggestions = []; + } + } + }; + + + $scope.assignValueAndHide = function(index, role) { + searchText[selectedUserIndex] = $scope.suggestions[index]; + updatePreviousList(); + convertToString(role); + $scope.suggestions = []; + }; + + }); diff --git a/zeppelin-web/src/app/notebook/notebook.css b/zeppelin-web/src/app/notebook/notebook.css index 9169e808bc..6cdc40796c 100644 --- a/zeppelin-web/src/app/notebook/notebook.css +++ b/zeppelin-web/src/app/notebook/notebook.css @@ -154,6 +154,20 @@ border: 1px solid #E5E5E5; } +.permissions .owners { + width:60px; + display: inline-block; +} + +.permissions .readers { + width:60px; + display: inline-block; +} + +.permissions .writers { + width:60px; + display: inline-block; +} /* Note Setting Panel */ @@ -222,3 +236,52 @@ text-decoration: none; cursor: default; } + +.userlist { + width: 230px; + font-family: Georgia, Times, serif; + font-size: 15px; + position: absolute; + z-index: 9999; +} + +.userlist ul { + list-style: none; +} + +.userlist ul li { + box-shadow: 3px 3px 5px #888888; + display: list-item; + text-decoration: none; + color: #000000; + background-color: #FFFFFF; + line-height: 30px; + border-bottom-style: none; + border-bottom-width: 1px; + border-bottom:1px #CCCCCC solid; + padding-left: 10px; + cursor: pointer; +} + +.userlist ul li:first-child { + border-top-right-radius: 5px; + border-top-left-radius: 5px; +} + +.userlist ul li:last-child { + border-bottom-right-radius: 5px; + border-bottom-left-radius: 5px; +} + +.userlist ul li strong { + margin-right: 10px; +} + +.userlist li:hover { + background-color: #E0E0E0; +} + +.userlist li:active, +.userlist li.active { + background-color: #428BCA; +} diff --git a/zeppelin-web/src/app/notebook/notebook.html b/zeppelin-web/src/app/notebook/notebook.html index d4b72bccef..de8cbdfe36 100644 --- a/zeppelin-web/src/app/notebook/notebook.html +++ b/zeppelin-web/src/app/notebook/notebook.html @@ -70,9 +70,49 @@ limitations under the License.

-

Owners : Owners can change permissions, read and write the note.

-

Readers : Readers can only read the note.

-

Writers : Writers can read and write the note.

+

Owners Owners can change permissions,read + and write the note.

+
+
    +
  • + {{suggestion}} +
    • +
+
+

Readers Readers can only read the note.

+
+
    +
  • + {{suggestion}} +
    • +
+
+

Writers Writers can read and write the note.

+
+
    +
  • + {{suggestion}} +
    • +
+

From 901102a728147a89e9d133667b0136e740a8fe1c Mon Sep 17 00:00:00 2001 From: Ralph Geerkens Date: Thu, 2 Jun 2016 23:10:54 +0200 Subject: [PATCH 06/10] [ZEPPELIN-915] New registration mechanism applied to JDBCInterpreter ### What is this PR for? This PR applies the new interpreter registration mechanism to the JDBCInterpreter. ### What type of PR is it? Improvement ### Todos * [x ] - Move interpreter registration properties from static block to interpreter-setting.json ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-915 ### How should this be tested? 1. apply patch 2. rm -r interpreter/jdbc 3. rm conf/interpreter.json 4. mvn clean package -DskipTests -pl jdbc 5. bin/zeppelin-daemon.sh start 6. run some paragraph with simple sql queries ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Ralph Geerkens Closes #954 from ralphge/pr915 and squashes the following commits: 63617ef [Ralph Geerkens] [ZEPPELIN-915] New registration mechanism applied to JDBCInterpreter --- .../apache/zeppelin/jdbc/JDBCInterpreter.java | 15 ------- .../main/resources/interpreter-setting.json | 39 +++++++++++++++++++ .../zeppelin/jdbc/JDBCInterpreterTest.java | 13 ++++++- 3 files changed, 51 insertions(+), 16 deletions(-) create mode 100644 jdbc/src/main/resources/interpreter-setting.json diff --git a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java index da1a69dd8e..6495264c26 100644 --- a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java +++ b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java @@ -106,21 +106,6 @@ public class JDBCInterpreter extends Interpreter { private final Map> propertyKeyUnusedConnectionListMap; private final Map paragraphIdConnectionMap; - - static { - Interpreter.register( - "sql", - "jdbc", - JDBCInterpreter.class.getName(), - new InterpreterPropertyBuilder() - .add(DEFAULT_URL, "jdbc:postgresql://localhost:5432/", "The URL for JDBC.") - .add(DEFAULT_USER, "gpadmin", "The JDBC user name") - .add(DEFAULT_PASSWORD, "", - "The JDBC user password") - .add(DEFAULT_DRIVER, "org.postgresql.Driver", "JDBC Driver Name") - .add(COMMON_MAX_LINE, MAX_LINE_DEFAULT, - "Max number of SQL result to display.").build()); - } public JDBCInterpreter(Properties property) { super(property); diff --git a/jdbc/src/main/resources/interpreter-setting.json b/jdbc/src/main/resources/interpreter-setting.json new file mode 100644 index 0000000000..97b2c61810 --- /dev/null +++ b/jdbc/src/main/resources/interpreter-setting.json @@ -0,0 +1,39 @@ +[ + { + "group": "jdbc", + "name": "sql", + "className": "org.apache.zeppelin.jdbc.JDBCInterpreter", + "properties": { + "default.url": { + "envName": null, + "propertyName": "default.url", + "defaultValue": "jdbc:postgresql://localhost:5432/", + "description": "The URL for JDBC." + }, + "default.user": { + "envName": null, + "propertyName": "default.user", + "defaultValue": "gpadmin", + "description": "The JDBC user name" + }, + "default.password": { + "envName": null, + "propertyName": "default.password", + "defaultValue": "", + "description": "The JDBC user password" + }, + "default.driver": { + "envName": null, + "propertyName": "default.driver", + "defaultValue": "org.postgresql.Driver", + "description": "JDBC Driver Name" + }, + "common.max_count": { + "envName": null, + "propertyName": "common.max_count", + "defaultValue": "1000", + "description": "Max number of SQL result to display." + } + } + } +] diff --git a/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java b/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java index 593d968bbb..065f4ed5b8 100644 --- a/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java +++ b/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java @@ -51,6 +51,17 @@ public class JDBCInterpreterTest extends BasicJDBCTestCaseAdapter { } return jdbcConnection; } + + public static Properties getJDBCTestProperties() { + Properties p = new Properties(); + p.setProperty("default.driver", "org.postgresql.Driver"); + p.setProperty("default.url", "jdbc:postgresql://localhost:5432/"); + p.setProperty("default.user", "gpadmin"); + p.setProperty("default.password", ""); + p.setProperty("common.max_count", "1000"); + + return p; + } @Before public void setUp() throws Exception { @@ -116,7 +127,7 @@ public class JDBCInterpreterTest extends BasicJDBCTestCaseAdapter { @Test public void testDefaultProperties() throws SQLException { - JDBCInterpreter jdbcInterpreter = new JDBCInterpreter(new Properties()); + JDBCInterpreter jdbcInterpreter = new JDBCInterpreter(getJDBCTestProperties()); assertEquals("org.postgresql.Driver", jdbcInterpreter.getProperty(DEFAULT_DRIVER)); assertEquals("jdbc:postgresql://localhost:5432/", jdbcInterpreter.getProperty(DEFAULT_URL)); From 6994319a8b209b1b5afbf24b81f517a47e3d86eb Mon Sep 17 00:00:00 2001 From: Zhong Wang Date: Sat, 13 Feb 2016 22:07:59 -0800 Subject: [PATCH 07/10] Exporting to TSV ### What is this PR for? Allow users to export data in a paragraph to a TSV file. There is already a [PR](https://github.com/apache/incubator-zeppelin/pull/6) for this, but it uses DataTables, which requires Flash. ### What type of PR is it? Feature ### Todos * [ ] - Hide/disable the button if it is not in table/chart view * [ ] - Find an icon for the new button * [ ] - Saved file name is not correct in Safari * [ ] - Test with large data set ### Is there a relevant Jira issue? [ZEPPELIN-672](https://issues.apache.org/jira/browse/ZEPPELIN-672) ### How should this be tested? 1. create a paragraph with data in %table view 2. select the "Export to TSV" item in the paragraph settings list ### Screenshots (if appropriate) ![vvzhsyciev](https://cloud.githubusercontent.com/assets/3282033/13032147/d589e88e-d29c-11e5-8763-96fca4db2fd0.gif) ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO Author: Zhong Wang Closes #714 from zhongneu/export-to-tsv and squashes the following commits: 3c9e9e6 [Zhong Wang] fix missing semicolons 089018e [Zhong Wang] first attempt to implement exporting to TSV --- .../notebook/paragraph/paragraph-control.html | 3 +++ .../paragraph/paragraph.controller.js | 19 ++++++++++++++++++- 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/zeppelin-web/src/app/notebook/paragraph/paragraph-control.html b/zeppelin-web/src/app/notebook/paragraph/paragraph-control.html index c35c4e5ade..7c51f3ad6f 100644 --- a/zeppelin-web/src/app/notebook/paragraph/paragraph-control.html +++ b/zeppelin-web/src/app/notebook/paragraph/paragraph-control.html @@ -82,6 +82,9 @@ limitations under the License.
  • Link this paragraph
    • +
  • + Export to TSV +
  • Clear output
    • diff --git a/zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js b/zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js index 3c86737376..c819565810 100644 --- a/zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js +++ b/zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js @@ -16,7 +16,7 @@ angular.module('zeppelinWebApp') .controller('ParagraphCtrl', function($scope,$rootScope, $route, $window, $element, $routeParams, $location, - $timeout, $compile, websocketMsgSrv, ngToast) { + $timeout, $compile, websocketMsgSrv, ngToast, SaveAsService) { var ANGULAR_FUNCTION_OBJECT_NAME_PREFIX = '_Z_ANGULAR_FUNC_'; $scope.parentNote = null; $scope.paragraph = null; @@ -2143,4 +2143,21 @@ angular.module('zeppelinWebApp') $scope.keepScrollDown = false; }; + $scope.exportToTSV = function () { + var data = $scope.paragraph.result; + var tsv = ''; + for (var titleIndex in $scope.paragraph.result.columnNames) { + tsv += $scope.paragraph.result.columnNames[titleIndex].name + '\t'; + } + tsv = tsv.substring(0, tsv.length - 1) + '\n'; + for (var r in $scope.paragraph.result.msgTable) { + var row = $scope.paragraph.result.msgTable[r]; + var tsvRow = ''; + for (var index in row) { + tsvRow += row[index].value + '\t'; + } + tsv += tsvRow.substring(0, tsvRow.length - 1) + '\n'; + } + SaveAsService.SaveAs(tsv, 'data', 'tsv'); + }; }); From e5c5cc2b82276e78935d627e73d900eff0e2501c Mon Sep 17 00:00:00 2001 From: mahmoudelgamal Date: Mon, 6 Jun 2016 10:18:24 +0200 Subject: [PATCH 08/10] [ZEPPELIN-935] Adding more configurations to livy interpreter ### What is this PR for? Extending the livy interpreter to allow manipulation in the configurations of Spark from zeppelin web ui. ### What type of PR is it? Improvement ### Todos * [Test case ] - Task * [Documentation ] - Task ### What is the Jira issue? * [ZEPPELIN-935] ### How should this be tested? - Run livy server in yarn mode or local mode - Start the Zeppelin and go to interpreter page, create livy interpreter with supported configurations ### Screenshots (if appropriate) ![](http://s33.postimg.org/rukcyp1in/Screenshot_from_2016_06_01_11_55_37.png) ![](http://s33.postimg.org/a8chtwtf3/Screenshot_from_2016_06_01_11_23_59.png) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: mahmoudelgamal Author: Fouad Closes #944 from mfelgamal/feature/livy-conf and squashes the following commits: ccf3c82 [mahmoudelgamal] Adding all configurations to doc bb2d5dc [mahmoudelgamal] Adding spark guide link c2b36f5 [mahmoudelgamal] Prefixing all spark properties with livy.* ddd99e1 [mahmoudelgamal] Removing default values to properties which belongs spark f988af0 [mahmoudelgamal] Supporting all spark configurations 72474b9 [mahmoudelgamal] Adding more configurations to livy interpreter 4c0d5f0 [mahmoudelgamal] Adding more configurations to livy interpreter 2586651 [Fouad] Merge pull request #2 from apache/master --- docs/interpreter/livy.md | 60 ++++++++++++++++++- .../org/apache/zeppelin/livy/LivyHelper.java | 34 ++++++----- .../zeppelin/livy/LivySparkInterpreter.java | 17 +++++- 3 files changed, 94 insertions(+), 17 deletions(-) diff --git a/docs/interpreter/livy.md b/docs/interpreter/livy.md index 295a5080f4..225cd817be 100644 --- a/docs/interpreter/livy.md +++ b/docs/interpreter/livy.md @@ -23,6 +23,11 @@ Additional requirements for the Livy interpreter are: * Livy server. ### Configuration +We added some common configurations for spark, and you can set any configuration you want. +This link contains all spark configurations: http://spark.apache.org/docs/latest/configuration.html#available-properties. +And instead of starting property with `spark.` it should be replaced with `livy.spark.`. +Example: `spark.master` to `livy.spark.master` + @@ -30,7 +35,7 @@ Additional requirements for the Livy interpreter are: - + @@ -44,6 +49,56 @@ Additional requirements for the Livy interpreter are: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    PropertyDescription
    zeppelin.livy.masterlivy.spark.master local[*] Spark master uri. ex) spark://masterhost:7077
    1000 Max number of SparkSQL result to display.
    livy.spark.driver.coresDriver cores. ex) 1, 2.
    livy.spark.driver.memoryDriver memory. ex) 512m, 32g.
    livy.spark.executor.instancesExecutor instances. ex) 1, 4.
    livy.spark.executor.coresNum cores per executor. ex) 1, 4.
    livy.spark.executor.memoryExecutor memory per worker instance. ex) 512m, 32g.
    livy.spark.dynamicAllocation.enabledUse dynamic resource allocation. ex) True, False.
    livy.spark.dynamicAllocation.cachedExecutorIdleTimeoutRemove an executor which has cached data blocks.
    livy.spark.dynamicAllocation.minExecutorsLower bound for the number of executors.
    livy.spark.dynamicAllocation.initialExecutorsInitial number of executors to run.
    livy.spark.dynamicAllocation.maxExecutorsUpper bound for the number of executors.
    @@ -105,3 +160,6 @@ The session would have timed out, you may need to restart the interpreter. > Blacklisted configuration values in session config: spark.master edit `conf/spark-blacklist.conf` file in livy server and comment out `#spark.master` line. + +if you choose to work on livy in `apps/spark/java` directory in https://github.com/cloudera/hue , +copy `spark-user-configurable-options.template` to `spark-user-configurable-options.conf` file in livy server and comment out `#spark.master` diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java b/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java index 27fc422948..7f3517eea5 100644 --- a/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java +++ b/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java @@ -20,6 +20,7 @@ package org.apache.zeppelin.livy; import com.google.gson.Gson; import com.google.gson.GsonBuilder; import com.google.gson.reflect.TypeToken; + import org.apache.commons.lang3.StringUtils; import org.apache.http.HttpResponse; import org.apache.http.client.HttpClient; @@ -40,7 +41,9 @@ import java.io.InputStreamReader; import java.nio.charset.Charset; import java.util.ArrayList; import java.util.HashMap; +import java.util.Iterator; import java.util.Map; +import java.util.Map.Entry; import java.util.Properties; @@ -60,27 +63,28 @@ public class LivyHelper { public Integer createSession(InterpreterContext context, String kind) throws Exception { try { + Map conf = new HashMap(); + + Iterator> it = property.entrySet().iterator(); + while (it.hasNext()) { + Entry pair = it.next(); + if (pair.getKey().toString().startsWith("livy.spark.") && + !pair.getValue().toString().isEmpty()) + conf.put(pair.getKey().toString().substring(5), pair.getValue().toString()); + } + + String confData = gson.toJson(conf); + String json = executeHTTP(property.getProperty("zeppelin.livy.url") + "/sessions", - "POST", + "POST", "{" + "\"kind\": \"" + kind + "\", " + - "\"master\": \"" + property.getProperty("zeppelin.livy.master") + "\", " + - "\"proxyUser\": \"" + context.getAuthenticationInfo().getUser() + "\"" + + "\"conf\": " + confData + ", " + + "\"proxyUser\": " + context.getAuthenticationInfo().getUser() + "}", context.getParagraphId() ); - if (json.contains("CreateInteractiveRequest[\\\"master\\\"]")) { - json = executeHTTP(property.getProperty("zeppelin.livy.url") + "/sessions", - "POST", - "{" + - "\"kind\": \"" + kind + "\", " + - "\"conf\":{\"spark.master\": \"" - + property.getProperty("zeppelin.livy.master") + "\"}," + - "\"proxyUser\": \"" + context.getAuthenticationInfo().getUser() + "\"" + - "}", - context.getParagraphId() - ); - } + Map jsonMap = (Map) gson.fromJson(json, new TypeToken>() { }.getType()); diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java index 23a6379beb..0a29d795f6 100644 --- a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java +++ b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java @@ -45,7 +45,22 @@ public class LivySparkInterpreter extends Interpreter { LivySparkInterpreter.class.getName(), new InterpreterPropertyBuilder() .add("zeppelin.livy.url", DEFAULT_URL, "The URL for Livy Server.") - .add("zeppelin.livy.master", LOCAL, "Spark master uri. ex) spark://masterhost:7077") + .add("livy.spark.master", LOCAL, "Spark master uri. ex) spark://masterhost:7077") + .add("livy.spark.driver.cores", "", "Driver cores. ex) 1, 2") + .add("livy.spark.driver.memory", "", "Driver memory. ex) 512m, 32g") + .add("livy.spark.executor.instances", "", "Executor instances. ex) 1, 4") + .add("livy.spark.executor.cores", "", "Num cores per executor. ex) 1, 4") + .add("livy.spark.executor.memory", "", + "Executor memory per worker instance. ex) 512m, 32g") + .add("livy.spark.dynamicAllocation.enabled", "", "Use dynamic resource allocation") + .add("livy.spark.dynamicAllocation.cachedExecutorIdleTimeout", "", + "Remove an executor which has cached data blocks") + .add("livy.spark.dynamicAllocation.minExecutors", "", + "Lower bound for the number of executors if dynamic allocation is enabled. ") + .add("livy.spark.dynamicAllocation.initialExecutors", "", + "Initial number of executors to run if dynamic allocation is enabled. ") + .add("livy.spark.dynamicAllocation.maxExecutors", "", + "Upper bound for the number of executors if dynamic allocation is enabled. ") .build() ); } From d3d93e7b132ec2ca2b202b39e8e35805858692e7 Mon Sep 17 00:00:00 2001 From: AhyoungRyu Date: Mon, 6 Jun 2016 16:09:15 -0700 Subject: [PATCH 09/10] Update README.md for fixing ignite flag ### What is this PR for? While Apache Ignite version in Zeppelin was bumped up by #916, but ignite build flag in [README.md](https://github.com/apache/incubator-zeppelin/blob/master/README.md#ignite-interpreter) is outdated so far. So I just changed `1.1.0-incubating` -> `1.6.0`. ### What type of PR is it? Hot Fix ### Todos ### What is the Jira issue? ### How should this be tested? Just see [here](https://github.com/apache/incubator-zeppelin/blob/master/README.md#ignite-interpreter). ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: AhyoungRyu Closes #970 from AhyoungRyu/update/README.md and squashes the following commits: 20d0155 [AhyoungRyu] Update README.md for fixing ignite flag --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 465fd69767..0a27bf85b0 100644 --- a/README.md +++ b/README.md @@ -196,7 +196,7 @@ mvn clean package -Pspark-1.5 -Pmapr50 -DskipTests #### Ignite Interpreter ```sh -mvn clean package -Dignite.version=1.1.0-incubating -DskipTests +mvn clean package -Dignite.version=1.6.0 -DskipTests ``` #### Scalding Interpreter From 70ab1a376d8b7ac136e12ac72e995af54d7d25d8 Mon Sep 17 00:00:00 2001 From: Mina Lee Date: Sat, 4 Jun 2016 21:54:02 -0700 Subject: [PATCH 10/10] [ZEPPELIN-952] Refine website style ### What is this PR for? - update document style (font, line-spacing) - apply same formats for documents - fix broke document styles ### What type of PR is it? Documentation ### What is the Jira issue? [ZEPPELIN-952](https://issues.apache.org/jira/browse/ZEPPELIN-952) ### Screenshots (if appropriate) **Before** screen shot 2016-06-04 at 9 51 38 pm **After** screen shot 2016-06-04 at 9 15 08 pm **Before** screen shot 2016-06-04 at 10 08 53 pm **After** screen shot 2016-06-04 at 10 08 18 pm **Before** screen shot 2016-06-04 at 10 10 47 pm **After** screen shot 2016-06-04 at 10 11 22 pm **Before** screen shot 2016-06-04 at 10 12 36 pm **After** screen shot 2016-06-04 at 10 12 55 pm ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Mina Lee Closes #962 from minahlee/ZEPPELIN-952 and squashes the following commits: f9bee91 [Mina Lee] Capitalize hawq 72481bd [Mina Lee] Update doc titles 495a074 [Mina Lee] remove old style.css 27ca869 [Mina Lee] use code block for file location in spark.md eb821f1 [Mina Lee] Change file location and rename file 72f8ec3 [Mina Lee] change storage doc layout and fix pre block 4202208 [Mina Lee] Apply same format for rest api docs 5875066 [Mina Lee] split display page into text and html 8bc5a6e [Mina Lee] prettify document 0cb953e [Mina Lee] remove incubating tag --- .../themes/zeppelin/_navigation.html | 13 +- docs/_includes/themes/zeppelin/default.html | 2 +- docs/assets/themes/zeppelin/css/style.css | 452 ++++++++++-------- docs/development/howtocontributewebsite.md | 4 +- docs/displaysystem/html.md | 25 + docs/displaysystem/table.md | 3 +- docs/displaysystem/{display.md => text.md} | 18 +- docs/index.md | 4 +- docs/interpreter/spark.md | 10 +- docs/manual/dynamicform.md | 12 +- .../explorezeppelinui.md} | 12 +- docs/{tutorial => quickstart}/tutorial.md | 2 +- docs/rest-api/rest-configuration.md | 113 ++--- docs/rest-api/rest-interpreter.md | 55 +-- docs/rest-api/rest-notebook.md | 275 ++++++----- docs/storage/storage.md | 18 +- 16 files changed, 520 insertions(+), 498 deletions(-) create mode 100644 docs/displaysystem/html.md rename docs/displaysystem/{display.md => text.md} (78%) rename docs/{ui_layout/zeppelin_layout.md => quickstart/explorezeppelinui.md} (96%) rename docs/{tutorial => quickstart}/tutorial.md (99%) diff --git a/docs/_includes/themes/zeppelin/_navigation.html b/docs/_includes/themes/zeppelin/_navigation.html index acaa89d908..179ede8ae1 100644 --- a/docs/_includes/themes/zeppelin/_navigation.html +++ b/docs/_includes/themes/zeppelin/_navigation.html @@ -20,6 +20,8 @@