Elgato_dark/zeppelin
1
0
Fork 0
mirror of https://github.com/apache/zeppelin synced 2026-05-24 09:38:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

merged with master

Browse source
This commit is contained in:
vgmartinez 2016-01-17 18:48:47 +01:00
parent e602621bd3
commit 8046692177
21 changed files with 2328 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
SECURITY-README.md Normal file
View file

@ -0,0 +1,43 @@
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# Shiro Authentication
To connect to Zeppelin, users will be asked to enter their credentials. Once logged, a user has access to all notes including other users notes.
This a a first step toward full security as implemented by this pull request (https://github.com/apache/incubator-zeppelin/pull/53).
# Security setup
1. Secure the HTTP channel: Comment the line "/** = anon" and uncomment the line "/** = authcBasic" in the file conf/shiro.ini. Read more about he shiro.ini file format at the following URL http://shiro.apache.org/configuration.html#Configuration-INISections.
2. Secure the Websocket channel : Set to property "zeppelin.anonymous.allowed" to "false" in the file conf/zeppelin-site.xml. You can start by renaming conf/zeppelin-site.xml.template to conf/zeppelin-site.xml
3. Start Zeppelin : bin/zeppelin.sh
4. point your browser to http://localhost:8080
5. Login using one of the user/password combinations defined in the conf/shiro.ini file.
# Implementation notes
## Vocabulary
username, owner and principal are used interchangeably to designate the currently authenticated user
## What are we securing ?
Zeppelin is basically a web application that spawn remote interpreters to run commands and return HTML fragments to be displayed on the user browser.
The scope of this PR is to require credentials to access Zeppelin. To achieve this, we use Apache Shiro.
## HTTP Endpoint security
Apache Shiro sits as a servlet filter between the browser and the exposed services and handles the required authentication without any programming required. (See Apache Shiro for more info).
## Websocket security
Securing the HTTP endpoints is not enough, since Zeppelin also communicates with the browser through websockets. To secure this channel, we take the following approach:
1. The browser on startup requests a ticket through HTTP
2. The Apache Shiro Servlet filter handles the user auth
3. Once the user is authenticated, a ticket is assigned to this user and the ticket is returned to the browser
All websockets communications require the username and ticket to be submitted by the browser. Upon receiving a websocket message, the server checks that the ticket received is the one assigned to the username through the HTTP request (step 3 above).

38
conf/shiro.ini Normal file
View file

@ -0,0 +1,38 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
[users]
# List of users with their password allowed to access Zeppelin.
# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections
admin = password1
user1 = password2
user2 = password3
# Sample LDAP configuration, for user Authentication, currently tested for single Realm
[main]
#ldapRealm = org.apache.shiro.realm.ldap.JndiLdapRealm
#ldapRealm.userDnTemplate = cn={0},cn=engg,ou=testdomain,dc=testdomain,dc=com
#ldapRealm.contextFactory.url = ldap://ldaphost:389
#ldapRealm.contextFactory.authenticationMechanism = SIMPLE
[urls]
# anon means the access is anonymous.
# authcBasic means Basic Auth Security
# To enfore security, comment the line below and uncomment the next one
/** = anon
#/** = authcBasic

342
postgresql/src/main/java/org/apache/zeppelin/postgresql/PostgreSqlInterpreter.java Normal file
View file

@ -0,0 +1,342 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one or more contributor license
* agreements. See the NOTICE file distributed with this work for additional information regarding
* copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance with the License. You may obtain a
* copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied. See the License for the specific language governing permissions and limitations under
* the License.
*/
package org.apache.zeppelin.postgresql;
import static org.apache.commons.lang.StringUtils.containsIgnoreCase;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import java.util.Set;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.scheduler.SchedulerFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.base.Function;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import com.google.common.collect.Sets.SetView;
/**
* PostgreSQL interpreter for Zeppelin. This interpreter can also be used for accessing HAWQ and
* GreenplumDB.
*
* <ul>
* <li>{@code postgresql.url} - JDBC URL to connect to.</li>
* <li>{@code postgresql.user} - JDBC user name..</li>
* <li>{@code postgresql.password} - JDBC password..</li>
* <li>{@code postgresql.driver.name} - JDBC driver name.</li>
* <li>{@code postgresql.max.result} - Max number of SQL result to display.</li>
* </ul>
*
* <p>
* How to use: <br/>
* {@code %psql.sql} <br/>
* {@code
* SELECT store_id, count(*)
* FROM retail_demo.order_lineitems_pxf
* GROUP BY store_id;
* }
* </p>
*
* For SQL auto-completion use the (Ctrl+.) shortcut.
*/
public class PostgreSqlInterpreter extends Interpreter {
private Logger logger = LoggerFactory.getLogger(PostgreSqlInterpreter.class);
private static final char WhITESPACE = ' ';
private static final char NEWLINE = '\n';
private static final char TAB = '\t';
private static final String TABLE_MAGIC_TAG = "%table ";
private static final String EXPLAIN_PREDICATE = "EXPLAIN ";
private static final String UPDATE_COUNT_HEADER = "Update Count";
static final String DEFAULT_JDBC_URL = "jdbc:postgresql://localhost:5432/";
static final String DEFAULT_JDBC_USER_PASSWORD = "";
static final String DEFAULT_JDBC_USER_NAME = "gpadmin";
static final String DEFAULT_JDBC_DRIVER_NAME = "org.postgresql.Driver";
static final String DEFAULT_MAX_RESULT = "1000";
static final String POSTGRESQL_SERVER_URL = "postgresql.url";
static final String POSTGRESQL_SERVER_USER = "postgresql.user";
static final String POSTGRESQL_SERVER_PASSWORD = "postgresql.password";
static final String POSTGRESQL_SERVER_DRIVER_NAME = "postgresql.driver.name";
static final String POSTGRESQL_SERVER_MAX_RESULT = "postgresql.max.result";
static final String EMPTY_COLUMN_VALUE = "";
static {
Interpreter.register(
"sql",
"psql",
PostgreSqlInterpreter.class.getName(),
new InterpreterPropertyBuilder()
.add(POSTGRESQL_SERVER_URL, DEFAULT_JDBC_URL, "The URL for PostgreSQL.")
.add(POSTGRESQL_SERVER_USER, DEFAULT_JDBC_USER_NAME, "The PostgreSQL user name")
.add(POSTGRESQL_SERVER_PASSWORD, DEFAULT_JDBC_USER_PASSWORD,
"The PostgreSQL user password")
.add(POSTGRESQL_SERVER_DRIVER_NAME, DEFAULT_JDBC_DRIVER_NAME, "JDBC Driver Name")
.add(POSTGRESQL_SERVER_MAX_RESULT, DEFAULT_MAX_RESULT,
"Max number of SQL result to display.").build());
}
private Connection jdbcConnection;
private Statement currentStatement;
private Exception exceptionOnConnect;
private int maxResult;
private SqlCompleter sqlCompleter;
private static final Function<CharSequence, String> sequenceToStringTransformer =
new Function<CharSequence, String>() {
public String apply(CharSequence seq) {
return seq.toString();
}
};
private static final List<String> NO_COMPLETION = new ArrayList<String>();
public PostgreSqlInterpreter(Properties property) {
super(property);
}
@Override
public void open() {
logger.info("Open psql connection!");
// Ensure that no previous connections are left open.
close();
try {
String driverName = getProperty(POSTGRESQL_SERVER_DRIVER_NAME);
String url = getProperty(POSTGRESQL_SERVER_URL);
String user = getProperty(POSTGRESQL_SERVER_USER);
String password = getProperty(POSTGRESQL_SERVER_PASSWORD);
maxResult = Integer.valueOf(getProperty(POSTGRESQL_SERVER_MAX_RESULT));
Class.forName(driverName);
jdbcConnection = DriverManager.getConnection(url, user, password);
sqlCompleter = createSqlCompleter(jdbcConnection);
exceptionOnConnect = null;
logger.info("Successfully created psql connection");
} catch (ClassNotFoundException | SQLException e) {
logger.error("Cannot open connection", e);
exceptionOnConnect = e;
close();
}
}
private SqlCompleter createSqlCompleter(Connection jdbcConnection) {
SqlCompleter completer = null;
try {
Set<String> keywordsCompletions = SqlCompleter.getSqlKeywordsCompletions(jdbcConnection);
Set<String> dataModelCompletions =
SqlCompleter.getDataModelMetadataCompletions(jdbcConnection);
SetView<String> allCompletions = Sets.union(keywordsCompletions, dataModelCompletions);
completer = new SqlCompleter(allCompletions, dataModelCompletions);
} catch (IOException | SQLException e) {
logger.error("Cannot create SQL completer", e);
}
return completer;
}
@Override
public void close() {
logger.info("Close psql connection!");
try {
if (getJdbcConnection() != null) {
getJdbcConnection().close();
}
} catch (SQLException e) {
logger.error("Cannot close connection", e);
} finally {
exceptionOnConnect = null;
}
}
private InterpreterResult executeSql(String sql) {
try {
if (exceptionOnConnect != null) {
return new InterpreterResult(Code.ERROR, exceptionOnConnect.getMessage());
}
currentStatement = getJdbcConnection().createStatement();
currentStatement.setMaxRows(maxResult);
StringBuilder msg = null;
boolean isTableType = false;
if (containsIgnoreCase(sql, EXPLAIN_PREDICATE)) {
msg = new StringBuilder();
} else {
msg = new StringBuilder(TABLE_MAGIC_TAG);
isTableType = true;
}
ResultSet resultSet = null;
try {
boolean isResultSetAvailable = currentStatement.execute(sql);
if (isResultSetAvailable) {
resultSet = currentStatement.getResultSet();
ResultSetMetaData md = resultSet.getMetaData();
for (int i = 1; i < md.getColumnCount() + 1; i++) {
if (i > 1) {
msg.append(TAB);
}
msg.append(replaceReservedChars(isTableType, md.getColumnName(i)));
}
msg.append(NEWLINE);
int displayRowCount = 0;
while (resultSet.next() && displayRowCount < getMaxResult()) {
for (int i = 1; i < md.getColumnCount() + 1; i++) {
msg.append(replaceReservedChars(isTableType, resultSet.getString(i)));
if (i != md.getColumnCount()) {
msg.append(TAB);
}
}
msg.append(NEWLINE);
displayRowCount++;
}
} else {
// Response contains either an update count or there are no results.
int updateCount = currentStatement.getUpdateCount();
msg.append(UPDATE_COUNT_HEADER).append(NEWLINE);
msg.append(updateCount).append(NEWLINE);
// In case of update event (e.g. isResultSetAvailable = false) update the completion
// meta-data.
if (sqlCompleter != null) {
sqlCompleter.updateDataModelMetaData(getJdbcConnection());
}
}
} finally {
try {
if (resultSet != null) {
resultSet.close();
}
currentStatement.close();
} finally {
currentStatement = null;
}
}
return new InterpreterResult(Code.SUCCESS, msg.toString());
} catch (SQLException ex) {
logger.error("Cannot run " + sql, ex);
return new InterpreterResult(Code.ERROR, ex.getMessage());
}
}
/**
* For %table response replace Tab and Newline characters from the content.
*/
private String replaceReservedChars(boolean isTableResponseType, String str) {
if (str == null) {
return EMPTY_COLUMN_VALUE;
}
return (!isTableResponseType) ? str : str.replace(TAB, WhITESPACE).replace(NEWLINE, WhITESPACE);
}
@Override
public InterpreterResult interpret(String cmd, InterpreterContext contextInterpreter) {
logger.info("Run SQL command '{}'", cmd);
return executeSql(cmd);
}
@Override
public void cancel(InterpreterContext context) {
logger.info("Cancel current query statement.");
if (currentStatement != null) {
try {
currentStatement.cancel();
} catch (SQLException ex) {
logger.error("SQLException in PostgreSqlInterpreter while cancel ", ex);
} finally {
currentStatement = null;
}
}
}
@Override
public FormType getFormType() {
return FormType.SIMPLE;
}
@Override
public int getProgress(InterpreterContext context) {
return 0;
}
@Override
public Scheduler getScheduler() {
return SchedulerFactory.singleton().createOrGetFIFOScheduler(
PostgreSqlInterpreter.class.getName() + this.hashCode());
}
@Override
public List<String> completion(String buf, int cursor) {
List<CharSequence> candidates = new ArrayList<CharSequence>();
if (sqlCompleter != null && sqlCompleter.complete(buf, cursor, candidates) >= 0) {
return Lists.transform(candidates, sequenceToStringTransformer);
} else {
return NO_COMPLETION;
}
}
public int getMaxResult() {
return maxResult;
}
// Test only method
protected Connection getJdbcConnection() {
return jdbcConnection;
}
}

181
spark/src/main/java/org/apache/zeppelin/spark/dep/SparkDependencyContext.java Normal file
View file

@ -0,0 +1,181 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.spark.dep;
import java.io.File;
import java.net.MalformedURLException;
import java.util.LinkedList;
import java.util.List;
import org.apache.zeppelin.dep.Booter;
import org.apache.zeppelin.dep.Dependency;
import org.apache.zeppelin.dep.Repository;
import org.sonatype.aether.RepositorySystem;
import org.sonatype.aether.RepositorySystemSession;
import org.sonatype.aether.artifact.Artifact;
import org.sonatype.aether.collection.CollectRequest;
import org.sonatype.aether.graph.DependencyFilter;
import org.sonatype.aether.repository.RemoteRepository;
import org.sonatype.aether.repository.Authentication;
import org.sonatype.aether.resolution.ArtifactResolutionException;
import org.sonatype.aether.resolution.ArtifactResult;
import org.sonatype.aether.resolution.DependencyRequest;
import org.sonatype.aether.resolution.DependencyResolutionException;
import org.sonatype.aether.util.artifact.DefaultArtifact;
import org.sonatype.aether.util.artifact.JavaScopes;
import org.sonatype.aether.util.filter.DependencyFilterUtils;
import org.sonatype.aether.util.filter.PatternExclusionsDependencyFilter;
/**
*
*/
public class SparkDependencyContext {
List<Dependency> dependencies = new LinkedList<Dependency>();
List<Repository> repositories = new LinkedList<Repository>();
List<File> files = new LinkedList<File>();
List<File> filesDist = new LinkedList<File>();
private RepositorySystem system = Booter.newRepositorySystem();
private RepositorySystemSession session;
private RemoteRepository mavenCentral = Booter.newCentralRepository();
private RemoteRepository mavenLocal = Booter.newLocalRepository();
private List<RemoteRepository> additionalRepos = new LinkedList<RemoteRepository>();
public SparkDependencyContext(String localRepoPath, String additionalRemoteRepository) {
session = Booter.newRepositorySystemSession(system, localRepoPath);
addRepoFromProperty(additionalRemoteRepository);
}
public Dependency load(String lib) {
Dependency dep = new Dependency(lib);
if (dependencies.contains(dep)) {
dependencies.remove(dep);
}
dependencies.add(dep);
return dep;
}
public Repository addRepo(String name) {
Repository rep = new Repository(name);
repositories.add(rep);
return rep;
}
public void reset() {
dependencies = new LinkedList<Dependency>();
repositories = new LinkedList<Repository>();
files = new LinkedList<File>();
filesDist = new LinkedList<File>();
}
private void addRepoFromProperty(String listOfRepo) {
if (listOfRepo != null) {
String[] repos = listOfRepo.split(";");
for (String repo : repos) {
String[] parts = repo.split(",");
if (parts.length == 3) {
String id = parts[0].trim();
String url = parts[1].trim();
boolean isSnapshot = Boolean.parseBoolean(parts[2].trim());
if (id.length() > 1 && url.length() > 1) {
RemoteRepository rr = new RemoteRepository(id, "default", url);
rr.setPolicy(isSnapshot, null);
additionalRepos.add(rr);
}
}
}
}
}
/**
* fetch all artifacts
* @return
* @throws MalformedURLException
* @throws ArtifactResolutionException
* @throws DependencyResolutionException
*/
public List<File> fetch() throws MalformedURLException,
DependencyResolutionException, ArtifactResolutionException {
for (Dependency dep : dependencies) {
if (!dep.isLocalFsArtifact()) {
List<ArtifactResult> artifacts = fetchArtifactWithDep(dep);
for (ArtifactResult artifact : artifacts) {
if (dep.isDist()) {
filesDist.add(artifact.getArtifact().getFile());
}
files.add(artifact.getArtifact().getFile());
}
} else {
if (dep.isDist()) {
filesDist.add(new File(dep.getGroupArtifactVersion()));
}
files.add(new File(dep.getGroupArtifactVersion()));
}
}
return files;
}
private List<ArtifactResult> fetchArtifactWithDep(Dependency dep)
throws DependencyResolutionException, ArtifactResolutionException {
Artifact artifact = new DefaultArtifact(
SparkDependencyResolver.inferScalaVersion(dep.getGroupArtifactVersion()));
DependencyFilter classpathFlter = DependencyFilterUtils
.classpathFilter(JavaScopes.COMPILE);
PatternExclusionsDependencyFilter exclusionFilter = new PatternExclusionsDependencyFilter(
SparkDependencyResolver.inferScalaVersion(dep.getExclusions()));
CollectRequest collectRequest = new CollectRequest();
collectRequest.setRoot(new org.sonatype.aether.graph.Dependency(artifact,
JavaScopes.COMPILE));
collectRequest.addRepository(mavenCentral);
collectRequest.addRepository(mavenLocal);
for (RemoteRepository repo : additionalRepos) {
collectRequest.addRepository(repo);
}
for (Repository repo : repositories) {
RemoteRepository rr = new RemoteRepository(repo.getName(), "default", repo.getUrl());
rr.setPolicy(repo.isSnapshot(), null);
Authentication auth = repo.getAuthentication();
if (auth != null) {
rr.setAuthentication(auth);
}
collectRequest.addRepository(rr);
}
DependencyRequest dependencyRequest = new DependencyRequest(collectRequest,
DependencyFilterUtils.andFilter(exclusionFilter, classpathFlter));
return system.resolveDependencies(session, dependencyRequest).getArtifactResults();
}
public List<File> getFiles() {
return files;
}
public List<File> getFilesDist() {
return filesDist;
}
}

351
spark/src/main/java/org/apache/zeppelin/spark/dep/SparkDependencyResolver.java Normal file
View file

@ -0,0 +1,351 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.spark.dep;
import java.io.File;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.net.URL;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import org.apache.commons.lang.StringUtils;
import org.apache.spark.SparkContext;
import org.apache.spark.repl.SparkIMain;
import org.apache.zeppelin.dep.AbstractDependencyResolver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.sonatype.aether.artifact.Artifact;
import org.sonatype.aether.collection.CollectRequest;
import org.sonatype.aether.graph.Dependency;
import org.sonatype.aether.graph.DependencyFilter;
import org.sonatype.aether.repository.RemoteRepository;
import org.sonatype.aether.resolution.ArtifactResult;
import org.sonatype.aether.resolution.DependencyRequest;
import org.sonatype.aether.util.artifact.DefaultArtifact;
import org.sonatype.aether.util.artifact.JavaScopes;
import org.sonatype.aether.util.filter.DependencyFilterUtils;
import org.sonatype.aether.util.filter.PatternExclusionsDependencyFilter;
import scala.Some;
import scala.collection.IndexedSeq;
import scala.reflect.io.AbstractFile;
import scala.tools.nsc.Global;
import scala.tools.nsc.backend.JavaPlatform;
import scala.tools.nsc.util.ClassPath;
import scala.tools.nsc.util.MergedClassPath;
/**
* Deps resolver.
* Add new dependencies from mvn repo (at runtime) to Spark interpreter group.
*/
public class SparkDependencyResolver extends AbstractDependencyResolver {
Logger logger = LoggerFactory.getLogger(SparkDependencyResolver.class);
private Global global;
private SparkIMain intp;
private SparkContext sc;
private final String[] exclusions = new String[] {"org.scala-lang:scala-library",
"org.scala-lang:scala-compiler",
"org.scala-lang:scala-reflect",
"org.scala-lang:scalap",
"org.apache.zeppelin:zeppelin-zengine",
"org.apache.zeppelin:zeppelin-spark",
"org.apache.zeppelin:zeppelin-server"};
public SparkDependencyResolver(SparkIMain intp, SparkContext sc, String localRepoPath,
String additionalRemoteRepository) {
super(localRepoPath);
this.intp = intp;
this.global = intp.global();
this.sc = sc;
addRepoFromProperty(additionalRemoteRepository);
}
private void addRepoFromProperty(String listOfRepo) {
if (listOfRepo != null) {
String[] repos = listOfRepo.split(";");
for (String repo : repos) {
String[] parts = repo.split(",");
if (parts.length == 3) {
String id = parts[0].trim();
String url = parts[1].trim();
boolean isSnapshot = Boolean.parseBoolean(parts[2].trim());
if (id.length() > 1 && url.length() > 1) {
addRepo(id, url, isSnapshot);
}
}
}
}
}
private void updateCompilerClassPath(URL[] urls) throws IllegalAccessException,
IllegalArgumentException, InvocationTargetException {
JavaPlatform platform = (JavaPlatform) global.platform();
MergedClassPath<AbstractFile> newClassPath = mergeUrlsIntoClassPath(platform, urls);
Method[] methods = platform.getClass().getMethods();
for (Method m : methods) {
if (m.getName().endsWith("currentClassPath_$eq")) {
m.invoke(platform, new Some(newClassPath));
break;
}
}
// NOTE: Must use reflection until this is exposed/fixed upstream in Scala
List<String> classPaths = new LinkedList<String>();
for (URL url : urls) {
classPaths.add(url.getPath());
}
// Reload all jars specified into our compiler
global.invalidateClassPathEntries(scala.collection.JavaConversions.asScalaBuffer(classPaths)
.toList());
}
// Until spark 1.1.x
// check https://github.com/apache/spark/commit/191d7cf2a655d032f160b9fa181730364681d0e7
private void updateRuntimeClassPath_1_x(URL[] urls) throws SecurityException,
IllegalAccessException, IllegalArgumentException,
InvocationTargetException, NoSuchMethodException {
ClassLoader cl = intp.classLoader().getParent();
Method addURL;
addURL = cl.getClass().getDeclaredMethod("addURL", new Class[] {URL.class});
addURL.setAccessible(true);
for (URL url : urls) {
addURL.invoke(cl, url);
}
}
private void updateRuntimeClassPath_2_x(URL[] urls) throws SecurityException,
IllegalAccessException, IllegalArgumentException,
InvocationTargetException, NoSuchMethodException {
ClassLoader cl = intp.classLoader().getParent();
Method addURL;
addURL = cl.getClass().getDeclaredMethod("addNewUrl", new Class[] {URL.class});
addURL.setAccessible(true);
for (URL url : urls) {
addURL.invoke(cl, url);
}
}
private MergedClassPath<AbstractFile> mergeUrlsIntoClassPath(JavaPlatform platform, URL[] urls) {
IndexedSeq<ClassPath<AbstractFile>> entries =
((MergedClassPath<AbstractFile>) platform.classPath()).entries();
List<ClassPath<AbstractFile>> cp = new LinkedList<ClassPath<AbstractFile>>();
for (int i = 0; i < entries.size(); i++) {
cp.add(entries.apply(i));
}
for (URL url : urls) {
AbstractFile file;
if ("file".equals(url.getProtocol())) {
File f = new File(url.getPath());
if (f.isDirectory()) {
file = AbstractFile.getDirectory(scala.reflect.io.File.jfile2path(f));
} else {
file = AbstractFile.getFile(scala.reflect.io.File.jfile2path(f));
}
} else {
file = AbstractFile.getURL(url);
}
ClassPath<AbstractFile> newcp = platform.classPath().context().newClassPath(file);
// distinct
if (cp.contains(newcp) == false) {
cp.add(newcp);
}
}
return new MergedClassPath(scala.collection.JavaConversions.asScalaBuffer(cp).toIndexedSeq(),
platform.classPath().context());
}
public List<String> load(String artifact,
boolean addSparkContext) throws Exception {
return load(artifact, new LinkedList<String>(), addSparkContext);
}
public List<String> load(String artifact, Collection<String> excludes,
boolean addSparkContext) throws Exception {
if (StringUtils.isBlank(artifact)) {
// Should throw here
throw new RuntimeException("Invalid artifact to load");
}
// <groupId>:<artifactId>[:<extension>[:<classifier>]]:<version>
int numSplits = artifact.split(":").length;
if (numSplits >= 3 && numSplits <= 6) {
return loadFromMvn(artifact, excludes, addSparkContext);
} else {
loadFromFs(artifact, addSparkContext);
LinkedList<String> libs = new LinkedList<String>();
libs.add(artifact);
return libs;
}
}
private void loadFromFs(String artifact, boolean addSparkContext) throws Exception {
File jarFile = new File(artifact);
intp.global().new Run();
if (sc.version().startsWith("1.1")) {
updateRuntimeClassPath_1_x(new URL[] {jarFile.toURI().toURL()});
} else {
updateRuntimeClassPath_2_x(new URL[] {jarFile.toURI().toURL()});
}
if (addSparkContext) {
sc.addJar(jarFile.getAbsolutePath());
}
}
private List<String> loadFromMvn(String artifact, Collection<String> excludes,
boolean addSparkContext) throws Exception {
List<String> loadedLibs = new LinkedList<String>();
Collection<String> allExclusions = new LinkedList<String>();
allExclusions.addAll(excludes);
allExclusions.addAll(Arrays.asList(exclusions));
List<ArtifactResult> listOfArtifact;
listOfArtifact = getArtifactsWithDep(artifact, allExclusions);
Iterator<ArtifactResult> it = listOfArtifact.iterator();
while (it.hasNext()) {
Artifact a = it.next().getArtifact();
String gav = a.getGroupId() + ":" + a.getArtifactId() + ":" + a.getVersion();
for (String exclude : allExclusions) {
if (gav.startsWith(exclude)) {
it.remove();
break;
}
}
}
List<URL> newClassPathList = new LinkedList<URL>();
List<File> files = new LinkedList<File>();
for (ArtifactResult artifactResult : listOfArtifact) {
logger.info("Load " + artifactResult.getArtifact().getGroupId() + ":"
+ artifactResult.getArtifact().getArtifactId() + ":"
+ artifactResult.getArtifact().getVersion());
newClassPathList.add(artifactResult.getArtifact().getFile().toURI().toURL());
files.add(artifactResult.getArtifact().getFile());
loadedLibs.add(artifactResult.getArtifact().getGroupId() + ":"
+ artifactResult.getArtifact().getArtifactId() + ":"
+ artifactResult.getArtifact().getVersion());
}
intp.global().new Run();
if (sc.version().startsWith("1.1")) {
updateRuntimeClassPath_1_x(newClassPathList.toArray(new URL[0]));
} else {
updateRuntimeClassPath_2_x(newClassPathList.toArray(new URL[0]));
}
updateCompilerClassPath(newClassPathList.toArray(new URL[0]));
if (addSparkContext) {
for (File f : files) {
sc.addJar(f.getAbsolutePath());
}
}
return loadedLibs;
}
/**
* @param dependency
* @param excludes list of pattern can either be of the form groupId:artifactId
* @return
* @throws Exception
*/
@Override
public List<ArtifactResult> getArtifactsWithDep(String dependency,
Collection<String> excludes) throws Exception {
Artifact artifact = new DefaultArtifact(inferScalaVersion(dependency));
DependencyFilter classpathFilter = DependencyFilterUtils.classpathFilter(JavaScopes.COMPILE);
PatternExclusionsDependencyFilter exclusionFilter =
new PatternExclusionsDependencyFilter(inferScalaVersion(excludes));
CollectRequest collectRequest = new CollectRequest();
collectRequest.setRoot(new Dependency(artifact, JavaScopes.COMPILE));
synchronized (repos) {
for (RemoteRepository repo : repos) {
collectRequest.addRepository(repo);
}
}
DependencyRequest dependencyRequest = new DependencyRequest(collectRequest,
DependencyFilterUtils.andFilter(exclusionFilter, classpathFilter));
return system.resolveDependencies(session, dependencyRequest).getArtifactResults();
}
public static Collection<String> inferScalaVersion(Collection<String> artifact) {
List<String> list = new LinkedList<String>();
for (String a : artifact) {
list.add(inferScalaVersion(a));
}
return list;
}
public static String inferScalaVersion(String artifact) {
int pos = artifact.indexOf(":");
if (pos < 0 || pos + 2 >= artifact.length()) {
// failed to infer
return artifact;
}
if (':' == artifact.charAt(pos + 1)) {
String restOfthem = "";
String versionSep = ":";
String groupId = artifact.substring(0, pos);
int nextPos = artifact.indexOf(":", pos + 2);
if (nextPos < 0) {
if (artifact.charAt(artifact.length() - 1) == '*') {
nextPos = artifact.length() - 1;
versionSep = "";
restOfthem = "*";
} else {
versionSep = "";
nextPos = artifact.length();
}
}
String artifactId = artifact.substring(pos + 2, nextPos);
if (nextPos < artifact.length()) {
if (!restOfthem.equals("*")) {
restOfthem = artifact.substring(nextPos + 1);
}
}
String [] version = scala.util.Properties.versionNumberString().split("[.]");
String scalaVersion = version[0] + "." + version[1];
return groupId + ":" + artifactId + "_" + scalaVersion + versionSep + restOfthem;
} else {
return artifact;
}
}
}

52
spark/src/test/java/org/apache/zeppelin/spark/dep/SparkDependencyResolverTest.java Normal file
View file

@ -0,0 +1,52 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.spark.dep;
import static org.junit.Assert.assertEquals;
import org.apache.zeppelin.spark.dep.SparkDependencyResolver;
import org.junit.Test;
public class SparkDependencyResolverTest {
@Test
public void testInferScalaVersion() {
String [] version = scala.util.Properties.versionNumberString().split("[.]");
String scalaVersion = version[0] + "." + version[1];
assertEquals("groupId:artifactId:version",
SparkDependencyResolver.inferScalaVersion("groupId:artifactId:version"));
assertEquals("groupId:artifactId_" + scalaVersion + ":version",
SparkDependencyResolver.inferScalaVersion("groupId::artifactId:version"));
assertEquals("groupId:artifactId:version::test",
SparkDependencyResolver.inferScalaVersion("groupId:artifactId:version::test"));
assertEquals("*",
SparkDependencyResolver.inferScalaVersion("*"));
assertEquals("groupId:*",
SparkDependencyResolver.inferScalaVersion("groupId:*"));
assertEquals("groupId:artifactId*",
SparkDependencyResolver.inferScalaVersion("groupId:artifactId*"));
assertEquals("groupId:artifactId_" + scalaVersion,
SparkDependencyResolver.inferScalaVersion("groupId::artifactId"));
assertEquals("groupId:artifactId_" + scalaVersion + "*",
SparkDependencyResolver.inferScalaVersion("groupId::artifactId*"));
assertEquals("groupId:artifactId_" + scalaVersion + ":*",
SparkDependencyResolver.inferScalaVersion("groupId::artifactId:*"));
}
}

70
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/AbstractDependencyResolver.java Normal file
View file

@ -0,0 +1,70 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import java.util.Collection;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import org.sonatype.aether.RepositorySystem;
import org.sonatype.aether.RepositorySystemSession;
import org.sonatype.aether.repository.RemoteRepository;
import org.sonatype.aether.resolution.ArtifactResult;
/**
* Abstract dependency resolver.
* Add new dependencies from mvn repo (at runtime) Zeppelin.
*/
public abstract class AbstractDependencyResolver {
protected RepositorySystem system = Booter.newRepositorySystem();
protected List<RemoteRepository> repos = new LinkedList<RemoteRepository>();
protected RepositorySystemSession session;
public AbstractDependencyResolver(String localRepoPath) {
session = Booter.newRepositorySystemSession(system, localRepoPath);
repos.add(Booter.newCentralRepository()); // add maven central
repos.add(Booter.newLocalRepository());
}
public void addRepo(String id, String url, boolean snapshot) {
synchronized (repos) {
delRepo(id);
RemoteRepository rr = new RemoteRepository(id, "default", url);
rr.setPolicy(snapshot, null);
repos.add(rr);
}
}
public RemoteRepository delRepo(String id) {
synchronized (repos) {
Iterator<RemoteRepository> it = repos.iterator();
if (it.hasNext()) {
RemoteRepository repo = it.next();
if (repo.getId().equals(id)) {
it.remove();
return repo;
}
}
}
return null;
}
public abstract List<ArtifactResult> getArtifactsWithDep(String dependency,
Collection<String> excludes) throws Exception;
}

72
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/Booter.java Normal file
View file

@ -0,0 +1,72 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import java.io.File;
import org.apache.maven.repository.internal.MavenRepositorySystemSession;
import org.sonatype.aether.RepositorySystem;
import org.sonatype.aether.RepositorySystemSession;
import org.sonatype.aether.repository.LocalRepository;
import org.sonatype.aether.repository.RemoteRepository;
/**
* Manage mvn repository.
*/
public class Booter {
public static RepositorySystem newRepositorySystem() {
return RepositorySystemFactory.newRepositorySystem();
}
public static RepositorySystemSession newRepositorySystemSession(
RepositorySystem system, String localRepoPath) {
MavenRepositorySystemSession session = new MavenRepositorySystemSession();
// find homedir
String home = System.getenv("ZEPPELIN_HOME");
if (home == null) {
home = System.getProperty("zeppelin.home");
}
if (home == null) {
home = "..";
}
String path = home + "/" + localRepoPath;
LocalRepository localRepo =
new LocalRepository(new File(path).getAbsolutePath());
session.setLocalRepositoryManager(system.newLocalRepositoryManager(localRepo));
// session.setTransferListener(new ConsoleTransferListener());
// session.setRepositoryListener(new ConsoleRepositoryListener());
// uncomment to generate dirty trees
// session.setDependencyGraphTransformer( null );
return session;
}
public static RemoteRepository newCentralRepository() {
return new RemoteRepository("central", "default", "http://repo1.maven.org/maven2/");
}
public static RemoteRepository newLocalRepository() {
return new RemoteRepository("local",
"default", "file://" + System.getProperty("user.home") + "/.m2/repository");
}
}

90
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/Dependency.java Normal file
View file

@ -0,0 +1,90 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import java.util.LinkedList;
import java.util.List;
/**
*
*/
public class Dependency {
private String groupArtifactVersion;
private boolean local = false;
private List<String> exclusions;
public Dependency(String groupArtifactVersion) {
this.groupArtifactVersion = groupArtifactVersion;
exclusions = new LinkedList<String>();
}
@Override
public boolean equals(Object o) {
if (!(o instanceof Dependency)) {
return false;
} else {
return ((Dependency) o).groupArtifactVersion.equals(groupArtifactVersion);
}
}
/**
* Don't add artifact into SparkContext (sc.addJar())
* @return
*/
public Dependency local() {
local = true;
return this;
}
public Dependency excludeAll() {
exclude("*");
return this;
}
/**
*
* @param exclusions comma or newline separated list of "groupId:ArtifactId"
* @return
*/
public Dependency exclude(String exclusions) {
for (String item : exclusions.split(",|\n")) {
this.exclusions.add(item);
}
return this;
}
public String getGroupArtifactVersion() {
return groupArtifactVersion;
}
public boolean isDist() {
return !local;
}
public List<String> getExclusions() {
return exclusions;
}
public boolean isLocalFsArtifact() {
int numSplits = groupArtifactVersion.split(":").length;
return !(numSplits >= 3 && numSplits <= 6);
}
}

148
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/DependencyContext.java Normal file
View file

@ -0,0 +1,148 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import java.io.File;
import java.net.MalformedURLException;
import java.util.LinkedList;
import java.util.List;
import org.sonatype.aether.RepositorySystem;
import org.sonatype.aether.RepositorySystemSession;
import org.sonatype.aether.artifact.Artifact;
import org.sonatype.aether.collection.CollectRequest;
import org.sonatype.aether.graph.DependencyFilter;
import org.sonatype.aether.repository.RemoteRepository;
import org.sonatype.aether.resolution.ArtifactResolutionException;
import org.sonatype.aether.resolution.ArtifactResult;
import org.sonatype.aether.resolution.DependencyRequest;
import org.sonatype.aether.resolution.DependencyResolutionException;
import org.sonatype.aether.util.artifact.DefaultArtifact;
import org.sonatype.aether.util.artifact.JavaScopes;
import org.sonatype.aether.util.filter.DependencyFilterUtils;
import org.sonatype.aether.util.filter.PatternExclusionsDependencyFilter;
/**
*
*/
public class DependencyContext {
List<Dependency> dependencies = new LinkedList<Dependency>();
List<Repository> repositories = new LinkedList<Repository>();
List<File> files = new LinkedList<File>();
List<File> filesDist = new LinkedList<File>();
private RepositorySystem system = Booter.newRepositorySystem();
private RepositorySystemSession session;
private RemoteRepository mavenCentral = Booter.newCentralRepository();
private RemoteRepository mavenLocal = Booter.newLocalRepository();
public DependencyContext(String localRepoPath) {
session = Booter.newRepositorySystemSession(system, localRepoPath);
}
public Dependency load(String lib) {
Dependency dep = new Dependency(lib);
if (dependencies.contains(dep)) {
dependencies.remove(dep);
}
dependencies.add(dep);
return dep;
}
public Repository addRepo(String name) {
Repository rep = new Repository(name);
repositories.add(rep);
return rep;
}
public void reset() {
dependencies = new LinkedList<Dependency>();
repositories = new LinkedList<Repository>();
files = new LinkedList<File>();
filesDist = new LinkedList<File>();
}
/**
* fetch all artifacts
* @return
* @throws MalformedURLException
* @throws ArtifactResolutionException
* @throws DependencyResolutionException
*/
public List<File> fetch() throws MalformedURLException,
DependencyResolutionException, ArtifactResolutionException {
for (Dependency dep : dependencies) {
if (!dep.isLocalFsArtifact()) {
List<ArtifactResult> artifacts = fetchArtifactWithDep(dep);
for (ArtifactResult artifact : artifacts) {
if (dep.isDist()) {
filesDist.add(artifact.getArtifact().getFile());
}
files.add(artifact.getArtifact().getFile());
}
} else {
if (dep.isDist()) {
filesDist.add(new File(dep.getGroupArtifactVersion()));
}
files.add(new File(dep.getGroupArtifactVersion()));
}
}
return files;
}
private List<ArtifactResult> fetchArtifactWithDep(Dependency dep)
throws DependencyResolutionException, ArtifactResolutionException {
Artifact artifact = new DefaultArtifact(dep.getGroupArtifactVersion());
DependencyFilter classpathFlter = DependencyFilterUtils
.classpathFilter(JavaScopes.COMPILE);
PatternExclusionsDependencyFilter exclusionFilter = new PatternExclusionsDependencyFilter(
dep.getExclusions());
CollectRequest collectRequest = new CollectRequest();
collectRequest.setRoot(new org.sonatype.aether.graph.Dependency(artifact,
JavaScopes.COMPILE));
collectRequest.addRepository(mavenCentral);
collectRequest.addRepository(mavenLocal);
for (Repository repo : repositories) {
RemoteRepository rr = new RemoteRepository(repo.getName(), "default", repo.getUrl());
rr.setPolicy(repo.isSnapshot(), null);
collectRequest.addRepository(rr);
}
DependencyRequest dependencyRequest = new DependencyRequest(collectRequest,
DependencyFilterUtils.andFilter(exclusionFilter, classpathFlter));
return system.resolveDependencies(session, dependencyRequest).getArtifactResults();
}
public List<File> getFiles() {
return files;
}
public List<File> getFilesDist() {
return filesDist;
}
}

163
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/DependencyResolver.java Normal file
View file

@ -0,0 +1,163 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import java.io.File;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.sonatype.aether.artifact.Artifact;
import org.sonatype.aether.collection.CollectRequest;
import org.sonatype.aether.graph.Dependency;
import org.sonatype.aether.graph.DependencyFilter;
import org.sonatype.aether.repository.RemoteRepository;
import org.sonatype.aether.resolution.ArtifactResult;
import org.sonatype.aether.resolution.DependencyRequest;
import org.sonatype.aether.util.artifact.DefaultArtifact;
import org.sonatype.aether.util.artifact.JavaScopes;
import org.sonatype.aether.util.filter.DependencyFilterUtils;
import org.sonatype.aether.util.filter.PatternExclusionsDependencyFilter;
/**
* Deps resolver.
* Add new dependencies from mvn repo (at runtime) to Zeppelin.
*/
public class DependencyResolver extends AbstractDependencyResolver {
Logger logger = LoggerFactory.getLogger(DependencyResolver.class);
private final String[] exclusions = new String[] {"org.apache.zeppelin:zeppelin-zengine",
"org.apache.zeppelin:zeppelin-interpreter",
"org.apache.zeppelin:zeppelin-server"};
public DependencyResolver(String localRepoPath) {
super(localRepoPath);
}
public List<File> load(String artifact) throws Exception {
return load(artifact, new LinkedList<String>());
}
public List<File> load(String artifact, String destPath) throws Exception {
return load(artifact, new LinkedList<String>(), destPath);
}
public synchronized List<File> load(String artifact, Collection<String> excludes)
throws Exception {
if (StringUtils.isBlank(artifact)) {
// Should throw here
throw new RuntimeException("Invalid artifact to load");
}
// <groupId>:<artifactId>[:<extension>[:<classifier>]]:<version>
int numSplits = artifact.split(":").length;
if (numSplits >= 3 && numSplits <= 6) {
return loadFromMvn(artifact, excludes);
} else {
LinkedList<File> libs = new LinkedList<File>();
libs.add(new File(artifact));
return libs;
}
}
public List<File> load(String artifact, Collection<String> excludes, String destPath)
throws Exception {
List<File> libs = load(artifact, excludes);
// find home dir
String home = System.getenv("ZEPPELIN_HOME");
if (home == null) {
home = System.getProperty("zeppelin.home");
}
if (home == null) {
home = "..";
}
for (File srcFile: libs) {
File destFile = new File(home + "/" + destPath, srcFile.getName());
if (!destFile.exists() || !FileUtils.contentEquals(srcFile, destFile)) {
FileUtils.copyFile(srcFile, destFile);
logger.info("copy {} to {}", srcFile.getAbsolutePath(), destPath);
}
}
return libs;
}
private List<File> loadFromMvn(String artifact, Collection<String> excludes) throws Exception {
Collection<String> allExclusions = new LinkedList<String>();
allExclusions.addAll(excludes);
allExclusions.addAll(Arrays.asList(exclusions));
List<ArtifactResult> listOfArtifact;
listOfArtifact = getArtifactsWithDep(artifact, allExclusions);
Iterator<ArtifactResult> it = listOfArtifact.iterator();
while (it.hasNext()) {
Artifact a = it.next().getArtifact();
String gav = a.getGroupId() + ":" + a.getArtifactId() + ":" + a.getVersion();
for (String exclude : allExclusions) {
if (gav.startsWith(exclude)) {
it.remove();
break;
}
}
}
List<File> files = new LinkedList<File>();
for (ArtifactResult artifactResult : listOfArtifact) {
files.add(artifactResult.getArtifact().getFile());
logger.info("load {}", artifactResult.getArtifact().getFile().getAbsolutePath());
}
return files;
}
/**
* @param dependency
* @param excludes list of pattern can either be of the form groupId:artifactId
* @return
* @throws Exception
*/
@Override
public List<ArtifactResult> getArtifactsWithDep(String dependency,
Collection<String> excludes) throws Exception {
Artifact artifact = new DefaultArtifact(dependency);
DependencyFilter classpathFilter = DependencyFilterUtils.classpathFilter(JavaScopes.COMPILE);
PatternExclusionsDependencyFilter exclusionFilter =
new PatternExclusionsDependencyFilter(excludes);
CollectRequest collectRequest = new CollectRequest();
collectRequest.setRoot(new Dependency(artifact, JavaScopes.COMPILE));
synchronized (repos) {
for (RemoteRepository repo : repos) {
collectRequest.addRepository(repo);
}
}
DependencyRequest dependencyRequest = new DependencyRequest(collectRequest,
DependencyFilterUtils.andFilter(exclusionFilter, classpathFilter));
return system.resolveDependencies(session, dependencyRequest).getArtifactResults();
}
}

80
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/Repository.java Normal file
View file

@ -0,0 +1,80 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import org.sonatype.aether.repository.Authentication;
/**
*
*
*/
public class Repository {
private boolean snapshot = false;
private String name;
private String url;
private String username = null;
private String password = null;
public Repository(String name){
this.name = name;
}
public Repository url(String url) {
this.url = url;
return this;
}
public Repository snapshot() {
snapshot = true;
return this;
}
public boolean isSnapshot() {
return snapshot;
}
public String getName() {
return name;
}
public String getUrl() {
return url;
}
public Repository username(String username) {
this.username = username;
return this;
}
public Repository password(String password) {
this.password = password;
return this;
}
public Repository credentials(String username, String password) {
this.username = username;
this.password = password;
return this;
}
public Authentication getAuthentication() {
Authentication auth = null;
if (this.username != null && this.password != null) {
auth = new Authentication(this.username, this.password);
}
return auth;
}
}

118
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/RepositoryListener.java Normal file
View file

@ -0,0 +1,118 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.sonatype.aether.AbstractRepositoryListener;
import org.sonatype.aether.RepositoryEvent;
/**
* Simple listener that print log.
*/
public class RepositoryListener extends AbstractRepositoryListener {
Logger logger = LoggerFactory.getLogger(RepositoryListener.class);
public RepositoryListener() {}
@Override
public void artifactDeployed(RepositoryEvent event) {
logger.info("Deployed " + event.getArtifact() + " to " + event.getRepository());
}
@Override
public void artifactDeploying(RepositoryEvent event) {
logger.info("Deploying " + event.getArtifact() + " to " + event.getRepository());
}
@Override
public void artifactDescriptorInvalid(RepositoryEvent event) {
logger.info("Invalid artifact descriptor for " + event.getArtifact() + ": "
+ event.getException().getMessage());
}
@Override
public void artifactDescriptorMissing(RepositoryEvent event) {
logger.info("Missing artifact descriptor for " + event.getArtifact());
}
@Override
public void artifactInstalled(RepositoryEvent event) {
logger.info("Installed " + event.getArtifact() + " to " + event.getFile());
}
@Override
public void artifactInstalling(RepositoryEvent event) {
logger.info("Installing " + event.getArtifact() + " to " + event.getFile());
}
@Override
public void artifactResolved(RepositoryEvent event) {
logger.info("Resolved artifact " + event.getArtifact() + " from " + event.getRepository());
}
@Override
public void artifactDownloading(RepositoryEvent event) {
logger.info("Downloading artifact " + event.getArtifact() + " from " + event.getRepository());
}
@Override
public void artifactDownloaded(RepositoryEvent event) {
logger.info("Downloaded artifact " + event.getArtifact() + " from " + event.getRepository());
}
@Override
public void artifactResolving(RepositoryEvent event) {
logger.info("Resolving artifact " + event.getArtifact());
}
@Override
public void metadataDeployed(RepositoryEvent event) {
logger.info("Deployed " + event.getMetadata() + " to " + event.getRepository());
}
@Override
public void metadataDeploying(RepositoryEvent event) {
logger.info("Deploying " + event.getMetadata() + " to " + event.getRepository());
}
@Override
public void metadataInstalled(RepositoryEvent event) {
logger.info("Installed " + event.getMetadata() + " to " + event.getFile());
}
@Override
public void metadataInstalling(RepositoryEvent event) {
logger.info("Installing " + event.getMetadata() + " to " + event.getFile());
}
@Override
public void metadataInvalid(RepositoryEvent event) {
logger.info("Invalid metadata " + event.getMetadata());
}
@Override
public void metadataResolved(RepositoryEvent event) {
logger.info("Resolved metadata " + event.getMetadata() + " from " + event.getRepository());
}
@Override
public void metadataResolving(RepositoryEvent event) {
logger.info("Resolving metadata " + event.getMetadata() + " from " + event.getRepository());
}
}

66
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/RepositorySystemFactory.java Normal file
View file

@ -0,0 +1,66 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import org.apache.maven.repository.internal.DefaultServiceLocator;
import org.apache.maven.wagon.Wagon;
import org.apache.maven.wagon.providers.http.HttpWagon;
import org.apache.maven.wagon.providers.http.LightweightHttpWagon;
import org.sonatype.aether.RepositorySystem;
import org.sonatype.aether.connector.file.FileRepositoryConnectorFactory;
import org.sonatype.aether.connector.wagon.WagonProvider;
import org.sonatype.aether.connector.wagon.WagonRepositoryConnectorFactory;
import org.sonatype.aether.spi.connector.RepositoryConnectorFactory;
/**
* Get maven repository instance.
*/
public class RepositorySystemFactory {
public static RepositorySystem newRepositorySystem() {
DefaultServiceLocator locator = new DefaultServiceLocator();
locator.addService(RepositoryConnectorFactory.class, FileRepositoryConnectorFactory.class);
locator.addService(RepositoryConnectorFactory.class, WagonRepositoryConnectorFactory.class);
locator.setServices(WagonProvider.class, new ManualWagonProvider());
return locator.getService(RepositorySystem.class);
}
/**
* ManualWagonProvider
*/
public static class ManualWagonProvider implements WagonProvider {
@Override
public Wagon lookup(String roleHint) throws Exception {
if ("http".equals(roleHint)) {
return new LightweightHttpWagon();
}
if ("https".equals(roleHint)) {
return new HttpWagon();
}
return null;
}
@Override
public void release(Wagon arg0) {
}
}
}

145
zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/TransferListener.java Normal file
View file

@ -0,0 +1,145 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import java.io.PrintStream;
import java.text.DecimalFormat;
import java.text.DecimalFormatSymbols;
import java.util.Locale;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.sonatype.aether.transfer.AbstractTransferListener;
import org.sonatype.aether.transfer.TransferEvent;
import org.sonatype.aether.transfer.TransferResource;
/**
* Simple listener that show deps downloading progress.
*/
public class TransferListener extends AbstractTransferListener {
Logger logger = LoggerFactory.getLogger(TransferListener.class);
private PrintStream out;
private Map<TransferResource, Long> downloads = new ConcurrentHashMap<TransferResource, Long>();
private int lastLength;
public TransferListener() {}
@Override
public void transferInitiated(TransferEvent event) {
String message =
event.getRequestType() == TransferEvent.RequestType.PUT ? "Uploading" : "Downloading";
logger.info(message + ": " + event.getResource().getRepositoryUrl()
+ event.getResource().getResourceName());
}
@Override
public void transferProgressed(TransferEvent event) {
TransferResource resource = event.getResource();
downloads.put(resource, Long.valueOf(event.getTransferredBytes()));
StringBuilder buffer = new StringBuilder(64);
for (Map.Entry<TransferResource, Long> entry : downloads.entrySet()) {
long total = entry.getKey().getContentLength();
long complete = entry.getValue().longValue();
buffer.append(getStatus(complete, total)).append(" ");
}
int pad = lastLength - buffer.length();
lastLength = buffer.length();
pad(buffer, pad);
buffer.append('\r');
logger.info(buffer.toString());
}
private String getStatus(long complete, long total) {
if (total >= 1024) {
return toKB(complete) + "/" + toKB(total) + " KB ";
} else if (total >= 0) {
return complete + "/" + total + " B ";
} else if (complete >= 1024) {
return toKB(complete) + " KB ";
} else {
return complete + " B ";
}
}
private void pad(StringBuilder buffer, int spaces) {
String block = " ";
while (spaces > 0) {
int n = Math.min(spaces, block.length());
buffer.append(block, 0, n);
spaces -= n;
}
}
@Override
public void transferSucceeded(TransferEvent event) {
transferCompleted(event);
TransferResource resource = event.getResource();
long contentLength = event.getTransferredBytes();
if (contentLength >= 0) {
String type =
(event.getRequestType() == TransferEvent.RequestType.PUT ? "Uploaded" : "Downloaded");
String len = contentLength >= 1024 ? toKB(contentLength) + " KB" : contentLength + " B";
String throughput = "";
long duration = System.currentTimeMillis() - resource.getTransferStartTime();
if (duration > 0) {
DecimalFormat format = new DecimalFormat("0.0", new DecimalFormatSymbols(Locale.ENGLISH));
double kbPerSec = (contentLength / 1024.0) / (duration / 1000.0);
throughput = " at " + format.format(kbPerSec) + " KB/sec";
}
logger.info(type + ": " + resource.getRepositoryUrl() + resource.getResourceName() + " ("
+ len + throughput + ")");
}
}
@Override
public void transferFailed(TransferEvent event) {
transferCompleted(event);
event.getException().printStackTrace(out);
}
private void transferCompleted(TransferEvent event) {
downloads.remove(event.getResource());
StringBuilder buffer = new StringBuilder(64);
pad(buffer, lastLength);
buffer.append('\r');
logger.info(buffer.toString());
}
@Override
public void transferCorrupted(TransferEvent event) {
event.getException().printStackTrace(out);
}
protected long toKB(long bytes) {
return (bytes + 1023) / 1024;
}
}

62
zeppelin-interpreter/src/test/java/org/apache/zeppelin/dep/DependencyResolverTest.java Normal file
View file

@ -0,0 +1,62 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.dep;
import static org.junit.Assert.assertTrue;
import java.io.File;
import org.apache.commons.io.FileUtils;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
public class DependencyResolverTest {
private static DependencyResolver resolver;
private static String testPath;
private static String testCopyPath;
private static String home;
@BeforeClass
public static void setUp() throws Exception {
testPath = "test-repo";
testCopyPath = "test-copy-repo";
resolver = new DependencyResolver(testPath);
home = System.getenv("ZEPPELIN_HOME");
if (home == null) {
home = System.getProperty("zeppelin.home");
}
if (home == null) {
home = "..";
}
}
@AfterClass
public static void tearDown() throws Exception {
FileUtils.deleteDirectory(new File(home + "/" + testPath));
FileUtils.deleteDirectory(new File(home + "/" + testCopyPath));
}
@Test
public void testLoad() throws Exception {
resolver.load("org.apache.commons:commons-lang3:3.4", testCopyPath);
assertTrue(new File(home + "/" + testPath + "/org/apache/commons/commons-lang3/3.4/").exists());
assertTrue(new File(home + "/" + testCopyPath + "/commons-lang3-3.4.jar").exists());
}
}

74
zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java Normal file
View file

@ -0,0 +1,74 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.rest;
import org.apache.zeppelin.conf.ZeppelinConfiguration;
import org.apache.zeppelin.server.JsonResponse;
import org.apache.zeppelin.ticket.TicketContainer;
import org.apache.zeppelin.utils.SecurityUtils;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Response;
import java.util.HashMap;
import java.util.Map;
/**
* Zeppelin security rest api endpoint.
*
*/
@Path("/security")
@Produces("application/json")
public class SecurityRestApi {
/**
* Required by Swagger.
*/
public SecurityRestApi() {
super();
}
/**
* Get ticket
* Returns username & ticket
* for anonymous access, username is always anonymous.
* After getting this ticket, access through websockets become safe
*
* @return 200 response
*/
@GET
@Path("ticket")
public Response ticket() {
ZeppelinConfiguration conf = ZeppelinConfiguration.create();
String principal = SecurityUtils.getPrincipal();
JsonResponse response;
// ticket set to anonymous for anonymous user. Simplify testing.
String ticket;
if ("anonymous".equals(principal))
ticket = "anonymous";
else
ticket = TicketContainer.instance.getTicket(principal);
Map<String, String> data = new HashMap<>();
data.put("principal", principal);
data.put("ticket", ticket);
response = new JsonResponse(Response.Status.OK, "", data);
return response.build();
}
}

82
zeppelin-server/src/main/java/org/apache/zeppelin/ticket/TicketContainer.java Normal file
View file

@ -0,0 +1,82 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.ticket;
import java.util.Calendar;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
/**
* Very simple ticket container
* No cleanup is done, since the same user accross different devices share the same ticket
* The Map size is at most the number of different user names having access to a Zeppelin instance
*/
public class TicketContainer {
private static class Entry {
public final String ticket;
// lastAccessTime still unused
public final long lastAccessTime;
Entry(String ticket) {
this.ticket = ticket;
this.lastAccessTime = Calendar.getInstance().getTimeInMillis();
}
}
private Map<String, Entry> sessions = new ConcurrentHashMap<>();
public static final TicketContainer instance = new TicketContainer();
/**
* For test use
* @param principal
* @param ticket
* @return true if ticket assigned to principal.
*/
public boolean isValid(String principal, String ticket) {
if ("anonymous".equals(principal) && "anonymous".equals(ticket))
return true;
Entry entry = sessions.get(principal);
return entry != null && entry.ticket.equals(ticket);
}
/**
* get or create ticket for Websocket authentication assigned to authenticated shiro user
* For unathenticated user (anonymous), always return ticket value "anonymous"
* @param principal
* @return
*/
public synchronized String getTicket(String principal) {
Entry entry = sessions.get(principal);
String ticket;
if (entry == null) {
if (principal.equals("anonymous"))
ticket = "anonymous";
else
ticket = UUID.randomUUID().toString();
} else {
ticket = entry.ticket;
}
entry = new Entry(ticket);
sessions.put(principal, entry);
return ticket;
}
}

31
zeppelin-server/src/main/resources/shiro.ini Normal file
View file

@ -0,0 +1,31 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
[users]
# List of users with their password allowed to access Zeppelin.
# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections
admin = password
[urls]
# anon means the access is anonymous.
# authcBasic means Basic Auth Security
# To enfore security, comment the line below and uncomment the next one
/** = anon
#/** = authcBasic

58
zeppelin-server/src/test/java/org/apache/zeppelin/rest/SecurityRestApiTest.java Normal file
View file

@ -0,0 +1,58 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.rest;
import com.google.gson.Gson;
import com.google.gson.reflect.TypeToken;
import org.apache.commons.httpclient.methods.GetMethod;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import java.io.IOException;
import java.util.Map;
import static org.junit.Assert.*;
public class SecurityRestApiTest extends AbstractTestRestApi {
Gson gson = new Gson();
@BeforeClass
public static void init() throws Exception {
AbstractTestRestApi.startUp();
}
@AfterClass
public static void destroy() throws Exception {
AbstractTestRestApi.shutDown();
}
@Test
public void testTicket() throws IOException {
GetMethod get = httpGet("/security/ticket");
get.addRequestHeader("Origin", "http://localhost");
Map<String, Object> resp = gson.fromJson(get.getResponseBodyAsString(),
new TypeToken<Map<String, Object>>(){}.getType());
Map<String, String> body = (Map<String, String>) resp.get("body");
assertEquals("anonymous", body.get("principal"));
assertEquals("anonymous", body.get("ticket"));
get.releaseConnection();
}
}

62
zeppelin-server/src/test/java/org/apache/zeppelin/ticket/TicketContainerTest.java Normal file
View file

@ -0,0 +1,62 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.zeppelin.ticket;
import org.junit.Before;
import org.junit.Test;
import java.net.UnknownHostException;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
public class TicketContainerTest {
private TicketContainer container;
@Before
public void setUp() throws Exception {
container = TicketContainer.instance;
}
@Test
public void isValidAnonymous() throws UnknownHostException {
boolean ok = container.isValid("anonymous", "anonymous");
assertTrue(ok);
}
@Test
public void isValidExistingPrincipal() throws UnknownHostException {
String ticket = container.getTicket("someuser1");
boolean ok = container.isValid("someuser1", ticket);
assertTrue(ok);
}
@Test
public void isValidNonExistingPrincipal() throws UnknownHostException {
boolean ok = container.isValid("unknownuser", "someticket");
assertFalse(ok);
}
@Test
public void isValidunkownTicket() throws UnknownHostException {
String ticket = container.getTicket("someuser2");
boolean ok = container.isValid("someuser2", ticket+"makeitinvalid");
assertFalse(ok);
}
}