zeppelin/docs/manual/dependencymanagement.md at eac6b438c261bdb94de96a2c42d8428776f4f104

mirror of https://github.com/apache/zeppelin synced 2026-05-24 09:38:26 +00:00

DuyHai DOAN 79fd3a0242 [ZEPPELIN-1376] Add proxy credentials for dependency repo for corporate firewall use-cases

### What is this PR for?
When using Zeppelin behind corporate firewall, sometimes the dependencies download just fails silently. This PR has 2 objectives:

* add proxy credentials information for dependencies repo
* raise clear error message in case of dependencies download failure

There are 3 commits.

The first one add extra inputs in the form for adding new repository

![add_repo](https://cloud.githubusercontent.com/assets/1532977/18017489/0b486fda-6bd2-11e6-90c7-ceda18c53575.png)

The second commit fixes some issues and display a clear and explicit error message when download of dependencies fail.

Before that, when the download fails, we can see the below behaviour

![irrelevant_double_error_message](https://cloud.githubusercontent.com/assets/1532977/18017541/3cf0de1e-6bd2-11e6-8285-af03f222e8d2.gif)

* the error message is displayed twice because the call twice the method `checkDownloadingDependencies();`. One in the success callback of:

```javascript
 $scope.updateInterpreterSetting = function(form, settingId) {
              ...
            $http.put(baseUrlSrv.getRestApiBase() + '/interpreter/setting/' + settingId, request)
              .success(function(data, status, headers, config) {
                $scope.interpreterSettings[index] = data.body;
                removeTMPSettings(index);
                thisConfirm.close();
                checkDownloadingDependencies();
                $route.reload();
              })
              .error(function(data, status, headers, config) {
             ...
    };
```

Another call is inside success callback of `getInterpreterSettings()`

```javascript
var getInterpreterSettings = function() {
      $http.get(baseUrlSrv.getRestApiBase() + '/interpreter/setting')
      .success(function(data, status, headers, config) {
        $scope.interpreterSettings = data.body;
        checkDownloadingDependencies();
      }).error(function(data, status, headers, config) {
      ....
```

The problem is that `$route.reload();` in the success callback of `updateInterpreterSetting()` will trigger `init()` then `getInterpreterSettings()` so `checkDownloadingDependencies()` is called twice.

I remove the call to `checkDownloadingDependencies()` from success callback of `updateInterpreterSetting()`

The second modification is on class `DependencyResolver`. In the screen capture above, we get a **cryptic** NullPointerException coming from `DefaultRepositorySystem`. I now catch this NPE to wrap it into a more sensible and clearer exception:

```java

  public List<ArtifactResult> getArtifactsWithDep(String dependency,
    Collection<String> excludes) throws RepositoryException {
    Artifact artifact = new DefaultArtifact(dependency);
    DependencyFilter classpathFilter = DependencyFilterUtils.classpathFilter(JavaScopes.COMPILE);
    PatternExclusionsDependencyFilter exclusionFilter =
            new PatternExclusionsDependencyFilter(excludes);

    CollectRequest collectRequest = new CollectRequest();
    collectRequest.setRoot(new Dependency(artifact, JavaScopes.COMPILE));

    synchronized (repos) {
      for (RemoteRepository repo : repos) {
        collectRequest.addRepository(repo);
      }
    }
    DependencyRequest dependencyRequest = new DependencyRequest(collectRequest,
            DependencyFilterUtils.andFilter(exclusionFilter, classpathFilter));

 //Catch NPE thrown by aether and give a proper error message
    try {
      return system.resolveDependencies(session, dependencyRequest).getArtifactResults();
    } catch (NullPointerException ex) {
      throw new RepositoryException(String.format("Cannot fetch dependencies for %s", dependency));
    }
  }
```

The result is much more cleaner

![dependencies_download_error_popup](https://cloud.githubusercontent.com/assets/1532977/18033855/1be5fe9a-6d2e-11e6-91f9-2f5ea66cab26.gif)

The last commit is just doc update

![updated_docs](https://cloud.githubusercontent.com/assets/1532977/18017797/97302f14-6bd3-11e6-97cc-77bd52f25cde.png)

### What type of PR is it?
[Improvement]

### Todos
* [ ] - Code Review
* [ ] - Simple test with no Internet connection
* [ ] - Test within a corporate firewall env with a third-party dependency, requiring download

### What is the Jira issue?
**[ZEPPELIN-1376]**

### How should this be tested?

##### Simple test
* `git fetch origin pull/1369/head:WebProxy`
* `git checkout WebProxy`
* `mvn clean package -DskipTests`
* `bin/zeppelin-daemon.sh restart`
* disconnect from the Internet (pull out the cable, shutdown wifi ...)
* add a random dependency to the Spark interpreter (take `info.archinnov:achilles-core:4.2.2` for example)
* validate the change, you should see an error popup on the top-right corner saying that Zeppelin cannot download the dependency

##### Corporate firewall test
* follow the steps above for simple test
* create a new repository (see how to **[here]**) and set the proxy information
* retry the steps above to ensure that the download is successful

### Screenshots (if appropriate)
See above

### Questions:
* Does the licenses files need update? --> **NO**
* Is there breaking changes for older versions? --> **NO**
* Does this needs documentation?  --> **YES, DONE**

[ZEPPELIN-1376]: https://issues.apache.org/jira/browse/ZEPPELIN-1376
[here]: http://localhost:4000/manual/dependencymanagement.html

Author: DuyHai DOAN <doanduyhai@gmail.com>
Author: doanduyhai <doanduyhai@apache.org>

Closes #1369 from doanduyhai/ZEPPELIN-1376 and squashes the following commits:

b8d44e7 [doanduyhai] [ZEPPELIN-1376] Improve error popup display
177fbd3 [DuyHai DOAN] [ZEPPELIN-1376] Fixes JS bug to display error popup for other interpreters
9f76ef4 [DuyHai DOAN] [ZEPPELIN-1376] Do not repeat the same error popup multiple times
b264193 [DuyHai DOAN] [ZEPPELIN-1376] Add unit test and fix impl for DependencyResolver to catch NPE
1913a0a [DuyHai DOAN] [ZEPPELIN-1376] Update documentation
f01be9b [DuyHai DOAN] [ZEPPELIN-1376] Raise clear error message in case of dependencies download failure
6f2b6f8 [DuyHai DOAN] [ZEPPELIN-1376] Add proxy credentials information for dependencies repo

2016-10-10 09:27:30 +02:00

3.3 KiB

Raw Blame History

layout	title	description	group
page	Dependency Management for Apache Spark Interpreter	Include external libraries to Apache Spark Interpreter by setting dependencies in interpreter menu.	manual

{% include JB/setup %}

Dependency Management for Interpreter

You can include external libraries to interpreter by setting dependencies in interpreter menu.

When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs in this menu.

Load libraries recursively from Maven repository
Load libraries from local filesystem
Add additional maven repository
Automatically add libraries to SparkCluster

Load Dependencies to Interpreter

Click 'Interpreter' menu in navigation bar.
Click 'edit' button of the interpreter which you want to load dependencies to.
Fill artifact and exclude field to your needs. You can enter not only groupId:artifactId:version but also local file in artifact field.
Press 'Save' to restart the interpreter with loaded libraries.

Add repository for dependency resolving

Press icon in 'Interpreter' menu on the top right side. It will show you available repository lists.
If you need to resolve dependencies from other than central maven repository or local ~/.m2 repository, hit icon next to repository lists.
Fill out the form and click 'Add' button, then you will be able to see that new repository is added.
Optionally, if you are behind a corporate firewall, you can specify also all proxy settings so that Zeppelin can download the dependencies using the given credentials

3.3 KiB Raw Blame History

Dependency Management for Interpreter

3.3 KiB

Raw Blame History