Commit graph

4637 commits

Author SHA1 Message Date
Kairo de Araujo
ffe2c11fcf WIP: TUF Python Client Example/Tutorial
It is a simple example of TUF ngclient implementation.

This example contains a README.rst that is a tutorial/how-to-use
this simple client using static test data from TUF repository.

The code aims to be straightforward implementation, using basic
concepts from Python and Command Line Interface.

This is part of theupdateframework#1518

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:57:53 +01:00
lukpueh
a93f618839
Merge pull request #1701 from MVrachev/lint-examples
Start linting the examples folder
2021-12-03 10:31:38 +01:00
Jussi Kukkonen
dd5deeecf0
Merge pull request #1672 from ivanayov/ivanayov/rolenames
Create constants for top-level rolenames
2021-12-02 14:37:22 +02:00
Martin Vrachev
d697f73da2 Linting in tox.ini: use variable for target dirs
Instead of providing a target directory for linting by each of the
tools use one variable which will be the source of truth about which
directories do we lint.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 14:12:05 +02:00
Martin Vrachev
1ae6a22b8e Start linting the examples folder
The examples folder currently contains a repository example and it's
good if we start linting its content and as a result add type
annotations.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 13:40:14 +02:00
Ivana Atanasova
00589f08e3 Apply top-level rolenames constants in tests
This applies the use of constants of top-level rolenames in the
tests instead of the previously hardcoded strings.
Fixes #1648

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-12-02 12:25:48 +02:00
Ivana Atanasova
d7c653470a Create constants for top-level rolenames
This is a change in the metadata API to remove hardcoded rolenames
and use constants instead.
Fixes #1648

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-12-02 12:16:25 +02:00
Jussi Kukkonen
d991362ff0
Merge pull request #1699 from MVrachev/move-configs
Move linters configurations in pyproject.toml
2021-12-02 12:11:13 +02:00
Martin Vrachev
5c8a86665f Fix small pylint error
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-01 16:28:15 +02:00
Martin Vrachev
ed8a06bcb3 Move part of isort options in pyproject.toml
We are using 4 linters: black, isort, pylint and mypy.
It's good if we use one file as a source for truth for all linter
configurations.

I tried multiple ways to use the src_path option,
so we can just call isort without pointing out the target folders, but I was not
successful.
I tried running isort with "isort --settings-path=pyproject.toml"
I got the error:
"Error: arguments passed in without any paths or content."

Additionally, I saw one project with source configuration https://github.com/Pylons/pyramid/blob/master/pyproject.toml,
but they had to give explicit folders too 8061fce297/tox.ini (L26)
and 8061fce297/tox.ini (L66)

It was a similar situation with "check" and "diff".
In the documentation it's said that for both check and diff are not
supported in configuration files.
See:
- https://pycqa.github.io/isort/docs/configuration/options.html#check
- https://pycqa.github.io/isort/docs/configuration/options.html#show-diff

Additionally, in two issues it was confirmed that in integration tests
we should use --check and --diff the way we did until now.

As a result, I moved part of the configuration options for isort inside
pyproject.toml without the actual directories that need to be linted
and "check" and "diff" options.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-01 16:28:15 +02:00
Martin Vrachev
4597761adb Move black configuration in pyproject.toml
We are using 4 linters: black, isort, pylint and mypy.
It's good if we use one file as a source for truth for all linter
configurations.

As a first step move black options in pyproject.toml.
I tried multiple ways to use the include option,
so we can just call black --config=pyproject.toml, but I was not
successful. Then I found this comment https://github.com/psf/black/issues/861#issuecomment-680411125
explaining that the path argument is mandatory.
As a result, I will move all configuration options for black inside
pyproject.toml without the actual directories that need to be linted.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-01 16:28:13 +02:00
Martin Vrachev
8ba3cc4394 Move mypy and pylint configs in pyproject.toml
This aims to add a single source of truth for pylint and mypy
configurations.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-01 15:49:41 +02:00
lukpueh
171f9ee51d
Merge pull request #1698 from jku/remove-roadmap
docs: Remove ROADMAP
2021-11-30 10:16:01 +01:00
Jussi Kukkonen
0a453e2fcc docs: Remove ROADMAP
This file is out of date to the point of being obsolete. An updated
ROADMAP document would be warmly welcome but an out of date roadmap
is worse than nothing.

Fixes #1525

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-11-30 10:22:17 +02:00
Jussi Kukkonen
bca1e67adc
Merge pull request #1686 from ivanayov/ivanayov/remove_url_normalisation
Remove URL normalisation
2021-11-30 10:02:45 +02:00
Jussi Kukkonen
2de883aedc
Merge pull request #1690 from kairoaraujo/issue#1558-delegate-roles-name-not-top-level-roles
Prevents delegate role name as top-level role name
2021-11-29 15:52:58 +02:00
Kairo de Araujo
72d68f34fb Remove the unnecessary set()
The set() is not required in the OrderedDict.

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-11-29 14:37:29 +01:00
lukpueh
51248db173
Merge pull request #1685 from lukpueh/repo-examples
Docs: Add repository tutorial based on metadata API
2021-11-29 13:31:16 +01:00
Lukas Puehringer
a1531d8c50 test: add test module for basic repo example
Adds new test module that executes the basic repo example
Python script and checks that it created certain (metadata)
files.

The test module is tailored for testing similar example scripts.

Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Jussi Kukkonen <jku@goto.fi>

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-11-29 13:25:26 +01:00
Lukas Puehringer
190c21b65b doc: add repository example based on metadata API
As 'repository_tool' and 'repository_lib' are being deprecated,
repository metadata must to be created and maintained manually
using the low-level Metadata API. The added example code shall
serve as temporary replacement until a new repository tool is
available.

The sample code contains the following repo workflows:
 - creation of top-level metadata
 - target file handling
 - consistent snapshots
 - key management
 - top-level delegation and signing thresholds
 - target delegation
 - in-band and out-of-band metadata signing
 - writing and reading metadata files
 - root key rotation

Co-authored-by: Teodora Sechkova <tsechkova@vmware.com>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Jussi Kukkonen <jku@goto.fi>

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-11-29 13:25:10 +01:00
Ivana Atanasova
aa591920a2 Remove URL normalisation
As a target path is a URL path it's not correct to consider it as
interchangeable with a filepath within every operation system. The
unquote is also removed as the ngclient cannot assume correctly
which encoding is intended and which not

Fixes #1483

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-11-25 15:42:28 +02:00
Jussi Kukkonen
600eb8636a
Merge pull request #1687 from MVrachev/address-mypy-warnings
Tests on the new implementation: address mypy warnings
2021-11-25 11:48:08 +02:00
Martin Vrachev
e2deff3148 Address mypy warnings
This commit includes manual fixes for a lot of mypy warnings.
When there were warnings that we are calling non-annotated function
in annotated context I decided to add annotations instead of ignoring
those warnings.
That's how I end up adding annotations in the whole tests/utils.py
module.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 20:50:10 +02:00
Martin Vrachev
0d4d7f820c Add two more invalid serialization tests
Those tests are about missing "keys" and "roles" attributes in
Targets.Delegations.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Martin Vrachev
1f77b107df Fix annotations in test_metadata_serialization
In test_metadata_serialization.py "test_case_data" is actually a string
when the decorator calls the actual test functions.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Martin Vrachev
e18b6ba506 Configure mypy to show error codes
By configuring mypy to show error codes when we get a warning by mypy
we will receive an error code as well.
Those error codes are useful when you want to disable specific mypy
warning for a line with:

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Kairo de Araujo
b6c482e1ee Ccheck empty strings, remove comprehensive lists
- Add the check for empty strings in the Delegate Role name
- Remove the comprehensive lists to make the code more readable
- Remove the test for empty file name from
``test_updater_with_simulator``

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-11-24 17:12:45 +01:00
Jussi Kukkonen
a24c4e95be
Merge pull request #1593 from sechkova/avoid_reloading_targets
ngclient: Avoid loading targets metadata twice
2021-11-24 14:15:19 +02:00
Jussi Kukkonen
8e6485b066
Merge pull request #1688 from MVrachev/delegationrole-attributes-validation
Delegations: paths & path_hash_prefixes validation
2021-11-24 14:12:02 +02:00
Kairo de Araujo
597fc88036 Fix reuse datasets/tests, better keyids, standards
- Reuse the dataset and the existing tests
- Fix the keyids in the tests datasets to be aligned
- Fix the ``ValueError`` message aligned to the existent messages

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-11-23 13:58:26 +01:00
Teodora Sechkova
4db4737504
Avoid loading targets metadata twice
When traversing the delegations tree looking for targets,
avoid re-loading already verified targets metadata.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-23 12:21:49 +02:00
Kairo de Araujo
f5bf6b0784 Prevents delegate role name as top-level role name
This commit adds the validation in the ``metadata.Delegations``
to prevent that one of the delegate role names given is a top-level
role name.

A ``ValueError`` will be raised if one of the roles names in the
list given to as delegated contains the role name as one of the
top-level roles.

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-11-22 16:33:06 +01:00
Jussi Kukkonen
acb201d6cc
Merge pull request #1680 from ivanayov/ivanayov/ngclient_loaded_metadata
Update ngclient to return loaded metadata
2021-11-22 16:12:42 +02:00
Jussi Kukkonen
fb645f9592
Merge pull request #1683 from ivanayov/ivanayov/ngclient_skip_visited_roles
Update ngclient to skip visited nodes on delegation tree traversal
2021-11-22 16:03:54 +02:00
Jussi Kukkonen
747ec19934
Merge pull request #1668 from MVrachev/validate-rest-of-metadata-api
Metadata API: make Root roles a Mapping
2021-11-22 16:02:09 +02:00
Martin Vrachev
2adddf7402 Delegations: paths & path_hash_prefixes validation
Add sanity types checks on Targets delegation paths and
path_hash_prefixes making sure that they are strings.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-22 14:43:43 +02:00
Ivana Atanasova
9c2bf6e784 Update ngclient to return loaded metadata
This changes `TrustedMetadataSet` to return new trusted Metadata
on successful calls of the `update_<role>` functions and also
changes `Updater._load_targets` to return loaded metadata as well

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-11-22 14:36:12 +02:00
Ivana Atanasova
aff96c31d1 Update ngclient to skip visited nodes on delegation tree traversal
This change edits the ngclient `Updater` to traverse the delegation
tree on nodes, instead of edges in order to skip already visited
nodes.

For more detailed clarification, please review
theupdateframework/specification#177

Fixes #1528

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-11-18 19:18:33 +02:00
Jussi Kukkonen
1b5df4ccfe
Merge pull request #1666 from sechkova/reposim-consistent-snapshot
RepositorySimulator: add non-consistent snapshot support
2021-11-18 14:03:17 +02:00
Teodora Sechkova
da1d975db0
Remove test_refresh_on_consistent_targets
Consistent snapshot and consistent targets are now
extensively tested in test_updater_consistent_snapshot.py.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-18 12:40:57 +02:00
Teodora Sechkova
a72fa62906
ngtests: Add consistent_snapshot tests
Add tests for ngclient.Updater toggling
'consitent_snapshot' and 'prefix_targets_with_hash'.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-18 11:14:12 +02:00
Teodora Sechkova
54969e8c76
RepositorySimulator: add non-consistent snapshot support
Extend URL partitioning to support serving non-versioned
metadata and non-prefixed target files.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-18 11:13:10 +02:00
Jussi Kukkonen
bb15ecfb99
Merge pull request #1677 from jku/advertize-annotations
Advertize type annotations with py.typed
2021-11-17 10:51:09 +02:00
Teodora Sechkova
67ff0424e7
Merge pull request #1654 from jku/make-refresh-optional
ngclient: Implicitly call refresh()
2021-11-17 10:50:34 +02:00
Jussi Kukkonen
6744f6a9c7
Merge pull request #1652 from jku/limit-github-token-visibility
GH actions: limit GitHub token visibility
2021-11-17 10:06:31 +02:00
Jussi Kukkonen
ec285a448c
Merge pull request #1674 from MVrachev/pylint-more-changes
Tests: address new pylint warnings
2021-11-16 09:57:40 +02:00
Martin Vrachev
29f936b76d Tests: address new pylint warnings
After the recent changes there are a couple of new pylint warnings that
appeared.
They are caused by the new test file that was added
test_updater_top_level_update.py and the limit of public functions was
reached in the TestMetadata class in test_api.py
The warnings should be addressed before enabling all of the linters
on the tests files.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-12 17:10:06 +02:00
Jussi Kukkonen
7951ed2b86 Advertize type annotations with py.typed
Existence of py.typed in the top-level package tells users of the
packages that the packages are annotated.

This should fix mypy errors like this in downstream projects:
  error: Skipping analyzing "tuf.api.metadata":
  found module but no type hints or library stubs

Fixes #1633

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-11-11 22:34:22 +02:00
Jussi Kukkonen
fa7990c354
Merge pull request #1670 from MVrachev/assert-raises-fix
Tests: self.assertRaises -> with self.assertRaises
2021-11-11 18:51:51 +02:00
Jussi Kukkonen
45f69a2639
Merge pull request #1649 from MVrachev/clarify-root-rotations-test
Clarify key rotations test cases by using keywords
2021-11-11 18:51:25 +02:00