Commit graph

4616 commits

Author SHA1 Message Date
lukpueh
51248db173
Merge pull request #1685 from lukpueh/repo-examples
Docs: Add repository tutorial based on metadata API
2021-11-29 13:31:16 +01:00
Lukas Puehringer
a1531d8c50 test: add test module for basic repo example
Adds new test module that executes the basic repo example
Python script and checks that it created certain (metadata)
files.

The test module is tailored for testing similar example scripts.

Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Jussi Kukkonen <jku@goto.fi>

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-11-29 13:25:26 +01:00
Lukas Puehringer
190c21b65b doc: add repository example based on metadata API
As 'repository_tool' and 'repository_lib' are being deprecated,
repository metadata must to be created and maintained manually
using the low-level Metadata API. The added example code shall
serve as temporary replacement until a new repository tool is
available.

The sample code contains the following repo workflows:
 - creation of top-level metadata
 - target file handling
 - consistent snapshots
 - key management
 - top-level delegation and signing thresholds
 - target delegation
 - in-band and out-of-band metadata signing
 - writing and reading metadata files
 - root key rotation

Co-authored-by: Teodora Sechkova <tsechkova@vmware.com>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Jussi Kukkonen <jku@goto.fi>

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-11-29 13:25:10 +01:00
Jussi Kukkonen
600eb8636a
Merge pull request #1687 from MVrachev/address-mypy-warnings
Tests on the new implementation: address mypy warnings
2021-11-25 11:48:08 +02:00
Martin Vrachev
e2deff3148 Address mypy warnings
This commit includes manual fixes for a lot of mypy warnings.
When there were warnings that we are calling non-annotated function
in annotated context I decided to add annotations instead of ignoring
those warnings.
That's how I end up adding annotations in the whole tests/utils.py
module.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 20:50:10 +02:00
Martin Vrachev
0d4d7f820c Add two more invalid serialization tests
Those tests are about missing "keys" and "roles" attributes in
Targets.Delegations.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Martin Vrachev
1f77b107df Fix annotations in test_metadata_serialization
In test_metadata_serialization.py "test_case_data" is actually a string
when the decorator calls the actual test functions.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Martin Vrachev
e18b6ba506 Configure mypy to show error codes
By configuring mypy to show error codes when we get a warning by mypy
we will receive an error code as well.
Those error codes are useful when you want to disable specific mypy
warning for a line with:

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Jussi Kukkonen
a24c4e95be
Merge pull request #1593 from sechkova/avoid_reloading_targets
ngclient: Avoid loading targets metadata twice
2021-11-24 14:15:19 +02:00
Jussi Kukkonen
8e6485b066
Merge pull request #1688 from MVrachev/delegationrole-attributes-validation
Delegations: paths & path_hash_prefixes validation
2021-11-24 14:12:02 +02:00
Teodora Sechkova
4db4737504
Avoid loading targets metadata twice
When traversing the delegations tree looking for targets,
avoid re-loading already verified targets metadata.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-23 12:21:49 +02:00
Jussi Kukkonen
acb201d6cc
Merge pull request #1680 from ivanayov/ivanayov/ngclient_loaded_metadata
Update ngclient to return loaded metadata
2021-11-22 16:12:42 +02:00
Jussi Kukkonen
fb645f9592
Merge pull request #1683 from ivanayov/ivanayov/ngclient_skip_visited_roles
Update ngclient to skip visited nodes on delegation tree traversal
2021-11-22 16:03:54 +02:00
Jussi Kukkonen
747ec19934
Merge pull request #1668 from MVrachev/validate-rest-of-metadata-api
Metadata API: make Root roles a Mapping
2021-11-22 16:02:09 +02:00
Martin Vrachev
2adddf7402 Delegations: paths & path_hash_prefixes validation
Add sanity types checks on Targets delegation paths and
path_hash_prefixes making sure that they are strings.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-22 14:43:43 +02:00
Ivana Atanasova
9c2bf6e784 Update ngclient to return loaded metadata
This changes `TrustedMetadataSet` to return new trusted Metadata
on successful calls of the `update_<role>` functions and also
changes `Updater._load_targets` to return loaded metadata as well

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-11-22 14:36:12 +02:00
Ivana Atanasova
aff96c31d1 Update ngclient to skip visited nodes on delegation tree traversal
This change edits the ngclient `Updater` to traverse the delegation
tree on nodes, instead of edges in order to skip already visited
nodes.

For more detailed clarification, please review
theupdateframework/specification#177

Fixes #1528

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-11-18 19:18:33 +02:00
Jussi Kukkonen
1b5df4ccfe
Merge pull request #1666 from sechkova/reposim-consistent-snapshot
RepositorySimulator: add non-consistent snapshot support
2021-11-18 14:03:17 +02:00
Teodora Sechkova
da1d975db0
Remove test_refresh_on_consistent_targets
Consistent snapshot and consistent targets are now
extensively tested in test_updater_consistent_snapshot.py.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-18 12:40:57 +02:00
Teodora Sechkova
a72fa62906
ngtests: Add consistent_snapshot tests
Add tests for ngclient.Updater toggling
'consitent_snapshot' and 'prefix_targets_with_hash'.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-18 11:14:12 +02:00
Teodora Sechkova
54969e8c76
RepositorySimulator: add non-consistent snapshot support
Extend URL partitioning to support serving non-versioned
metadata and non-prefixed target files.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-18 11:13:10 +02:00
Jussi Kukkonen
bb15ecfb99
Merge pull request #1677 from jku/advertize-annotations
Advertize type annotations with py.typed
2021-11-17 10:51:09 +02:00
Teodora Sechkova
67ff0424e7
Merge pull request #1654 from jku/make-refresh-optional
ngclient: Implicitly call refresh()
2021-11-17 10:50:34 +02:00
Jussi Kukkonen
6744f6a9c7
Merge pull request #1652 from jku/limit-github-token-visibility
GH actions: limit GitHub token visibility
2021-11-17 10:06:31 +02:00
Jussi Kukkonen
ec285a448c
Merge pull request #1674 from MVrachev/pylint-more-changes
Tests: address new pylint warnings
2021-11-16 09:57:40 +02:00
Martin Vrachev
29f936b76d Tests: address new pylint warnings
After the recent changes there are a couple of new pylint warnings that
appeared.
They are caused by the new test file that was added
test_updater_top_level_update.py and the limit of public functions was
reached in the TestMetadata class in test_api.py
The warnings should be addressed before enabling all of the linters
on the tests files.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-12 17:10:06 +02:00
Jussi Kukkonen
7951ed2b86 Advertize type annotations with py.typed
Existence of py.typed in the top-level package tells users of the
packages that the packages are annotated.

This should fix mypy errors like this in downstream projects:
  error: Skipping analyzing "tuf.api.metadata":
  found module but no type hints or library stubs

Fixes #1633

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-11-11 22:34:22 +02:00
Jussi Kukkonen
fa7990c354
Merge pull request #1670 from MVrachev/assert-raises-fix
Tests: self.assertRaises -> with self.assertRaises
2021-11-11 18:51:51 +02:00
Jussi Kukkonen
45f69a2639
Merge pull request #1649 from MVrachev/clarify-root-rotations-test
Clarify key rotations test cases by using keywords
2021-11-11 18:51:25 +02:00
Jussi Kukkonen
41b20df767
Merge pull request #1664 from MVrachev/apply-pylint
New implementation tests: address pylint warnings
2021-11-11 16:10:26 +02:00
Martin Vrachev
d4187f3186 Address pylint warnings on tests for the new code
Address or disable pylint warnings raised on all test files inside
the "tests/" directory testing the code of the new implementation.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-11 15:39:56 +02:00
Martin Vrachev
74c657dc9b Pylint config: add _ as a good variable name
_ is often used when a function returns multiple values and you need
a sub-portion of them. Then, those values that are unnecessary can be
named _.
Currently, pylint warns us that this is not a good variable name, so
fix that.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-11 15:37:49 +02:00
Martin Vrachev
de27f09a44 Pylint: disable duplicate-code for the new code
Pylint reported a couple of warnings flagged as "duplicate-code".
We were truly duplicating code - one of the examples was when we
imported the same objects from tuf/api/metadata.py:
MetaFile, Role, Root, Snapshot, TargetFile, Targets, and Timestamp
in two separate modules.
So, I thought we do want to be repetitive here and include that code at
both modules. The problem is that besides importing the above
classes the modules imported other classes from tuf.api.metadata.py
and there was no way to disable this check.
I searched and found out that this is a known problem:
https://github.com/PyCQA/pylint/issues/214.

That's why the only solution I see is to disable this warning
temporarily and hoping that one day when this issue is fixed we will
remember to turn it on again.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-11 15:37:49 +02:00
Martin Vrachev
2042a54dcf disable black for part of key rotations tests
In order to make the `RootVersion` test cases understandable it's
better if we use keywords and don't split the different RootVersions
into multiple lines.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-11 15:31:04 +02:00
Jussi Kukkonen
5a9b3fa963
Merge pull request #1671 from MVrachev/split-test
Tests: test_api split test_sign_verify()
2021-11-11 10:32:12 +02:00
Martin Vrachev
8c6e157519 Tests: test_api split test_sign_verify()
test_sign_verify() is testing too many cases and after the recent
pylint warning about the usage of too many local variables it became
clear it's time to split this test function.

I decided to split it logically as half of the function was about
failures connected with verify.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-10 15:55:34 +02:00
Martin Vrachev
3c80c5bcd0 Tests: self.assertRaises -> with self.assertRaises
Change the instances of "self.assertRaises" to "with self.assertRaises"
as using "self.assertRaises" can lead to long statements separated to
multiline expressions as pointed out by Jussi here:
https://github.com/theupdateframework/python-tuf/pull/1658#discussion_r741725382

On another hand "with self.assertRaises()" looks a lot better:
589ed9e0d4/tests/test_api.py (L131)

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-10 15:27:03 +02:00
Jussi Kukkonen
0088ebd444
Merge pull request #1636 from sechkova/ng-tests-metadata-update
ngclient: top-level-roles update tests
2021-11-10 10:09:15 +02:00
Teodora Sechkova
d66c3baf27
RepoSim: remove metadata version check
Except for 'root' role, RepositorySimulator does not
keep previous metadata versions, it always serves the latest
one. The metadata version check during fetch serves mostly
for informative purposes and removing it allows generating test
metadata with mismatching version.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-09 16:22:38 +02:00
Teodora Sechkova
8418d5267f
ngtests: Add asserts for expected version
Define _assert_version_equals for checking if the
local metadata file's version is as expected.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-09 14:18:24 +02:00
Teodora Sechkova
e51642a290
ngtests: Fix formatiing and linter issues
Fix formatting and some potential linter and typing
errors.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-09 14:18:17 +02:00
Teodora Sechkova
8a2c7857ac
ngtests: Add addtional asserts for files on disk
Extend the TestRefresh cases with additional checks
for expected metadata files and their content written
on the file system.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-09 14:18:05 +02:00
Martin Vrachev
29da5da4bb Metadata API: make root roles Mapping
Mark roles as a Mapping to indicate that users should not add or remove
values from the dictionary during the lifetime of the Root object)

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-08 19:30:49 +02:00
Teodora Sechkova
954331c8af
ngtests: Add top-level-roles update tests
Add ngclient/updater tests following the top-level-roles metadata
update from the specification (Detailed client workflow)
using RepositorySimulator.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-11-08 17:47:15 +02:00
Martin Vrachev
beb8087bf5 Clarify key rotations test cases by using keywords
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-05 16:46:33 +02:00
Jussi Kukkonen
8ae944ccb2
Merge pull request #1659 from MVrachev/disable-pylint-format
pylintc for new code: disable format checker
2021-11-05 16:19:22 +02:00
Martin Vrachev
b137fbcc3f plyintrc: remove redundant format section
The "FORMAT" section in pylint is no longer needed after the format
checker is disabled.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-05 15:07:39 +02:00
lukpueh
d70c3b32f2
Merge pull request #1663 from jku/gh-actions-set-permissions
github: explicitly set workflow permissions
2021-11-04 10:52:45 +01:00
Jussi Kukkonen
e073fea819 github: explicitly set workflow permissions
* current workflow only needs to read git content
* if the workflow in the future does need write access, it's good to
  see permissions explicitly changing

For context: "pull_request" runs never have write access anyway, so this
significantly changes only the "push" runs that happen when branches are
merged to develop.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-11-04 11:39:05 +02:00
Jussi Kukkonen
ce4a60eb26
Merge pull request #1658 from MVrachev/apply-linters
New implementation tests: apply black and isort automatic linting fixes
2021-11-04 09:32:56 +02:00