Commit graph

6224 commits

Author SHA1 Message Date
Jussi Kukkonen
37a4d41aad Metadata API: Implement threshold verification
The delegating Metadata (root or targets) verifies that the delegated
metadata is signed by required threshold of keys for the delegated
role.

Calling the function on non-delegator-metadata or giving a rolename
that is not actually delegated by the delegator is considered a
programming error and ValueError is raised.

If the threshold is not reached, UnsignedMetadataError is raised.

Tweak type annotation of Delegations.keys to match the one for
Root.keys (so they can be assigned to same local variable).

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 15:13:00 +03:00
Jussi Kukkonen
745a8f7680
Merge pull request #1408 from jku/merge-ngclient
Merge ngclient: a new client library implementation
2021-07-05 12:33:15 +03:00
Jussi Kukkonen
ffff7f5597 ngclient: Improve dosctrings and error messages
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 12:10:49 +03:00
Jussi Kukkonen
1b404f3328 ngclient Updater: Fix annotations/docstrings
* updated_targets() both takes and returns a list
* download_target() argument can come from either
  updated_targets() or get_one_valid_targetinfo()

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 10:45:19 +03:00
Jussi Kukkonen
82dcb507c7 ngclient TrustedMetadataSet: Improve docstrings
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 10:45:19 +03:00
Jussi Kukkonen
4811e880c0 Rename MetadataBundle to TrustedMetadataSet
TrustedMetadataSet is a long name but
 * it better describes the main feature
 * the name isn't used in too many places

Change the variable names "bundle" -> "trusted_set"

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 10:45:19 +03:00
Teodora Sechkova
a31f2ecb4a Document fast-forward attack recovery
Document why deleting the timestamp and snapshot files
is not needed to recover from a fast-forward attack.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-07-05 10:45:19 +03:00
Teodora Sechkova
0f69f28710 Check snapshot value for None
For consistency with the rest if the checks.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-07-05 10:45:19 +03:00
Teodora Sechkova
10b28f5c75 Improve comments
Remove outdated comments.
Add explanations to non-obvious cases.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-07-05 10:45:19 +03:00
Teodora Sechkova
78a0f4d3b5 Remove unsued imports in tests
Remove unused and outdated imports in test_updater_ng.py

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-07-05 10:45:19 +03:00
Teodora Sechkova
a3ebc7c3bb Improve README
Improve the README text.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-07-05 10:45:19 +03:00
Jussi Kukkonen
fac151da3d ngclient: Add initial testing
This testing lacks coverage but demonstrates the happy cases.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 10:45:19 +03:00
Jussi Kukkonen
765c6fe020 ngclient: Add new client code
This is a new client library implementation using the Metadata API
(the only "old" file used is exceptions.py)
* Functional but largely still untested
* Requires work before it can replace tuf.client but should be good
  enough to include in the repo

The major changes compared to current client so far are:
 * Use of Metadata API
 * Major simplification in the way downloads are handled due to removing
   mirrors support
 * Separating tracking of valid metadata into a separate component
 * There's no MultiRepoUpdater

We do not expect other major changes (in the sense of moving large
amounts of code) but do plan to possibly improve the client API. The
API has already changed so this is not going to be a 1:1 compatible
implementation, but porting should not be difficult.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 10:45:19 +03:00
Jussi Kukkonen
775c5c6a81 tox: Enable testing and linting ngclient
* Use the same rules as tuf/api
* omit ngclient from coverage limits for now: #1309

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 10:45:19 +03:00
Jussi Kukkonen
6da206516b ngclient: Add new modules
Start building the next-gen client: Copy existing components from the
current client.

All of these files have some changes compared to the already existing
copies (because ngclient uses the same linting rules as Metadata API).
* download.py is likely to see major changes in the future.
* requests_fetcher is likely to see some minor changes (like allowing
  compression)

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 10:45:19 +03:00
Jussi Kukkonen
54a2f069ce exceptions: Make more errors RepositoryErrors
Try to make errors derive from RepositoryError if they can be the
result of malicious or malfunctioning remote repository: The idea
here is that client can handle just RepositoryError instead of
individual errors (as it cannot do anything about any of them).

Also improve variable naming.

This is backwards compatible.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 10:45:15 +03:00
Marina Moore
c0e6673a5b
Merge pull request #1472 from mnm678/licencing
Remove license from Thandy files that are no longer present
2021-07-02 10:25:27 -04:00
Joshua Lock
7815f055e7
Merge pull request #1455 from jku/mypy-tweaks
Harden mypy configuration
2021-07-02 10:27:20 +01:00
Martin Vrachev
f34cc7e2cb Metadata API: simplify testing unrecognized_fields
We have merged ADR 8 allowing for unrecognized fields and we have
added tests for that which are too specific and not scalable.

Now, I use table testing which we have used initially in https://github.com/theupdateframework/tuf/pull/1416
to test unrecognized fields support in a cleaner and much more readable
way.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-07-01 15:40:47 +03:00
Jussi Kukkonen
e61055afd2
Merge pull request #1468 from theupdateframework/dependabot/pip/urllib3-1.26.6
build(deps): bump urllib3 from 1.26.5 to 1.26.6
2021-06-30 09:14:54 +03:00
Marina Moore
f5d08bb0e8 Remove license from Thandy files that are no longer present
There are no longer any files in TUF that came from Thandy, and
so the dual license is no longer required.

Signed-off-by: Marina Moore <mnm678@gmail.com>
2021-06-29 12:15:07 -04:00
Joshua Lock
ef9a0081b8
Merge pull request #1442 from jku/fix-sign-return-value
Metadata API: Fix Metadata.sign() return value annotation
2021-06-29 14:09:45 +01:00
dependabot[bot]
8ceab4df08
build(deps): bump urllib3 from 1.26.5 to 1.26.6
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.5 to 1.26.6.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.6/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.5...1.26.6)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-28 10:06:39 +00:00
Jussi Kukkonen
505b82a82c Metadata API: Fix Metadata.sign() return value
We've been returning Signature objects since 49aa0fc167.

Also add a test case that does something with the returned signature.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-23 14:35:36 +03:00
Jussi Kukkonen
f3d26552ed mypy: disallow untyped calls
report an error whenever a function with type annotations calls a
function defined without annotations.

Also include exceptions.py in mypy checks.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-23 10:51:33 +03:00
Jussi Kukkonen
929b4b2a6b exceptions: Add type annotations
exceptions.py is used from metadata.py: Annotating it is required if we
want to enable "no_untyped_call" in mypy.

The only change is UnsignedMetadataError: I would have removed argument
"signable" altogether but figured this is not worth breaking API. So
now "signable" is still not used for anything but it is now optional.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-23 10:51:33 +03:00
Jussi Kukkonen
e69f34ba89 mypy: Enable stricter configuration
We're quite close to being able to enable only "strict = True" but not
quite there. In the mean time enable some useful individual settings.

disallow_untyped_defs:
    report an error whenever code contains a function definition
    without type annotations
warn_redundant_casts:
    report an error whenever code uses an unnecessary cast
warn_unused_ignores:
    report an error whenever code uses an unnecessary # type: ignore
    comment
warn_unreachable:
    report an error whenever code is determined to be unreachable or
    redundant

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-23 10:51:33 +03:00
Jussi Kukkonen
f59fee59b0 Metadata API: Remove unreachable code
unrecognized_fields is always initialized

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-23 10:51:33 +03:00
Jussi Kukkonen
81def2dfcc Metadata API: Add missing type annotations
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-23 10:51:33 +03:00
Jussi Kukkonen
f458e9230a
Merge pull request #1456 from jku/tighten-comments
Metadata API: Rewrite comments
2021-06-23 10:20:12 +03:00
Jussi Kukkonen
7108ea2e0e
Merge pull request #1454 from sechkova/hashes-handle-sslib-errors
BaseFile._verify_hashes: handle sslib errors
2021-06-23 10:19:14 +03:00
Jussi Kukkonen
97da5ab991
Merge pull request #1416 from MVrachev/comprehensive-testing
New API: Comprehensive serialization testing
2021-06-23 10:17:50 +03:00
Jussi Kukkonen
b860ad813f
Merge pull request #1453 from avelichka/develop
Add 'ecdsa' to the list of supported key types
2021-06-23 10:17:19 +03:00
Jussi Kukkonen
79f4f41979 Metadata: Improve DelegatedRole docstring
Explain the ways a delegation can happen: Do not try to cover the
complete process (specification should do that) but offer enough
details that the complexity is not completely hidden from the viewer.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-22 15:33:26 +03:00
Jussi Kukkonen
38b6d440c0 Metadata API: Rewrite comments
Try to keep dostrings and comments to the point, avoid mentioning
details if they are not necessary or are likely to become outdated
and try to minimize number of comment lines.

Co-authored-by: Joshua Lock <jlock@vmware.com>
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-22 15:33:07 +03:00
Martin Vrachev
d0828bd81b New API: Comprehensive serialization testing
The idea of this commit is to separate (de)serialization testing outside
test_api.py and make sure we are testing from_dict/to_dict for all
possible valid data for all classes.

Jussi in his comment here:
https://github.com/theupdateframework/tuf/issues/1391#issuecomment-849390669
proposed using decorators when creating comprehensive testing
for metadata serialization.
The main problems he pointed out is that:
1) there is a lot of code needed to generate the data for each case
2) the test implementation scales badly when you want to add new
cases for your tests, then you would have to add code as well
3) the dictionary format is not visible - we are loading external files
and assuming they are not changed and valid

In this change, I am using a decorator with an argument that complicates
the implementation of the decorator and requires three nested functions,
but the advantages are that we are resolving the above three problems:
1) we don't need new code when adding a new test case
2) a small amount of hardcoded data is required for each new test
3) the dictionaries are all in the test module without the need of
creating new directories and copying data.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-06-22 12:51:40 +03:00
Teodora Sechkova
752a741d3a
Handle sslib exceptions in BaseFile._verify_hashes
Securesystemslib digest() and digest_fileobject()
calls raise sslib specific exceptions that need to be
handled and re-raised as TUF exceptions.

Updated tests in test_api.py accordingly.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-06-22 11:35:00 +03:00
Jussi Kukkonen
e6f743bbe3
Merge pull request #1435 from jku/handle-exceptions-in-verify
Handle exceptions in verify
2021-06-22 09:55:07 +03:00
Jussi Kukkonen
cfc7638ebe
Merge pull request #1451 from sechkova/hash-validation
api/metadata input validation: length and hashes
2021-06-22 09:54:09 +03:00
Velichka Atanasova
06c8fb8cd3 Add a test for the 'ecdsa' key type
Changes tests/repository_data/keystore/root_key3* to be an ecdsa key,
created and encrypted with the generate_ecdsa_key and
encrypt_key methods of securesystemslib.keys.
The test_updater_root_rotation_integration.py test
tests both repotool and updater.

Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
2021-06-21 16:32:49 +03:00
Teodora Sechkova
328f637264
Remove trailing comma from test data
A trailing comma makes any element a one-item tuple.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-06-17 14:38:02 +03:00
Teodora Sechkova
e30faa89be
Remove empty hash dict check from MetaFile
The check for an empty hash dictionary is now part
of the hash validation function.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-06-17 14:38:02 +03:00
Teodora Sechkova
03f39b01e7
Add hash and length validation
- valid length: greater than zero
- valid hashes: a non-empty dictionary of type Dict[str, str]

Checking the validity of hash algorithms is not part
of the metadata input validation and is done by
securesystemslib during  hash verification.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-06-17 14:38:01 +03:00
Velichka Atanasova
0fa6c6f2ca Add 'ecdsa' to the list of supported key types
Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
2021-06-17 11:48:30 +03:00
Jussi Kukkonen
70aff4c1a0 tests: Improve verify_signature tests
Test unknown signature algorithm/scheme.

Also shorten the incorrect (but syntactically valid) signature a bit.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-17 10:50:06 +03:00
Jussi Kukkonen
743c4408d4 Metadata API: Clean up verify_signature() exceptions
Aim to only raise UnsignedMetadataError from verify_signature().

Some of the situations could be things like UnsupportedAlgorithmError
-- where the underlying reason may be a missing dependency -- but it
seems impossible for a client to know whether it's that or whether it
is broken or malicious server side.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-17 10:48:40 +03:00
Jussi Kukkonen
4952b987a5 tests: Test verify_signature with explicit serializer
This is just a tiny bit more test coverage.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-17 10:48:09 +03:00
Jussi Kukkonen
39ed706d72
Merge pull request #1437 from sechkova/hash-verification
Add hash and length verification to MetaFile and TargetFile
2021-06-16 22:18:03 +03:00
Jussi Kukkonen
51c26b7b7a
Merge pull request #1428 from theupdateframework/dependabot/pip/certifi-2021.5.30
build(deps): bump certifi from 2020.12.5 to 2021.5.30
2021-06-16 20:30:22 +03:00
Jussi Kukkonen
6c4e2be196
Merge pull request #1450 from MVrachev/threshold-validation
Metadata API: Add simple threshold validation
2021-06-16 19:56:26 +03:00