Commit graph

159 commits

Author SHA1 Message Date
Martin Vrachev
6db3f69b61 Add small missing tests
Add a test triggering the MetaFile version validation and a TargetFile
test accessing custom.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2022-02-07 17:07:48 +02:00
Martin Vrachev
9533c3f974 Metadata API: add exception tests
Add missing tests testing raising documented
exceptions for "Metadata.sign()",
"Metadata.to_file()" and "Metadata.from_file()".

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2022-01-27 17:34:00 +02:00
Martin Vrachev
0666520e62 Fix type annotation in test_api
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2022-01-27 17:05:57 +02:00
Martin Vrachev
32a4545f0b Replace UnsupportedAlgorithmError with ValueError
UnsupportedAlgorithmError is a detailed securesystemslib exception
and there is no need for TUF to redefine it.
Moreover which hash "algorithms" are allowed is work for
securesystemslib not for TUF.

It's only used once inside "Targetfile.from_data()" and there it's used
to denote that there is a problem with the given argument.
That's why this error can be just replaced with "ValueError".

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2022-01-19 16:52:19 +02:00
Martin Vrachev
4b61be9cf7 Add tuf/api/exceptions.py
Add tuf/api/exceptions.py for exceptions in the new code.
I copied the exceptions from tuf/exceptions.py with a few important
decisions:
1. I only added the exceptions that are used in the new code
2. I removed the general "Error" class as we can directly inherit
Exceptions
3. I tried grouping the exceptions by relevance
4. I removed the second argument "UnsignedMetadataError" as it's only
kept for backward compatibility and is not used
5. I tried following the new code style guidelines and linted the file
with our linters.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2022-01-19 16:52:19 +02:00
Lukas Puehringer
8620f389a8 Metadata API: Remove Signed.bump_version() method
Remove `bump_version()` method, which is just an alias for "+= 1"
on the version attribute. For a slim low-level API it seems okay to
just directly access/modify the attribute.

The extra level of abstraction of "bumping a version" is more
appropriate for a repository library (see #1136).

This patch also removes a related unit test and updates another one
to directly do `(...).version +=`.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2022-01-11 10:56:41 +01:00
Lukas Puehringer
9f2c593813 Metadata API: Remove Signed.bump_expiration()
Remove `bump_expiration()` method, which is unlikely to be used as
is, i.e.  bump to "current expiration date plus delta". A more
realistic use case is to bump to "now plus delta" (see #1727 for
details).

Moreover, bump_expiration can either way easily be replaced by a
one-liner expression using the 'datetime' module. A corresponding
code snippet is added to the `expires` property's docstring.  Note:
`expires` became a property with a millisec-removing setter (for
spec conformance) in  #1712, which further reduces the need for a
convenience bump_expiration method.

This patch also removes a related unit test and updates another
one.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-12-22 11:40:36 +01:00
Lukas Puehringer
f22f357934 Metadata API: Remove 3 'update' methods + tests
Remove ambiguous, unspecific, opinionated and trivial 'update'
methods, which can be replaced by feasible one-liners that assign
values directly to the object attribute to be *updated*. (see #1627
for details).

Reasons to have these methods would be increased usability in terms of
- reduced work
- immediate feedback on invalid assignments

However, given above described issues, the reasons against the
methods as they are now seem to outweigh the reasons for them.
Furthermore, it seems easier to re-add similar methods, which
addressed these issues, after the upcoming 1.0.0 release than to
remove or modify them.

This patch also removes the corresponding tests as they become
irrelevant (there is no need to test object assignment).  In the
case of the timestamp test, the removal also includes redundant
test logic, which is already tested in `test_metadata_base`.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-12-21 09:48:31 +01:00
Martin Vrachev
181b1a7e6d Metadata API: remove dateutil requirement
I added "dateutil" as a possible argument type for
`Metadata.bump_expiration()` as we are already testing for this and
implying it should be supported.
The problem is that "dateutil" is not added as a nontest requirement
and after a discussion, we decided we don't want to add it as well.

That's why we decided to remove "dateutil" mentions from the code
and not confuse our users we support it.

We will create a separate issue discussing the validity of
`Metadata.bump_expiration()`.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-14 11:23:40 +02:00
Martin Vrachev
a728717eda Fix more black warnings on test files
The changes are automatic linting fixes from black.
The target files are only those who test the new code.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-09 15:52:40 +02:00
Ivana Atanasova
00589f08e3 Apply top-level rolenames constants in tests
This applies the use of constants of top-level rolenames in the
tests instead of the previously hardcoded strings.
Fixes #1648

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-12-02 12:25:48 +02:00
Martin Vrachev
e2deff3148 Address mypy warnings
This commit includes manual fixes for a lot of mypy warnings.
When there were warnings that we are calling non-annotated function
in annotated context I decided to add annotations instead of ignoring
those warnings.
That's how I end up adding annotations in the whole tests/utils.py
module.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 20:50:10 +02:00
Martin Vrachev
29f936b76d Tests: address new pylint warnings
After the recent changes there are a couple of new pylint warnings that
appeared.
They are caused by the new test file that was added
test_updater_top_level_update.py and the limit of public functions was
reached in the TestMetadata class in test_api.py
The warnings should be addressed before enabling all of the linters
on the tests files.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-12 17:10:06 +02:00
Jussi Kukkonen
fa7990c354
Merge pull request #1670 from MVrachev/assert-raises-fix
Tests: self.assertRaises -> with self.assertRaises
2021-11-11 18:51:51 +02:00
Martin Vrachev
d4187f3186 Address pylint warnings on tests for the new code
Address or disable pylint warnings raised on all test files inside
the "tests/" directory testing the code of the new implementation.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-11 15:39:56 +02:00
Martin Vrachev
8c6e157519 Tests: test_api split test_sign_verify()
test_sign_verify() is testing too many cases and after the recent
pylint warning about the usage of too many local variables it became
clear it's time to split this test function.

I decided to split it logically as half of the function was about
failures connected with verify.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-10 15:55:34 +02:00
Martin Vrachev
3c80c5bcd0 Tests: self.assertRaises -> with self.assertRaises
Change the instances of "self.assertRaises" to "with self.assertRaises"
as using "self.assertRaises" can lead to long statements separated to
multiline expressions as pointed out by Jussi here:
https://github.com/theupdateframework/python-tuf/pull/1658#discussion_r741725382

On another hand "with self.assertRaises()" looks a lot better:
589ed9e0d4/tests/test_api.py (L131)

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-10 15:27:03 +02:00
Martin Vrachev
6fe36a00c4 Rename & simplify a couple of tests in test_api.py
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-02 19:31:46 +02:00
Martin Vrachev
2e9ef79762 Apply isort on the tests of the new code
All of the changes included are a result of applying isort
on our tests on the new code.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-02 19:31:25 +02:00
Martin Vrachev
c98b429643 Apply black on the tests of the new code
All of the changes included are a result of applying black
on our tests on the new code.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-02 19:30:35 +02:00
Martin Vrachev
f00295f147 API CHANGE: ValueError in add/remove key in Root
This is an API change to the exceptions thrown in Root.add_key()
and Root.remove_key().
The reason for that change is that in my opinion the correct exceptions
in these cases should be "ValueError" instead of "KeyError" as
the problems are in the given values - role doesn't exist or
key is not used by a particular role.

Additionally, document the thrown exceptions in "Root.add_key" and
add a test which invokes that exception.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-21 12:20:09 +03:00
Martin Vrachev
e27070305f Metadata API: Add key helpers in Targets
Root class has the functionality to add and remove keys for delegated
metadata (add_key()/remove_key()) but the other delegator Targets does
not.
It should provide the same/similar functionality.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-21 12:20:09 +03:00
Martin Vrachev
510078b542 Move tests to test_metadata_serialization
Move the Delegation class serialization tests from "test_api.py"
to test_metadata_serialization.py module focused on serialization
testing.

Additionally, a test for empty keys and roles will be added in my
upcomming pr #1511.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-21 12:20:09 +03:00
Martin Vrachev
bf12e7565f Metadata API: change meta type in Timestamp
In Timestamp, the only valid "meta" value is the dictionary representing
meta information for the snapshot file. This makes the API unnecessarily
complicated and requires validation that only information about snapshot
is available inside "meta".
Together with the python-tuf maintainers, we decided that snapshot meta
information will not be represented by a "meta" dictionary but instead
by a MetaFile instance and with this it will diverge from the
specification.
Additionally, to prevent confusion, I will rename the "meta" attribute
to "snapshot_meta" as this attribute will be related only to meta
information about snapshot.

This decision is coherent with ADR9 and the rationale
behind it is to provide easier, safer, and direct access to the
snapshot meta information.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-20 14:09:38 +03:00
Velichka Atanasova
65fd1aaf8a Add an option to create TargetFile from data/file
This is a repository tooling use case but also helpful when testing.
It could be useful when we need to update the targets object.

Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
2021-09-10 13:59:33 +03:00
Jussi Kukkonen
017425e497 tests: Improve DelegatedRole coverage
Test basic cases of DelegatedRole.is_delegated_path()

This is not trying to be an extensive test of possible methods of delegation:
this gives us basic coverage for all code paths.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-01 13:34:50 +03:00
Jussi Kukkonen
c127b6b5a3 tests: Improve add_key/remove_key API tests
Make sure key is not removed from root.signed.keys on remove_key()
if another role is still using the key.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-01 13:34:50 +03:00
Jussi Kukkonen
e1ec782f19
Merge pull request #1520 from avelichka/sslib-key
Add Key.from_securesystemslib_key
2021-08-31 09:42:39 +03:00
Jussi Kukkonen
7d77eeec31
Merge pull request #1512 from MVrachev/glob-pattern-matching
Implement glob-like pattern matching
2021-08-31 09:42:27 +03:00
Jussi Kukkonen
7731738590
Merge pull request #1514 from MVrachev/filename-in-targetfile
Metadata API: include target target name in TargetFile
2021-08-27 11:02:49 +03:00
Martin Vrachev
b18176db9b Implement glob-like pattern matching
According to the recently updated version of the specification the shell
style wildcard matching is glob-like (see https://github.com/theupdateframework/specification/pull/174),
and therefore a path separator in a path should not be matched by a
wildcard in the PATHPATTERN.

That's not what happens with `fnmatch.fnmatch()` which doesn't
see "/" separator as a special symbol.
For example: fnmatch.fnmatch("targets/foo.tgz", "*.tgz") will return
True which is not what glob-like implementation will do.

We should make sure that target_path and the pathpattern contain the
same number of directories and because each part of the pathpattern
could include a glob pattern we should check that fnmatch.fnmatch() is
true on each target and pathpattern directory fragment separated by "/".

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-08-26 19:57:52 +03:00
Martin Vrachev
9229a405e3 Remove filename argument from Targets.update()
After the addition of "path" argument in the TargetFile class the
filename argument in Targets.update() became redundant.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-08-26 19:48:46 +03:00
Martin Vrachev
91b0c59602 Metadata API: include target path in targetfile
Currently, TargetFile instances do not contain the path relative URL of
the file they represent. The API itself does not need it but it could be
useful for users of the API.

As an example, the current client returns a dict for
get_one_valid_targetinfo(): that dict contains a filepath field and
a targetinfo field (essentially TargetFile).
We would like to keep a similar API, but avoid hand-crafted dicts.
It would be much nicer to return a TargetFile that would contain the
full "metadata" of the targetfile.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-08-26 19:48:45 +03:00
Velichka Atanasova
c875b7ed04 Add Key.from_securesystemslib_key
The securesystemslib key dictionary representation includes
the private key in keyval. TUF key doesn't handle it in any way,
but considering that we allow unrecognized symbols in the format,
we should exclude the private key otherwise this could lead to
misuse.
A call to securesystemslib.keys.format_keyval_to_metadata
with the default private=False would do exactly that.

Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
2021-08-26 15:37:25 +03:00
Martin Vrachev
45743444b1 Remove duplicate serialization tests from test_api
I was looking at how can we simplify or split test_api.py when I noticed
that the test cases covered by those two test functions are already
covered in the test_metadata_serialization.py module in the
"invalid_keys" and "invalid_roles" datasets.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-08-25 16:03:03 +03:00
Jussi Kukkonen
13e20e9954 Metadata API: Make Metadata Generic
When we use Metadata, it is helpful if the specific signed type (and all of
the signed types attribute types are correctly annotated. Currently this is
not possible.

Making Metadata Generic with constraint T, where

    T = TypeVar("T", "Root", "Timestamp", "Snapshot", "Targets")

allows these annotations. Using Generic annotations is completely
optional so all existing code still works -- the changes in test code
are done to make IDE annotations more useful in the test code, not
because they are required.

Examples:

    md = Metadata[Root].from_bytes(data)
    md:Metadata[Root] = Metadata.from_bytes(data)

In both examples md.signed is now statically typed as "Root" allowing IDE
annotations and static type checking by mypy.

Note that it's not possible to validate that "data" actually contains a
root metadata at runtime in these examples as the annotations are _not_
visible at runtime at all: new constructors would have to be added for that.

from_file() is now a class method like from_bytes() to make sure both
have the same definition of "T" when from_file() calls from_bytes():
This makes mypy happy.

Partially fixes #1433

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-08-16 16:38:21 +03:00
Martin Vrachev
4f37de1b53 Metadata API: add Metadata.to_bytes()
Metadata.to_bytes() is missing from the API and that is now becoming
annoying when writing the tests.
I think it makes sense to add:
it'll complete the serializing counterparts to from_bytes()/from_file().

We can also reuse to_bytes() in to_file() and that way ensure we don't
import the JSONSerializer locally twice.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-07-15 15:38:14 +03:00
Joshua Lock
bd5912bcc7
Merge pull request #1436 from jku/verify-delegate
Metadata API: Implement threshold verification
2021-07-12 11:45:57 +01:00
Jussi Kukkonen
271d5b7810 Metadata API: verify_delegate: refactor
* Rename arguments so connection between the role name and the
  metadata is stronger.
* Also add a comment on the list comprehension + next() trick.
* Add return value annotation
* Raise early if delegations is None to make the flow more obvious
  (and modify test case so we have coverage for the new case)

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-08 20:16:42 +03:00
Jussi Kukkonen
d00af4c101 tests: Improve verify_delegate() tests
Make sure verify_delegate() succeeds when threshold is reached even if
some signatures fail to verify.

Make sure higher threshold (2/2) works.

Change error type for "Call is valid only on delegator metadata" error.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 15:13:00 +03:00
Jussi Kukkonen
37a4d41aad Metadata API: Implement threshold verification
The delegating Metadata (root or targets) verifies that the delegated
metadata is signed by required threshold of keys for the delegated
role.

Calling the function on non-delegator-metadata or giving a rolename
that is not actually delegated by the delegator is considered a
programming error and ValueError is raised.

If the threshold is not reached, UnsignedMetadataError is raised.

Tweak type annotation of Delegations.keys to match the one for
Root.keys (so they can be assigned to same local variable).

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-07-05 15:13:00 +03:00
Martin Vrachev
f34cc7e2cb Metadata API: simplify testing unrecognized_fields
We have merged ADR 8 allowing for unrecognized fields and we have
added tests for that which are too specific and not scalable.

Now, I use table testing which we have used initially in https://github.com/theupdateframework/tuf/pull/1416
to test unrecognized fields support in a cleaner and much more readable
way.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-07-01 15:40:47 +03:00
Jussi Kukkonen
505b82a82c Metadata API: Fix Metadata.sign() return value
We've been returning Signature objects since 49aa0fc167.

Also add a test case that does something with the returned signature.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-23 14:35:36 +03:00
Jussi Kukkonen
7108ea2e0e
Merge pull request #1454 from sechkova/hashes-handle-sslib-errors
BaseFile._verify_hashes: handle sslib errors
2021-06-23 10:19:14 +03:00
Jussi Kukkonen
97da5ab991
Merge pull request #1416 from MVrachev/comprehensive-testing
New API: Comprehensive serialization testing
2021-06-23 10:17:50 +03:00
Martin Vrachev
d0828bd81b New API: Comprehensive serialization testing
The idea of this commit is to separate (de)serialization testing outside
test_api.py and make sure we are testing from_dict/to_dict for all
possible valid data for all classes.

Jussi in his comment here:
https://github.com/theupdateframework/tuf/issues/1391#issuecomment-849390669
proposed using decorators when creating comprehensive testing
for metadata serialization.
The main problems he pointed out is that:
1) there is a lot of code needed to generate the data for each case
2) the test implementation scales badly when you want to add new
cases for your tests, then you would have to add code as well
3) the dictionary format is not visible - we are loading external files
and assuming they are not changed and valid

In this change, I am using a decorator with an argument that complicates
the implementation of the decorator and requires three nested functions,
but the advantages are that we are resolving the above three problems:
1) we don't need new code when adding a new test case
2) a small amount of hardcoded data is required for each new test
3) the dictionaries are all in the test module without the need of
creating new directories and copying data.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-06-22 12:51:40 +03:00
Teodora Sechkova
752a741d3a
Handle sslib exceptions in BaseFile._verify_hashes
Securesystemslib digest() and digest_fileobject()
calls raise sslib specific exceptions that need to be
handled and re-raised as TUF exceptions.

Updated tests in test_api.py accordingly.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-06-22 11:35:00 +03:00
Jussi Kukkonen
e6f743bbe3
Merge pull request #1435 from jku/handle-exceptions-in-verify
Handle exceptions in verify
2021-06-22 09:55:07 +03:00
Teodora Sechkova
328f637264
Remove trailing comma from test data
A trailing comma makes any element a one-item tuple.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-06-17 14:38:02 +03:00
Jussi Kukkonen
70aff4c1a0 tests: Improve verify_signature tests
Test unknown signature algorithm/scheme.

Also shorten the incorrect (but syntactically valid) signature a bit.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-17 10:50:06 +03:00
Jussi Kukkonen
743c4408d4 Metadata API: Clean up verify_signature() exceptions
Aim to only raise UnsignedMetadataError from verify_signature().

Some of the situations could be things like UnsupportedAlgorithmError
-- where the underlying reason may be a missing dependency -- but it
seems impossible for a client to know whether it's that or whether it
is broken or malicious server side.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-17 10:48:40 +03:00
Jussi Kukkonen
4952b987a5 tests: Test verify_signature with explicit serializer
This is just a tiny bit more test coverage.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-17 10:48:09 +03:00
Jussi Kukkonen
39ed706d72
Merge pull request #1437 from sechkova/hash-verification
Add hash and length verification to MetaFile and TargetFile
2021-06-16 22:18:03 +03:00
Jussi Kukkonen
fa2268df5a
Merge pull request #1449 from MVrachev/key-validation
Metadata API: Add Key attributes types validation
2021-06-16 19:55:34 +03:00
Martin Vrachev
a9dc24adea Metadata API: Add a comment for Key validation
Clarify that we don't semantically validate "Key" instances during
initialization and that this is a responsibility of securesystemslib.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-06-16 16:37:48 +03:00
Teodora Sechkova
22facb7753
Update test_api
Add tests for hash and length verification.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-06-11 11:59:09 +03:00
Jussi Kukkonen
e3190fca9d Metadata API: Store signatures as dict
store signatures in a Dict of keyid to Signature. This ensures
signature uniqueness. Raise in from_dict() if input contains multiple
different signatures for a keyid.

This changes Metadata object API, and makes it slightly different from
the file format: this is justified by making the API safer to use and
easier to validate.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-09 11:21:38 +03:00
Jussi Kukkonen
de78251b00
Merge pull request #1423 from jku/verify-in-key
Metadata API: implement sig verification in Key, store id in key
2021-06-09 10:14:45 +03:00
Jussi Kukkonen
414dfc829f Metadata API: Move signature verification to Key
This is likely not needed by users of the API (as they are interested
in the higher level functionality "verify delegate metadata with
threshold of signatures").

Moving verify to Key makes the API cleaner because including both
"verify myself" and "verify a delegate with threshold" can look awkward
in Metadata, and because the ugly Securesystemslib integration is now
Key class implementation detail (see Key.to_securesystemslib_key()).

Also raise on verify failure instead of returning false: this was found
to confuse API users (and was arguably not a pythonic way to handle it).

* Name the function verify_signature() to make it clear what is being
  verified.
* Assume only one signature per keyid exists: see #1422
* Raise only UnsignedMetadataError (when no signatures or verify failure),
  the remaining lower level errors will be handled in #1351
* Stop using a "keystore" in tests for the public keys: everything we
  need is in metadata already

This changes API, but also should not be something API users want to
call in the future when "verify a delegate with threshold" exists.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-07 13:20:26 +03:00
Jussi Kukkonen
41a6daca75 Metadata API: Add id to Key
This simplifies life for API users as usually a key needs its
identifier: this is already visible in how update() becomes simpler
in the API.

The downside is that 'from_dict()' now has two arguments (so arguably
the name is not great anymore but it still does _mostly_ the same job
as other from_dicts).

This is an API change, if a minor one.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-06-07 13:20:22 +03:00
Martin Vrachev
737c249067 new API: make sure targets in Targets can be empty
This change is relevant to the new metadata class Targets.

In the specification, when describing the Targets metadata file format
and more precisely "TARGETPATH" (or targets containing the actual
target files) it's said:
"It is allowed to have a TARGETS object with no TARGETPATH elements.
This can be used to indicate that no target files are available."

If there is no "TARGETPATH" keys for the dictionary "targets", this
would mean that "Targets.targets" is {}.
Make sure we test for that.

See: https://theupdateframework.github.io/specification/latest/#targetpath

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-25 13:15:00 +03:00
Martin Vrachev
15bf88231d Remove testing "update()" with "version" only
We have tests which make sure we can use `Timestamp.update()` and
`Snapshot.update()` with MetaFile instance storing only version
(because length and hashes are optional).
Those tests were created to make sure that we are actually supporting
optional hashes and length when we call `update` for those classes, but
after we changed the `update()` signature to accept `MetaFile` instance
the tests are obsolete.
The reason is that length and hashes can be optional because of the
MetaFile implementation, no the update function itself and we have
other tests validating creating a MetaFie instance without hashes and
length.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-19 13:21:32 +03:00
Martin Vrachev
37de69050a Change "update()" argument types
Currently, when we call Targets/Snapshot/Timestamp.update() we are
passing all of the necessary values to create MetaFile/Targets File
respectively.
This is not needed, given that one of the reasons we have created
MetaFile and TargetFile is to make the API easier to use.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-19 13:21:32 +03:00
Martin Vrachev
408732f4ff Add MetaFile/TargetFile specific tests
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-19 13:21:32 +03:00
Martin Vrachev
60bbb165a8 New API: Add TargetFile class
In the top-level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.

As written in the spec "targets" in "targets.json" has defined the
"custom" field serving the same purpose as "unrecognized_fields" in the
implementation.
That's why to conform against the spec and support "custom" and allow
"unrecognized_fields" everywhere where it's not sensitive we can define
custom as property which actually access data stored in
unrecognized_fields.
For context read ADR 8 in tuf/docs/adr.

Additionally, after adding the TargetFile class, when we create a
Targets an object we are now calling from dict twice - one for the main
Targets class and one for each of the complex attributes
TargetFile.from_dict() and Delegations.from_dict().
Given that the "from_dict" methods have the side effect of destroying
the given dictionary, we would need to start using deepcopy()
for our tests.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-19 13:21:31 +03:00
Martin Vrachev
3771a77ffe New API: Add MetaFile class
In the top-level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.

Additionally, after adding the MetaFile class, when we create an object
we are now calling from dict twice - one for the main class (Timestamp,
Snapshot) and one for the pacticular complex attribute -
MetaFile.from_dict(). Given that the "from_dict" methods have the
side effect of destroying the given dictionary, we would need to
start using deepcopy() for our tests.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-19 13:04:24 +03:00
Jussi Kukkonen
8348523b77
Merge pull request #1394 from MVrachev/optional-consistent-snapshot
Metadata API: Make consistent_snapshot optional
2021-05-18 21:11:57 +03:00
Martin Vrachev
de2644f3d0 Breaking: new API: consistent_snapshot optional
NOTE: making consistent_snapshot optional requires using a default value
for the argument in __init__ in Root and thus consistent_snapshot should
be rearranged in the end.
Read more: https://github.com/theupdateframework/tuf/pull/1394#issuecomment-842134961

From chapter 7 in the spec (version 1.0.17)
"Finally, the root metadata should write the Boolean
"consistent_snapshot" attribute at the root level of its keys of
attributes.
If consistent snapshots are not written by the repository,
then the attribute may either be left unspecified or be set to the
False value. Otherwise, it must be set to the True value."

We want to make sure we support repositories
without consistent_snapshot set.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-17 11:41:52 +03:00
Jussi Kukkonen
14f5957eee Metadata API: Don't do equality comparisons on containers
Use either "if X is not None:" or a try-except instead of a "if X:".

I believe Targets.from_dict() was not really broken with previous code
but it looks suspicious and did fail the added test with a strange
exception: I expect the from_dict() methods to mainly fail with
KeyErrors, ValueErrors or AttributeErrors if file format structure
is incorrect.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-05-14 16:12:50 +03:00
Jussi Kukkonen
aa480b1280 Metadata API: Fix DelegatedRole serialization issue
A DelegatedRole with paths=[] fails to serialize correctly (paths is not
included in the output json).

Fix the issue, modify tests to notice a regression.

Fixes #1389

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-05-14 14:02:29 +03:00
Teodora Sechkova
167e1793d0 Fix Root.add_key() argument's type
After the implementation of a Key class representing
the public portion of a key, the method add_key() should
take an argument of type Key, instead of a dictionary.

Test cases are updated accordingly.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-05-13 11:23:15 +03:00
Martin Vrachev
42b3269ac1 Remove forgotten debug line from a test
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-11 16:04:25 +03:00
Martin Vrachev
f98f4f750a Add Delegation/DelegatednRole specific tests
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-11 16:04:25 +03:00
Martin Vrachev
8e4d3b9a84 Add DelegatedRole and Delegations
In the top level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.

DelegatedRole shares a couple of fields with the Role class and that's
why it inherits it.
I decided to use a separate Delegations class because I thought it will
make it easier to read, verify and add additional helper functions.
Also, I tried to make sure that I test each level of the delegations
representation for support of storing unrecognized fields.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-11 16:03:56 +03:00
Jussi Kukkonen
17121bad28 Metadata API: remove_key() improvements
We should not do multiple lookups through data structures if one is
enough (here we have extra lookups on both roles and keyids).

Also in this case raising on missing key seems like the preferable
alternative so even a try-except is not needed.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-05-10 17:02:12 +03:00
Martin Vrachev
139bfc0ea9 BREAKING CHANGE: Make delegations optional
According to the spec, delegations in targets are marked as optional:
https://theupdateframework.github.io/specification/latest/#file-formats-targets
and a pr, clarifying that even more, is approved:
https://github.com/theupdateframework/specification/pull/157.

This is a possible breaking change.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-10 15:27:37 +03:00
Martin Vrachev
71c4992cea Make length and hashes optional in Timestamp
As per the specification (v1.0.1) length and hashes fields
in timestamp and snapshot metadata are optional.
We have implement this in the older API
(see https://github.com/theupdateframework/tuf/pull/1031) and we should
implement it in the new API.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-10 15:27:37 +03:00
Martin Vrachev
c0d72d460c New API: Add root use case in couple of tests
Add a use case for the root class to be tested in test_generic_read
and test_read_write_read_compare tests in test_apy.py

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-05-10 15:27:37 +03:00
Martin Vrachev
1ce94b95cb keyid: verify adding an existing key is ignored
Verify that adding an already existing key to keyid for a particular
role in Root won't create duplicate key.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-04-29 12:26:40 +03:00
Martin Vrachev
54a535e4c3 New API: Add Key/Role specific tests
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-04-27 15:26:41 +03:00
Martin Vrachev
eab8c318f0 Add a Role class and integrate it into Root
In the top level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-04-27 15:13:40 +03:00
Martin Vrachev
11e0a87bde Add Key class and integrate it into Root
In the top level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-04-27 15:09:48 +03:00
Jussi Kukkonen
feb340f8da
Merge pull request #1345 from MVrachev/implement-adr-8
New metadata API: add support for ADR 0008
2021-04-22 21:54:32 +03:00
Martin Vrachev
79391f1d85 New API: accept metadata with unrecognized fields
In order to support ADR 0008 we would want to accept unrecognized
fields in all metadata classes.
Input that contains unknown fields in the 'signed' dictionary should
successfully deserialize into a Metadata object, and that object should
successfully serialize with the unknown fields intact.

Also, we should test that we support unrecognized fields when adding
new classes or modifying existing ones to make sure we support
ADR 0008.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-04-22 17:11:05 +03:00
Jussi Kukkonen
4e8738ffa5 api: Add Metadata.from_bytes()
This is essentially short-hand for
    JSONDeserializer().deserialize(data)
but seems much easier for the API user so may be worth it.

Metadata.from_file() now uses Metadata.from_bytes() internally.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-04-21 09:36:34 +03:00
Velichka Atanasova
1eaef0093b Add is_expired method to the Signed class
Checks metadata expiration against a reference time (a naive datetime in UTC).
If not provided, checks against the current UTC date and time.
Returns True if expiration time is less than the reference time.

Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
2021-04-14 13:28:36 +03:00
Martin Vrachev
49aa0fc167 Make new API compatible with the Signing interface
In the securesystemslib pr https://github.com/secure-systems-lab/securesystemslib/pull/319
I added a new Signer interface with the purpose of supporting multiple
signing implementations.
Additionally, I added the SSlibSigner implementation of that interface
which implements the signing operation for rsa, ed25519 and ecdsa
schemes.
With this commit, I integrate the SSlibSigner into the new API in tuf.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-03-10 10:57:45 +01:00
Lukas Puehringer
8e9afc96f9 Revert "Move to/from_dict metadata API methods..."
Revert an earlier commit that moved to/from_dict metadata class
model methods to a util module of the serialization sub-package.

We keep to/from_dict methods on the metadata classes because:
- It seems **idiomatic** (see e.g. 3rd-party libaries such as attrs,
pydantic, marshmallow, or built-ins that provide default or
customizable dict representation for higher-level objects).
The idiomatic choice should make usage more intuitive.
- It feels better **structured** when each method is encapsulated
within the corresponding class, which in turn should make
maintaining/modifying/extending the class model easier.
- It allows us to remove function-scope imports (see subsequent
commit).

Caveat:
Now that "the meat" of the sub-packaged JSON serializer is
implemented on the class, it might make it harder to create a
non-dict based serializer by copy-paste-amending the JSON
serializer.

However, the benefits from above seem to outweigh the disadvantage.

See option 5 of ADR0006 for further details (#1270).

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-03-04 12:46:16 +01:00
Lukas Puehringer
e1be085c3c Move to/from_dict metadata API methods to util
Add tuf.api.serialization.util module with functions to
convert between TUF metadata class model and the corresponding
dictionary representation. These functions replace the
corresponding to/from_dict classmethods.

Configure api/pylintrc to exempt '_type' from protected member
access warning, because the underscore prefix here is only used to
avoid name shadowing.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-03-04 12:33:28 +01:00
Lukas Puehringer
240fb547af Use custom errors in serializer.json sub-package
Re-raise all errors that happen during de/serialization as custom
De/SerializationError.

Whilelist 'e', which is idiomatic for error, in api/pylintrc, and
inline exempt broad-except, which are okay if re-raised.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-03-04 12:33:28 +01:00
Lukas Puehringer
499f1c858e Adopt serialization sub-package in metadata API
- Rename Metadata methods:
  - to_json_file -> to_file
  - from_json_file -> from_file
- Remove Metadata.from_json/to_json
- Remove Signed.to_canonical_bytes
- Accept optional de/serializer arguments:
  - from_file (default: JSONDeserializer)
  - to_file (default: JSONSerializer)
  - sign, verify (default: CanonicalJSONSerializer)
- inline disable pylint cyclic-import checks

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-03-04 12:33:18 +01:00
Joshua Lock
d144141ec7 tests: remove check for python >= 3.6 in test_api
Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-03-03 09:38:39 +00:00
Martin Vrachev
028d1bc9f7 Make "utils" import more definite
Currently, we are importing the "utils" module in tests/utils
with "import utils".
This could become a problem when there is another module with
the same general name "utils" and could lead to import mistakes.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2020-11-23 22:17:31 +02:00
Teodora Sechkova
5bfd9dd94c
Add tests for root and targets metadata
Add test cases for Root(Signed) and Targets(Signed) classes.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2020-11-09 11:38:26 +02:00
Teodora Sechkova
fe439d2014
Fix passing dictionaries by reference in TestMetadata
Use deepcopy to ensure that the dictionaries with expected data
are not referencing the same memory as the tested ones.

Add a check asserting that metadata is not equal prior to its
update.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2020-11-09 11:38:25 +02:00
Joshua Lock
38a86393c2 Make unit test files executable
It's convenient to be able to run unit test scripts directly, rather than
having to pass them as arguments to Python. This is already possible for
several of our unit tests, make it possible for all by setting the execute
bit.

Signed-off-by: Joshua Lock <jlock@vmware.com>
2020-10-05 10:36:31 +01:00
Jussi Kukkonen
03b15fb4be tests: Configure logging for all test files
all test_*.py files now accept zero or more '-v' to increase tuf
logging level. The default is now ERROR.

default: ERROR
"-v":    ERROR, but unittest prints test names
"-vv":   WARNING
"-vvv":  INFO
"-vvvv": DEBUG

Example to run a single test with DEBUG level:
  python3 test_updater.py -vvvv TestUpdater.test_4_refresh

Also make test_log.py restore the log level it modifies during test.

Fixes #1093

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2020-09-15 21:36:50 +03:00
Lukas Puehringer
73dd72d54d Raise on bad signature count in Metadata.verify
Change Metadata.verify(key) behavior to raise an exception if
none or multiple signatures for the passed key are found on the
Metadata object.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2020-09-10 16:18:28 +02:00
Lukas Puehringer
387169fc11 Add from_json metadata convenience wrapper
Add convenience wrapper that takes a json string and passes it
to from_dict to create a Metadata object.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2020-09-10 16:18:28 +02:00
Lukas Puehringer
f63dce6ddd Refactor metadata constructors and add factory
This commit better separates the Metadata class model from the
Metadata wireline format, by tailoring the constructors
towards class-based parameters and adding an additional
factory classmethod that creates Metadata objects based on the
wireline json/dictionary metadata representation. (pythonic
way of constructor overloading).

This 'from_dict' factory method recurses into the 'from_dict'
methods of each contained complex field/attribute that is also
represented by a class. Currently 'signed' is the only such
attribute.

This commit further:
- Changes optional constructor keyword arguments to mandatory
positional arguments: Reduces code and simplifies usage by
restricting it. For now, users are unlikely to call
constructor directly anyway, but the 'from_dict' factory (or
its 'from_json_file' wrapper) instead.

- Removes Signed.__expiration (datetime) vs. Signed.expires
(datestring) dichotomy: Keeping only one representation of the
same attribute in memory makes the interface simpler and less
ambiguous. We choose the datetime object, because it is more
convenient to modify. Transformation from and to the string
format required by the tuf wireline format is performed in the
corresponding metadata de/serialization methods, i.e.
('to_dict' and 'from_dict').

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2020-09-10 16:18:28 +02:00