This allows using some more nice annotations from 3.10
while still being compatible with even Python 3.8.
These are all annotation changes, should not modify any functionality.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
New Securesystemslib Keys can now be instantiated in two ways:
* deserialize via Key.from_dict() as before
* generate new keys via implementation specific methods
Fix all cases where we call Key() or Key.from_securesystemslib_key()
and use SSlibKey methods instead. Fix related tests.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
This commit contains 2 API changes in "Delegations" class from
tuf/api/metadata.py:
1. roles argment is made optional
2. unrecognized_fields argument becomes the 4-th rather than the 3-rd
as it used to be
In this commit, I add support for succinct_roles roles inside
Delegations class. This change is related to TAP 15 proposal.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Clarify explicitly that exactly one of "paths" and "path_hash_prefixes"
must be set inside DelegatedRole.
Also simplify the check for "paths" and "path_hash_prefixes".
Finally, add a test case inside the "test_metadata_serialization.py"
test file about wrong keyids type for "Role" serialization.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add SuccinctRoles class containing the information from the
succint_roles dict described in TAP 15.
This allows for easy mypy checks on the types, easy enforcement on
TAP 15 restrictions (as for example that "bit_length" must be between 1
and 32) and support for unrecognized fields inside succinct_roles
without much of a hassle.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The Document formats section (chapter 4) of the
specification says the following:
"All of the formats described below include the ability to add more
attribute-value fields to objects for backward-compatible format
changes. Implementers who encounter undefined attribute-value pairs in
the format must include the data when calculating hashes or verifying
signatures and must preserve the data when re-serializing."
I initially thought it's applicable only to the SIGNED fields as
"undefined attribute-value pairs in the format must include the data
when calculating hashes or verifying signatures"
This doesn't mean that the sentence before that excludes "Metadata" as a
possible place for additional fields.
The other maintainers agreed with me and we are going to add support for
'unrecognized_fields" inside "Metadata".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Replace the usage of Metadata.to_dict inside
test_valid_metadata_serialization and instead use Metadata.to_bytes()
in order to test that the full serialization cycle is working as
expected:
Metadata.from_bytes -> Metadata.to_bytes
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
There is no need to copy "case_dict" inside serialization test
functions in test_metadata_serialization.py when we are testing
invalid arguments.
These dictionaries are not be used after calling "from_dict" and
it doesn't matter if they are empty afterward.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Move the duplicating signatures tests from test_metadata_base function
in test_api.py into test_metadata_serialization.py.
This is a more logical place to store this test case as
test_metadata_base is actually focused on testing
Metadata.signed.is_expired.
That also is the reason why I renamed test_metadata_base to
test_metadata_signed_is_expired.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Most of the test_invalid_signed_serialization subtests are currently
failing because "_type": "signed" and then the test tries to deserialize
them as Snapshot (which fails a type check).
Correct the type to "snapshot" so that we can fail in the correct places
during serialization instead.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
All TUF implementations used to use "1.0" as the spec version and most
of them have never modified that value since.
Accept two-part spec_version for legacy compatibility: it is strictly
speaking against the current spec (which requires semver) but there
should be no harm in doing this and it allows us to deserialize
metadata generated by e.g. go-tuf.
Fixes#1751
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
keyids are ordered in the data we deserialize: Not preserving that order
breaks canonicalization. Set does not preserve order.
Change Role.keyids type from Set to List. This is strictly speaking
an API change but a minor one: keyids are supposed to be changed
via add_key()/remove_key().
Add tests for this for both Role and DelegatedRole. Shorten a related
exception message.
Fix#1752
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This commit includes manual fixes for a lot of mypy warnings.
When there were warnings that we are calling non-annotated function
in annotated context I decided to add annotations instead of ignoring
those warnings.
That's how I end up adding annotations in the whole tests/utils.py
module.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In test_metadata_serialization.py "test_case_data" is actually a string
when the decorator calls the actual test functions.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
- Add the check for empty strings in the Delegate Role name
- Remove the comprehensive lists to make the code more readable
- Remove the test for empty file name from
``test_updater_with_simulator``
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
- Reuse the dataset and the existing tests
- Fix the keyids in the tests datasets to be aligned
- Fix the ``ValueError`` message aligned to the existent messages
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
This commit adds the validation in the ``metadata.Delegations``
to prevent that one of the delegate role names given is a top-level
role name.
A ``ValueError`` will be raised if one of the roles names in the
list given to as delegated contains the role name as one of the
top-level roles.
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Add sanity types checks on Targets delegation paths and
path_hash_prefixes making sure that they are strings.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Address or disable pylint warnings raised on all test files inside
the "tests/" directory testing the code of the new implementation.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Validate that root role names are 4 and that they are exactly
"root", "snapshot", "targets" and "timestamp" as described in
the spec:
https://theupdateframework.github.io/specification/latest/#root-role
Additionally, fix the valid_roots dataset, so each of the cases contains
the top metadata role names inside the roles dictionary.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Generalize the decorator used in test_metadata_serialization.py and
move it inside tests/utils.py, so it can be reused in other similar
situations.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Move the Delegation class serialization tests from "test_api.py"
to test_metadata_serialization.py module focused on serialization
testing.
Additionally, a test for empty keys and roles will be added in my
upcomming pr #1511.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The spec does not say anything about role name uniqueness in a
delegations object, but I believe we cannot safely allow multiple roles
with the same role name in the roles array of a delegations object.
If we did then the roles could have different keyids, and then we would
end up in a situation where metadata may be both a valid delegation
and an invalid delegation at the same time, depending on how the role
gets chosen and that does not seem like the intention of the design.
There is an issue open in the specification with number 167 about
that issue.
Regardless of the Metadata API, I think we should enforce role name
uniqueness.
I chose to change the data structure containing roles to
OrderedDict, where keys are role names and values are DelegatedRole
instances.
This made sense to me as role names are the unique identifier of a role
and their order is important to the way they are traversed afterward.
Note: we can't use OrderedDict as type annotation until we drop support
for Python 3.6:
https://docs.python.org/3/library/typing.html#typing.OrderedDict
That's why I used quotes around "OrderedDict" annotation, because I
can't import it.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Currently, TargetFile instances do not contain the path relative URL of
the file they represent. The API itself does not need it but it could be
useful for users of the API.
As an example, the current client returns a dict for
get_one_valid_targetinfo(): that dict contains a filepath field and
a targetinfo field (essentially TargetFile).
We would like to keep a similar API, but avoid hand-crafted dicts.
It would be much nicer to return a TargetFile that would contain the
full "metadata" of the targetfile.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Those tests are needed to cover use cases when syntatcticly as
standalone objects the metadata classes and their helper classes defined
in tuf/api/metadata.py are valid even if they cannot be verified.
An example where an object is valid, but cannot be verified is
if we have a Role instance with an empty list of "keyids".
This instance is valid and can be created, but cannot be verified
because there is a requirement that the threshold should be above
1, meaning that there should be at least 1 element inside the "keyids"
list to complete successful threshold verification.
The situation is the same for the rest of the tests I am adding to this
commit:
- Root object without keys
- Root object without roles
- DelegationRole object with empty "keyids"
- DelegationRole object with an empty list of "paths"
- DelegationRole object with an empty list of "path_hash_prefixes"
all of these objects can be instantiated, but cannot complete
successfully threshold verification.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Test metadata (de)serialization with input data containing containers
with zero or more elements.
Here is the status for the different use cases:
Root keys:
- many keys: added
Root roles:
- many roles: added
Root role keyids:
- many keids: already added in https://github.com/theupdateframework/tuf/pull/1481
MetaFile hashes:
- many hashes: already tested
- zero hashes: added. Testing as invalid test case.
Timestamp meta:
- zero elements: already tested
- many elements: added
Snapshot meta:
- zero items: added
- many items: added
Delegation keys:
- many keys: added
Delegation role keyids:
- many keyids: added
Delegation role paths:
- many paths: already tested
Delegation role path_hash_prefixes:
- many path_hash_path_prefixes: already tested
Delegation roles:
- zero roles: added
- multiple roles: added
Targets targets:
- zero items: already tested
- multiple items: added
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The specification does not state clearly what is the
behaviour when none of delegation's "paths" and
"path_hash_prefixes" is set. See #1497.
Until this issue is clarified, copy current
Updater which raises an error in such case.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
We made Role.keyids a set because the keyids are supposed
to be unique and this still makes sense.
However, the data should also preserve order
(when deserialized and serialized) and currently, it does not.
This is fairly serious since writing signed data potentially modifies
the data (making the signature invalid).
The simplest solution (as proposed by Teodora) is to sort the
set during serialization and that would ensure the order of the items.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
A while ago we decided that it's best to research each of the individuals
attributes one by one and identify what level of validation it needs
compared to how we use it:
https://github.com/theupdateframework/tuf/pull/1366#issuecomment-829288790.
This work is ongoing and there are a couple of commits already merged
for this:
- 6c5d970799
- f20664d2fc
- 41afb1e134
We want to be able to test the attributes validation against known bad
values.
The way we want to do that is with table testing we have added
using decorators for our metadata classes defined in New API:
https://github.com/theupdateframework/tuf/pull/1416.
This gives us an easy way to add new cases for each of the attributes and
not depend on external files.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We have merged ADR 8 allowing for unrecognized fields and we have
added tests for that which are too specific and not scalable.
Now, I use table testing which we have used initially in https://github.com/theupdateframework/tuf/pull/1416
to test unrecognized fields support in a cleaner and much more readable
way.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The idea of this commit is to separate (de)serialization testing outside
test_api.py and make sure we are testing from_dict/to_dict for all
possible valid data for all classes.
Jussi in his comment here:
https://github.com/theupdateframework/tuf/issues/1391#issuecomment-849390669
proposed using decorators when creating comprehensive testing
for metadata serialization.
The main problems he pointed out is that:
1) there is a lot of code needed to generate the data for each case
2) the test implementation scales badly when you want to add new
cases for your tests, then you would have to add code as well
3) the dictionary format is not visible - we are loading external files
and assuming they are not changed and valid
In this change, I am using a decorator with an argument that complicates
the implementation of the decorator and requires three nested functions,
but the advantages are that we are resolving the above three problems:
1) we don't need new code when adding a new test case
2) a small amount of hardcoded data is required for each new test
3) the dictionaries are all in the test module without the need of
creating new directories and copying data.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
