According to the recently updated version of the specification the shell
style wildcard matching is glob-like (see https://github.com/theupdateframework/specification/pull/174),
and therefore a path separator in a path should not be matched by a
wildcard in the PATHPATTERN.
That's not what happens with `fnmatch.fnmatch()` which doesn't
see "/" separator as a special symbol.
For example: fnmatch.fnmatch("targets/foo.tgz", "*.tgz") will return
True which is not what glob-like implementation will do.
We should make sure that target_path and the pathpattern contain the
same number of directories and because each part of the pathpattern
could include a glob pattern we should check that fnmatch.fnmatch() is
true on each target and pathpattern directory fragment separated by "/".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
After the addition of "path" argument in the TargetFile class the
filename argument in Targets.update() became redundant.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Currently, TargetFile instances do not contain the path relative URL of
the file they represent. The API itself does not need it but it could be
useful for users of the API.
As an example, the current client returns a dict for
get_one_valid_targetinfo(): that dict contains a filepath field and
a targetinfo field (essentially TargetFile).
We would like to keep a similar API, but avoid hand-crafted dicts.
It would be much nicer to return a TargetFile that would contain the
full "metadata" of the targetfile.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The securesystemslib key dictionary representation includes
the private key in keyval. TUF key doesn't handle it in any way,
but considering that we allow unrecognized symbols in the format,
we should exclude the private key otherwise this could lead to
misuse.
A call to securesystemslib.keys.format_keyval_to_metadata
with the default private=False would do exactly that.
Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
I was looking at how can we simplify or split test_api.py when I noticed
that the test cases covered by those two test functions are already
covered in the test_metadata_serialization.py module in the
"invalid_keys" and "invalid_roles" datasets.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
When we use Metadata, it is helpful if the specific signed type (and all of
the signed types attribute types are correctly annotated. Currently this is
not possible.
Making Metadata Generic with constraint T, where
T = TypeVar("T", "Root", "Timestamp", "Snapshot", "Targets")
allows these annotations. Using Generic annotations is completely
optional so all existing code still works -- the changes in test code
are done to make IDE annotations more useful in the test code, not
because they are required.
Examples:
md = Metadata[Root].from_bytes(data)
md:Metadata[Root] = Metadata.from_bytes(data)
In both examples md.signed is now statically typed as "Root" allowing IDE
annotations and static type checking by mypy.
Note that it's not possible to validate that "data" actually contains a
root metadata at runtime in these examples as the annotations are _not_
visible at runtime at all: new constructors would have to be added for that.
from_file() is now a class method like from_bytes() to make sure both
have the same definition of "T" when from_file() calls from_bytes():
This makes mypy happy.
Partially fixes#1433
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Metadata.to_bytes() is missing from the API and that is now becoming
annoying when writing the tests.
I think it makes sense to add:
it'll complete the serializing counterparts to from_bytes()/from_file().
We can also reuse to_bytes() in to_file() and that way ensure we don't
import the JSONSerializer locally twice.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
* Rename arguments so connection between the role name and the
metadata is stronger.
* Also add a comment on the list comprehension + next() trick.
* Add return value annotation
* Raise early if delegations is None to make the flow more obvious
(and modify test case so we have coverage for the new case)
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Make sure verify_delegate() succeeds when threshold is reached even if
some signatures fail to verify.
Make sure higher threshold (2/2) works.
Change error type for "Call is valid only on delegator metadata" error.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
The delegating Metadata (root or targets) verifies that the delegated
metadata is signed by required threshold of keys for the delegated
role.
Calling the function on non-delegator-metadata or giving a rolename
that is not actually delegated by the delegator is considered a
programming error and ValueError is raised.
If the threshold is not reached, UnsignedMetadataError is raised.
Tweak type annotation of Delegations.keys to match the one for
Root.keys (so they can be assigned to same local variable).
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
We have merged ADR 8 allowing for unrecognized fields and we have
added tests for that which are too specific and not scalable.
Now, I use table testing which we have used initially in https://github.com/theupdateframework/tuf/pull/1416
to test unrecognized fields support in a cleaner and much more readable
way.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We've been returning Signature objects since 49aa0fc167.
Also add a test case that does something with the returned signature.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
The idea of this commit is to separate (de)serialization testing outside
test_api.py and make sure we are testing from_dict/to_dict for all
possible valid data for all classes.
Jussi in his comment here:
https://github.com/theupdateframework/tuf/issues/1391#issuecomment-849390669
proposed using decorators when creating comprehensive testing
for metadata serialization.
The main problems he pointed out is that:
1) there is a lot of code needed to generate the data for each case
2) the test implementation scales badly when you want to add new
cases for your tests, then you would have to add code as well
3) the dictionary format is not visible - we are loading external files
and assuming they are not changed and valid
In this change, I am using a decorator with an argument that complicates
the implementation of the decorator and requires three nested functions,
but the advantages are that we are resolving the above three problems:
1) we don't need new code when adding a new test case
2) a small amount of hardcoded data is required for each new test
3) the dictionaries are all in the test module without the need of
creating new directories and copying data.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Securesystemslib digest() and digest_fileobject()
calls raise sslib specific exceptions that need to be
handled and re-raised as TUF exceptions.
Updated tests in test_api.py accordingly.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Test unknown signature algorithm/scheme.
Also shorten the incorrect (but syntactically valid) signature a bit.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Aim to only raise UnsignedMetadataError from verify_signature().
Some of the situations could be things like UnsupportedAlgorithmError
-- where the underlying reason may be a missing dependency -- but it
seems impossible for a client to know whether it's that or whether it
is broken or malicious server side.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Clarify that we don't semantically validate "Key" instances during
initialization and that this is a responsibility of securesystemslib.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
store signatures in a Dict of keyid to Signature. This ensures
signature uniqueness. Raise in from_dict() if input contains multiple
different signatures for a keyid.
This changes Metadata object API, and makes it slightly different from
the file format: this is justified by making the API safer to use and
easier to validate.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This is likely not needed by users of the API (as they are interested
in the higher level functionality "verify delegate metadata with
threshold of signatures").
Moving verify to Key makes the API cleaner because including both
"verify myself" and "verify a delegate with threshold" can look awkward
in Metadata, and because the ugly Securesystemslib integration is now
Key class implementation detail (see Key.to_securesystemslib_key()).
Also raise on verify failure instead of returning false: this was found
to confuse API users (and was arguably not a pythonic way to handle it).
* Name the function verify_signature() to make it clear what is being
verified.
* Assume only one signature per keyid exists: see #1422
* Raise only UnsignedMetadataError (when no signatures or verify failure),
the remaining lower level errors will be handled in #1351
* Stop using a "keystore" in tests for the public keys: everything we
need is in metadata already
This changes API, but also should not be something API users want to
call in the future when "verify a delegate with threshold" exists.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This simplifies life for API users as usually a key needs its
identifier: this is already visible in how update() becomes simpler
in the API.
The downside is that 'from_dict()' now has two arguments (so arguably
the name is not great anymore but it still does _mostly_ the same job
as other from_dicts).
This is an API change, if a minor one.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This change is relevant to the new metadata class Targets.
In the specification, when describing the Targets metadata file format
and more precisely "TARGETPATH" (or targets containing the actual
target files) it's said:
"It is allowed to have a TARGETS object with no TARGETPATH elements.
This can be used to indicate that no target files are available."
If there is no "TARGETPATH" keys for the dictionary "targets", this
would mean that "Targets.targets" is {}.
Make sure we test for that.
See: https://theupdateframework.github.io/specification/latest/#targetpath
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We have tests which make sure we can use `Timestamp.update()` and
`Snapshot.update()` with MetaFile instance storing only version
(because length and hashes are optional).
Those tests were created to make sure that we are actually supporting
optional hashes and length when we call `update` for those classes, but
after we changed the `update()` signature to accept `MetaFile` instance
the tests are obsolete.
The reason is that length and hashes can be optional because of the
MetaFile implementation, no the update function itself and we have
other tests validating creating a MetaFie instance without hashes and
length.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Currently, when we call Targets/Snapshot/Timestamp.update() we are
passing all of the necessary values to create MetaFile/Targets File
respectively.
This is not needed, given that one of the reasons we have created
MetaFile and TargetFile is to make the API easier to use.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In the top-level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.
As written in the spec "targets" in "targets.json" has defined the
"custom" field serving the same purpose as "unrecognized_fields" in the
implementation.
That's why to conform against the spec and support "custom" and allow
"unrecognized_fields" everywhere where it's not sensitive we can define
custom as property which actually access data stored in
unrecognized_fields.
For context read ADR 8 in tuf/docs/adr.
Additionally, after adding the TargetFile class, when we create a
Targets an object we are now calling from dict twice - one for the main
Targets class and one for each of the complex attributes
TargetFile.from_dict() and Delegations.from_dict().
Given that the "from_dict" methods have the side effect of destroying
the given dictionary, we would need to start using deepcopy()
for our tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In the top-level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.
Additionally, after adding the MetaFile class, when we create an object
we are now calling from dict twice - one for the main class (Timestamp,
Snapshot) and one for the pacticular complex attribute -
MetaFile.from_dict(). Given that the "from_dict" methods have the
side effect of destroying the given dictionary, we would need to
start using deepcopy() for our tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
NOTE: making consistent_snapshot optional requires using a default value
for the argument in __init__ in Root and thus consistent_snapshot should
be rearranged in the end.
Read more: https://github.com/theupdateframework/tuf/pull/1394#issuecomment-842134961
From chapter 7 in the spec (version 1.0.17)
"Finally, the root metadata should write the Boolean
"consistent_snapshot" attribute at the root level of its keys of
attributes.
If consistent snapshots are not written by the repository,
then the attribute may either be left unspecified or be set to the
False value. Otherwise, it must be set to the True value."
We want to make sure we support repositories
without consistent_snapshot set.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Use either "if X is not None:" or a try-except instead of a "if X:".
I believe Targets.from_dict() was not really broken with previous code
but it looks suspicious and did fail the added test with a strange
exception: I expect the from_dict() methods to mainly fail with
KeyErrors, ValueErrors or AttributeErrors if file format structure
is incorrect.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
A DelegatedRole with paths=[] fails to serialize correctly (paths is not
included in the output json).
Fix the issue, modify tests to notice a regression.
Fixes#1389
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
After the implementation of a Key class representing
the public portion of a key, the method add_key() should
take an argument of type Key, instead of a dictionary.
Test cases are updated accordingly.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
In the top level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.
DelegatedRole shares a couple of fields with the Role class and that's
why it inherits it.
I decided to use a separate Delegations class because I thought it will
make it easier to read, verify and add additional helper functions.
Also, I tried to make sure that I test each level of the delegations
representation for support of storing unrecognized fields.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We should not do multiple lookups through data structures if one is
enough (here we have extra lookups on both roles and keyids).
Also in this case raising on missing key seems like the preferable
alternative so even a try-except is not needed.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
As per the specification (v1.0.1) length and hashes fields
in timestamp and snapshot metadata are optional.
We have implement this in the older API
(see https://github.com/theupdateframework/tuf/pull/1031) and we should
implement it in the new API.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
