According to the recently updated version of the specification the shell
style wildcard matching is glob-like (see https://github.com/theupdateframework/specification/pull/174),
and therefore a path separator in a path should not be matched by a
wildcard in the PATHPATTERN.
That's not what happens with `fnmatch.fnmatch()` which doesn't
see "/" separator as a special symbol.
For example: fnmatch.fnmatch("targets/foo.tgz", "*.tgz") will return
True which is not what glob-like implementation will do.
We should make sure that target_path and the pathpattern contain the
same number of directories and because each part of the pathpattern
could include a glob pattern we should check that fnmatch.fnmatch() is
true on each target and pathpattern directory fragment separated by "/".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
After the addition of "path" argument in the TargetFile class the
filename argument in Targets.update() became redundant.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Currently, TargetFile instances do not contain the path relative URL of
the file they represent. The API itself does not need it but it could be
useful for users of the API.
As an example, the current client returns a dict for
get_one_valid_targetinfo(): that dict contains a filepath field and
a targetinfo field (essentially TargetFile).
We would like to keep a similar API, but avoid hand-crafted dicts.
It would be much nicer to return a TargetFile that would contain the
full "metadata" of the targetfile.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The securesystemslib key dictionary representation includes
the private key in keyval. TUF key doesn't handle it in any way,
but considering that we allow unrecognized symbols in the format,
we should exclude the private key otherwise this could lead to
misuse.
A call to securesystemslib.keys.format_keyval_to_metadata
with the default private=False would do exactly that.
Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
I was looking at how can we simplify or split test_api.py when I noticed
that the test cases covered by those two test functions are already
covered in the test_metadata_serialization.py module in the
"invalid_keys" and "invalid_roles" datasets.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Those tests are needed to cover use cases when syntatcticly as
standalone objects the metadata classes and their helper classes defined
in tuf/api/metadata.py are valid even if they cannot be verified.
An example where an object is valid, but cannot be verified is
if we have a Role instance with an empty list of "keyids".
This instance is valid and can be created, but cannot be verified
because there is a requirement that the threshold should be above
1, meaning that there should be at least 1 element inside the "keyids"
list to complete successful threshold verification.
The situation is the same for the rest of the tests I am adding to this
commit:
- Root object without keys
- Root object without roles
- DelegationRole object with empty "keyids"
- DelegationRole object with an empty list of "paths"
- DelegationRole object with an empty list of "path_hash_prefixes"
all of these objects can be instantiated, but cannot complete
successfully threshold verification.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Test metadata (de)serialization with input data containing containers
with zero or more elements.
Here is the status for the different use cases:
Root keys:
- many keys: added
Root roles:
- many roles: added
Root role keyids:
- many keids: already added in https://github.com/theupdateframework/tuf/pull/1481
MetaFile hashes:
- many hashes: already tested
- zero hashes: added. Testing as invalid test case.
Timestamp meta:
- zero elements: already tested
- many elements: added
Snapshot meta:
- zero items: added
- many items: added
Delegation keys:
- many keys: added
Delegation role keyids:
- many keyids: added
Delegation role paths:
- many paths: already tested
Delegation role path_hash_prefixes:
- many path_hash_path_prefixes: already tested
Delegation roles:
- zero roles: added
- multiple roles: added
Targets targets:
- zero items: already tested
- multiple items: added
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Spec does not explicitly say so but the intent is that a snapshot
metadata can be trusted for rollback protection checks of newer
snapshots even if current snapshot version does not match the version
in current timestamp meta.
Only do the snapshot version check for the "final" snapshot by doing it
when targets is updated.
Improve test names and comments.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
While this is not explicitly said in the spec, the intention is that
expired timestamp and snapshot should be used for rollback protection
checks on newer timestamp/snapshot (but not for anything else).
Move the expiry checks to the "next" metadata update: timestamp expiry
is checked when snapshot is loaded, and snapshot expiry is checked
when targets is loaded.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
The usefulness was debatable to begin with, and now that it has become
clear that rollback protection requires a second "final verification"
step for all three root, timestamp and snapshot it is clear that
root_update_finished() is not good design.
update_root() still accepts expired root metadata but now the final
root expiry is checked when the "next" metadata (timestamp) is loaded.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Add support for prefixing targets with their hashes when downloading or
using HASH.FILENAME.EXT as target names.
The introduction of prefix_targets_with_hash was necessary, because
there are use cases like Warehouse where you could use
consistent_snapshot, but without adding a hash prefix to your targets.
When prefix_targets_with_hash is set to True, target files conforming
the format HASH.FILENAME.EXT will be downloaded from the server, but
they will be saved on the client side without their hash prefixes or
FILENAME.EXT.
This makes sure the client won't understand the usage of
prefix_targets_with_hash.
Still, if you want to use HASH.FILENAME.EXT as target names when
downloading, then additionally you need to provide consistent_snapshot
set to True in your root.json. The reason is that the specification uses
consistent_snapshot for the same purpose:
"If consistent snapshots are not used (see § 6.2 Consistent snapshots),
then the filename used to download the target file is of the fixed form
FILENAME.EXT (e.g., foobar.tar.gz). Otherwise, the filename is of the
form HASH.FILENAME.EXT
(e.g., c14aeb4ac9f4a8fc0d83d12482b9197452f6adf3eb710e3b1e2b79e8d14cb681.foobar.tar.gz),
where HASH is one of the hashes of the targets file listed in the
targets metadata file found earlier in step § 5.6 Update the targets role.
In either case, the client MUST write the file to non-volatile
storage as FILENAME.EXT."
The same behavior of using two flags is used in the legacy code when
calling tuf.client.updater.download_target() in a repository using
prefix_targets_with_hash and consistent_snapshot.
See chapter 5.7.3:
https://theupdateframework.github.io/specification/latest/index.html#fetch-target
By default, prefix_targets_with_hash is set to true to make it easier
to the user to provide uniquely identifiable targets file names by
using consistent_snapshot set to True.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
When we use Metadata, it is helpful if the specific signed type (and all of
the signed types attribute types are correctly annotated. Currently this is
not possible.
Making Metadata Generic with constraint T, where
T = TypeVar("T", "Root", "Timestamp", "Snapshot", "Targets")
allows these annotations. Using Generic annotations is completely
optional so all existing code still works -- the changes in test code
are done to make IDE annotations more useful in the test code, not
because they are required.
Examples:
md = Metadata[Root].from_bytes(data)
md:Metadata[Root] = Metadata.from_bytes(data)
In both examples md.signed is now statically typed as "Root" allowing IDE
annotations and static type checking by mypy.
Note that it's not possible to validate that "data" actually contains a
root metadata at runtime in these examples as the annotations are _not_
visible at runtime at all: new constructors would have to be added for that.
from_file() is now a class method like from_bytes() to make sure both
have the same definition of "T" when from_file() calls from_bytes():
This makes mypy happy.
Partially fixes#1433
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
The specification does not state clearly what is the
behaviour when none of delegation's "paths" and
"path_hash_prefixes" is set. See #1497.
Until this issue is clarified, copy current
Updater which raises an error in such case.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Instead of using general abstract modification functions embed smaller
modification functions inside each test where it's needed and
create modify_metadata function that does all of the common stuff like:
- instantiating a metadata object
- calling the modification function
- signing the modified object
- serializing back to bytes.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The current situation with the TrustedMetadataSet testing is that
we don't have a mnimimal amount of unit tests testing the different
branches in the various API functionality in the class.
This commit proposes simple unit tests covering almost all of the
branches in the API functions and increasing the unit test coverage
(as reported from the "coverage" tool) from 74 % to 97 %.
The code could be complicated at places, because the different
branches in the update_* functions depend on other metadata classes
as well.
Still, I hope we can find a way and simplify the code.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Metadata.to_bytes() is missing from the API and that is now becoming
annoying when writing the tests.
I think it makes sense to add:
it'll complete the serializing counterparts to from_bytes()/from_file().
We can also reuse to_bytes() in to_file() and that way ensure we don't
import the JSONSerializer locally twice.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Updated/removed documented commands and comments which were referencing Python2. Also updated links to documentation referencing Python2 docs (unchanged where needed)
Signed-off-by: Samuel Gregorovic <samuelgregorovic@gmail.com>
Signed-off-by: samuelgregorovic <samuelgregorovic@gmail.com>
We made Role.keyids a set because the keyids are supposed
to be unique and this still makes sense.
However, the data should also preserve order
(when deserialized and serialized) and currently, it does not.
This is fairly serious since writing signed data potentially modifies
the data (making the signature invalid).
The simplest solution (as proposed by Teodora) is to sort the
set during serialization and that would ensure the order of the items.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
* Rename arguments so connection between the role name and the
metadata is stronger.
* Also add a comment on the list comprehension + next() trick.
* Add return value annotation
* Raise early if delegations is None to make the flow more obvious
(and modify test case so we have coverage for the new case)
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
A while ago we decided that it's best to research each of the individuals
attributes one by one and identify what level of validation it needs
compared to how we use it:
https://github.com/theupdateframework/tuf/pull/1366#issuecomment-829288790.
This work is ongoing and there are a couple of commits already merged
for this:
- 6c5d970799
- f20664d2fc
- 41afb1e134
We want to be able to test the attributes validation against known bad
values.
The way we want to do that is with table testing we have added
using decorators for our metadata classes defined in New API:
https://github.com/theupdateframework/tuf/pull/1416.
This gives us an easy way to add new cases for each of the attributes and
not depend on external files.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Make sure verify_delegate() succeeds when threshold is reached even if
some signatures fail to verify.
Make sure higher threshold (2/2) works.
Change error type for "Call is valid only on delegator metadata" error.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
The delegating Metadata (root or targets) verifies that the delegated
metadata is signed by required threshold of keys for the delegated
role.
Calling the function on non-delegator-metadata or giving a rolename
that is not actually delegated by the delegator is considered a
programming error and ValueError is raised.
If the threshold is not reached, UnsignedMetadataError is raised.
Tweak type annotation of Delegations.keys to match the one for
Root.keys (so they can be assigned to same local variable).
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
TrustedMetadataSet is a long name but
* it better describes the main feature
* the name isn't used in too many places
Change the variable names "bundle" -> "trusted_set"
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
We have merged ADR 8 allowing for unrecognized fields and we have
added tests for that which are too specific and not scalable.
Now, I use table testing which we have used initially in https://github.com/theupdateframework/tuf/pull/1416
to test unrecognized fields support in a cleaner and much more readable
way.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We've been returning Signature objects since 49aa0fc167.
Also add a test case that does something with the returned signature.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
