This is useful for those who want to use the default fetcher
but modify some attributes
The file itself could be moved to tuf/ngclient/ but this is not done yet
as sigstore-python is using this internal module. Move can be done once
sigstore-python 1.0 is no longer relevant.
Fixes#2268
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
verify_delegate() unfortunately needs an almost complete rewrite
as the Key.verify_signature() API change affects it quite a bit.
Refactoring the role and key lookup into a separate method makes the
code readable again.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Key.verify_signature() API has changed:
* argument is bytes, not metadata
* raised error now comes from securesystemslib
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
New Securesystemslib Keys can now be instantiated in two ways:
* deserialize via Key.from_dict() as before
* generate new keys via implementation specific methods
Fix all cases where we call Key() or Key.from_securesystemslib_key()
and use SSlibKey methods instead. Fix related tests.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
If role is delegated but missing from snapshot, we currently raise a
undocumented KeyError: a generic RepositoryError seems better as callers
are expected to handle it (and adding a more specific error seems
useless as this is a repository software bug, not just expired metadata or
something).
The same check is also done later in TrustedMetadataSet but I think
keeping the check in both is clearest.
Fixes#2195
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Bump the supported specification version to 1.0.31 and additionally
update the generated test metadata as it has to be up to date with the
latest changes.
The new changes in the specification version 1.0.31 clarify the
requirement for the new root version as compared to the old root version
in step 5.3.5:
https://theupdateframework.github.io/specification/latest/#update-root
We already do what the specification suggests in the new changes, so
no other changes are required.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
* move to the test file that contains all the other download tests
* don't write 1000 files: it can be slow in CI
* Compare file content to what was originally written
(also read the whole file content)
* Remove try-except that seems unused
Signed-off-by: Jussi Kukkonen <jku@goto.fi>
If we have a goal of making metadata that can just be served to clients,
then these (and the parent metadata) should have versioned filenames.
Change the file names of the delegated files in the
hashed_bin_delegation.py to versioned.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add a basic example script showing all features of the succinct hash bin
delegations and the available API calls of SuccinctRoles.
The explanations are used to promote the usage of succinct hash bin
delegations by explaining it well enough so our users can understand
the API limitations and how to use them and at the same time I tried not
going into too many details of the SuccinctRoles math as its
implementation is inside tuf/api/metadata.py and there there are
explanations about that.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add test downloading a target file when succonct_roles is used and as
such test the whole updater downloading workflow.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add a test case when there is a bin name with the desired prefix, but
which cannot be cast to a hexadecimal number.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Bump the supported specification version to 1.0.30 and additionally
update the generated test metadata as it has to be up to date with the
latest changes.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In the spec version 1.0.30, a new change has been added considering what
should happen if there is a new timestamp with the same version.
It says the following:
"In case they [versions] are equal, discard the new
timestamp metadata and abort the update cycle.
This is normal and it shouldn't raise any error."
In other words, if there is a new timestamp with the same version, then
stop the update process and use the old timestamp.
Those changes reflect these latest specification modifications.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Test traversing the delegation tree when there is a Targets using a
delegation with succinct roles.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add support for Targets using delegation with succinct_roles.
For that purpose, we needed a method that can add succinct_roles
information with its all corresponding bins to the target metadata
and self.md_delegates attribute in RepositorySimulator.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Here is the list of all breaking API changes:
1) The "role" and "key" arguments in "Root.add_key()" are in reverse
order - "key" becomes first and "role" second.
2) "Root.remove_key()" has been renamed to "Root.revoke_key()".
3) The "role" and "keyid" arguments in "Root.revoke_key()" are in
reverse order - "keyid" becomes first and "role" second.
4) The "role" and "key" arguments in "Targets.add_key()" are in reverse
order - "key" becomes first and "role" second.
5) "Targets.remove_key()" has been renamed to "Targets.revoke_key()".
6) The "role" and "keyid" arguments in "Targets.revoke_key()" are in
reverse order - "keyid" becomes first and "role" second.
7) In both methods "Targets.add_key()" and "Targets.revoke_key()" the
"role" argument becomes an optional with a default value of None.
Those changes are made in an effort to make those methods logical
for both cases when standard roles and succinct_roles are used.
The "Root" API change was done in order to preserve naming and argument
order consistency with "Targets" API.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit contains 2 API changes in "Delegations" class from
tuf/api/metadata.py:
1. roles argment is made optional
2. unrecognized_fields argument becomes the 4-th rather than the 3-rd
as it used to be
In this commit, I add support for succinct_roles roles inside
Delegations class. This change is related to TAP 15 proposal.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Clarify explicitly that exactly one of "paths" and "path_hash_prefixes"
must be set inside DelegatedRole.
Also simplify the check for "paths" and "path_hash_prefixes".
Finally, add a test case inside the "test_metadata_serialization.py"
test file about wrong keyids type for "Role" serialization.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add two helper methods in SuccinctRoles.
Those methods proved useful in the testing code, but I believe they have
a potential value for production code as well.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add SuccinctRoles class containing the information from the
succint_roles dict described in TAP 15.
This allows for easy mypy checks on the types, easy enforcement on
TAP 15 restrictions (as for example that "bit_length" must be between 1
and 32) and support for unrecognized fields inside succinct_roles
without much of a hassle.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
There is a lot of repetitive code inside test_metadata_eq_.py.
Remove it by using the decorator.
I am initializing the object instances in setUpClass instead of doing it
inside the test function in order to escape the need for
reinitialization of the instances on each attribute.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Inside test_metadata_eq_.py we test the __eq__ implementations of all
classes. In order to do this, we change the attribute of the object and
then compare them to the unchanged version of those objects.
Usually, we do it in the following steps:
1. create an initial version "a"
2. create a copy of "a" called "b"
3. iterate all attributes inside "b" and change them to a given value
4. check that "a" and "b" are different
We do however forget to restore the object `b` to its initial state
which means we don't check the `__eq__` correctly as we stop on the
first, the found difference which could be of an older attribute changed
in one of the past iterations.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Test was supposed to test a threshold that is higher than number of
signatures, but it actually was just using completely unsigned metadata.
This still doesn't test the case where _trusted_ metadata defines a
threshold that new metadata does not reach: only the case where new
metadata defines threshold that it does not meet (this case is covered
in updater tests though).
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This change adds tests coverage for `path_hash_prefixes` and
verifies that role names matching specific prefixed successfully
find and download the corresponding metadata files
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
I created a new script called "generate_md.py" which can be used
to easily generate a repository. Additionally, I created a new
test file making sure that the locally stored metadata files and
the newly generated metadata roles are the same.
This will allow us to test that we are not changing the metadata
file structure when making changes.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This change improves the logic of expired metadata tests, so that
it is explicitly visible what the expiry time and the versions are
and when update/refresh is called in that period
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change fixes the expired metadata tests to mock `datetime`
as previously they mocked `time` incorrectly, which did not affect
update methods, as they use `datetime.datetime.utcnow()` to
calculate now
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change verifies that when local metadata has expired, it is
still used to verify new metadata that's pulled from remote
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This tests that an expired timestamp/snapshot/targets when loaded
from cache is not stored as final but is used to verify the new
timestamp
Fixes#1681
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This test covers `targetinfo`, `target_path`, `target_base_url`,
`metadata_dir` and `filepath` input validation of the `Updater`
methods
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
After we have dropped OrderedDict in e3b267e2e0
we are relying on python3.7+ default behavior to preserve the insertion
order, but there is one caveat.
When comparing dictionaries the order is still irrelevant compared to
OrderedDict. For example:
>>> OrderedDict([(1,1), (2,2)]) == OrderedDict([(2,2), (1,1)])
False
>>> dict([(1,1), (2,2)]) == dict([(2,2), (1,1)])
True
There are two special attributes, defined in the specification, where
the order makes a difference when comparing two objects:
- Metadata.signatures
- Targets.delegations.roles.
We want to make sure that the order in those two cases makes a
difference when comparing two objects and that's why those changes
are required inside two __eq__ implementations.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Test the "__eq__" implementation for all classes defined in
tuf/api/metadata.py
The tests are many but simple. The idea is to test each of the metadata
classes one by one and with this to make sure there are no possible
cases missed.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The Document formats section (chapter 4) of the
specification says the following:
"All of the formats described below include the ability to add more
attribute-value fields to objects for backward-compatible format
changes. Implementers who encounter undefined attribute-value pairs in
the format must include the data when calculating hashes or verifying
signatures and must preserve the data when re-serializing."
I initially thought it's applicable only to the SIGNED fields as
"undefined attribute-value pairs in the format must include the data
when calculating hashes or verifying signatures"
This doesn't mean that the sentence before that excludes "Metadata" as a
possible place for additional fields.
The other maintainers agreed with me and we are going to add support for
'unrecognized_fields" inside "Metadata".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Replace the usage of Metadata.to_dict inside
test_valid_metadata_serialization and instead use Metadata.to_bytes()
in order to test that the full serialization cycle is working as
expected:
Metadata.from_bytes -> Metadata.to_bytes
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
There is no need to copy "case_dict" inside serialization test
functions in test_metadata_serialization.py when we are testing
invalid arguments.
These dictionaries are not be used after calling "from_dict" and
it doesn't matter if they are empty afterward.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Move the duplicating signatures tests from test_metadata_base function
in test_api.py into test_metadata_serialization.py.
This is a more logical place to store this test case as
test_metadata_base is actually focused on testing
Metadata.signed.is_expired.
That also is the reason why I renamed test_metadata_base to
test_metadata_signed_is_expired.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
If a securesystemslib.FormatError is raised inside
Key.from_securesystemslib_key() then reraise ValueError.
This is done so that our users don't have to import securesystemslib
in order to handle the error and because the securesystemslib error
itself is securesystemslib implementation-specific.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
I've not supported many renames but I'm suggesting this one:
FetcherHTTPError was created because we needed to signal 403/404
from the fetcher to updater. At that time the download error hierarchy
in general was not thought out.
Now we have a couple of different errors all derived from
DownloadError. I believe it does not make sense to point out "Fetcher"
in one of their names: DownloadHTTPError makes it clearer this is a
specific type of DownloadError.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Fetcher interface should only raise DownloadErrors,
regardless of the implementation.
* Make sure fetch() wraps non-DownloadError errors in a DownloadError
* Make the abstract function private _fetch()
* Try to be more consistent in doscstrings
This now makes the example client more sensible (when server does not
respond):
$ ./client_example.py download qwerty
...
Failed to download target qwerty: Failed to download url http://127.0.0.1:8000/metadata/2.root.json
(here the latter part of the error string comes from DownloadError
raised by FetcherInterface.fetch())
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
- Update linter config to no longer distinguish between legacy
and new implementation. This requires addressing a linter warning
in an until now not linted module (tuf/__init__.py).
- Remove obsolete rules in MANIFEST.in (source distribution) and
tests/.coveragerc (test coverage).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove tests for legacy client, repository/developer tool and
command line tools, which will be removed in subsequent commits.
This commits also removes obsolete test tooling:
- Regarding simple_https_server + test certificates -- http/https
is no longer handled by tuf client directly but transparently by
the underlying requests module used by the default fetcher
implementation.
- For details about unittest_toolbox see #1792
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Old tests are not touched as they will be removed in subsequent
commits, along with the custom log module.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove metadata generation scripts based on legacy
repository/developer tools, which will be removed in subsequent
commits.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Most of the test_invalid_signed_serialization subtests are currently
failing because "_type": "signed" and then the test tries to deserialize
them as Snapshot (which fails a type check).
Correct the type to "snapshot" so that we can fail in the correct places
during serialization instead.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
All TUF implementations used to use "1.0" as the spec version and most
of them have never modified that value since.
Accept two-part spec_version for legacy compatibility: it is strictly
speaking against the current spec (which requires semver) but there
should be no harm in doing this and it allows us to deserialize
metadata generated by e.g. go-tuf.
Fixes#1751
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This commit tries to deal with two interests:
* metadata is highly repetitive and compressible: allowing compression
would be good
* there may be broken web servers (see
404838abcc/src/pip/_internal/download.py (L842))
that have problems with compression on already compressed target files
We can make things better for that first interest while we have no real
data for the second interest -- our current workarounds to avoid
compression are based on hearsay, not testing.
Now that individual fetchers are possible I suggest we simplify
ngclient and allow compression. As an example the pip Fetcher
could still use the pip response chunking code with all their
workarounds -- pip certainly has better capability to maintain
a mountain of workarounds and also has endless amounts of real-world
testing compared to python-tuf.
Details:
* Stop modifying Accept-Encoding (Requests default includes gzip)
* Don't use response.raw in RequestsFetcher as there is no need:
This was a workaround for false "Content-encoding: gzip" inserted by
a broken server -- and the workaround was only possible because we
knew we never asked for compression
* Fix issue in test_session_get_timeout(): it's not mocking the error
that requests really raises in this case
Fixes#1251
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Update new test modules to stop using unittest_toolbox, in
preparation for its removal in #1790.
The tools provided by unittest_toolbox can easily (in a more
obvious way) be replaced by using the standard library modules
`tempfile` and `random` (no more used) directly.
In the case of tempdir and -file creation/removal, skipping the use
of unittest_toolbox, which does this by default, also uncovers some
test cleanup failures, which would occur when temporary test
directories were removed while a test server hadn't released them.
(see `except OSError: pass` in unittest_toolbox's `tearDown`
method)
**Change details**
**test_fetcher_ng.py:**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs. -Move now manual creation of an exemplary targets file to
setUpClass, as the same file is used by all tests. And remove it
explicitly in tearDownClass after killing the server (see note
about failure above). - Trigger URL parsing error with a hardcoded
invalid URL string instead of a random string.
**test_updater_ng.py**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs.
- Explicitly create tmp test dirs in setUp, but don't remove
them in tearDown to avoid above mentioned failures. They will be
removed all at once when removing the tmp root test dir in
tearDownClass
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Define TESTS_DIR constant in tests/util.py as full path to the
parent directory of the util module. This may be used to reliably
read other files in tests dir, such es "repository_data" or
"simple_server", regardless of cwd.
This commit also replaces a couple of `getcwd() + "filename"` with
`TESTS_DIR + filename`, so that in the future (post #1790) we
should be able to invoke the tests from anywhere, not only from
within the tests directory as is now the case.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
After we drop support for python3.6 we can relly that dictionaries
preserve the insertion order:
https://docs.python.org/3.7/whatsnew/3.7.html
This means we can replace the usage of OrderedDict with a standard
dictionaries.
Something we have to keep in mind is that even thought the insertion
order is preserved the equality comparison for normal dicts is
insensitive for normal dicts compared to OrderedDict
For example:
>>> OrderedDict([(1,1), (2,2)]) == OrderedDict([(2,2), (1,1)])
False
>>> dict([(1,1), (2,2)]) == dict([(2,2), (1,1)])
True
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
LengthOrHashMismatchError is a thrown when there are problems with
metadata verification or problems from the repository side when looking
it from the user's perspective.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
UnsupportedAlgorithmError is a detailed securesystemslib exception
and there is no need for TUF to redefine it.
Moreover which hash "algorithms" are allowed is work for
securesystemslib not for TUF.
It's only used once inside "Targetfile.from_data()" and there it's used
to denote that there is a problem with the given argument.
That's why this error can be just replaced with "ValueError".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
ReplayedMetadataError is a subset of
BadVersionNumberError and in a discussion with
Jussi we realized that ReplayedMetadataError can
be replaced by BadVersionNumberError with a
good message.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
URLParsingError is a specific download error and
is not clear what benefit it provides.
It's used only once in the new code and the
message says everything you need to know about
the exception.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add tuf/api/exceptions.py for exceptions in the new code.
I copied the exceptions from tuf/exceptions.py with a few important
decisions:
1. I only added the exceptions that are used in the new code
2. I removed the general "Error" class as we can directly inherit
Exceptions
3. I tried grouping the exceptions by relevance
4. I removed the second argument "UnsignedMetadataError" as it's only
kept for backward compatibility and is not used
5. I tried following the new code style guidelines and linted the file
with our linters.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit explicitly encodes role names. Mostly this encoding is already
happening in ``requests`` for what is not a URL.
The "/" in a role name will now be encoded.
Also, a slight change in the RepositorySimulator will align with the tests.
This commit partially covers issue #1634
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Add a method to rotate roles keys into RepositorySimulator (only
top-level roles are supported for now). Rotation is used in four
places already and this refactoring makes the tests easier to
understand.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
When calling updater._persist_metadata() there is a possibility that
writing the temporary file to storage can succeed, but moving it with
os.replace could fail with OSError.
Make sure we are removing the newly created temporary file in that case.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Remove `bump_version()` method, which is just an alias for "+= 1"
on the version attribute. For a slim low-level API it seems okay to
just directly access/modify the attribute.
The extra level of abstraction of "bumping a version" is more
appropriate for a repository library (see #1136).
This patch also removes a related unit test and updates another one
to directly do `(...).version +=`.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
New pylint warnings appeared related to changes
in urlib3:
- tests/test_fetcher_ng.py:128: error: Argument 1 to "ReadTimeoutError"
has incompatible type "None"; expected "ConnectionPool" [arg-type]
- tests/test_fetcher_ng.py:128: error: Argument 2 to "ReadTimeoutError"
has incompatible type "None"; expected "str" [arg-type]
I noticed these error in this CI run:
https://github.com/theupdateframework/python-tuf/runs/4764931441?check_suite_focus=true
I fixed them by creating a urllib3.HTTPConnectionPool() instance as
the first argument and replaced the second argument with an empty
string.
This seems to do the job.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
After making a successful update of valid metadata which stores it
in cache and performing a second update with a new updater while
the metadata is already stored in cache, this test verifies that
timestamp, snaphot and targets are loaded from cache and not
downloaded
Fixes#1681
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
