The Document formats section (chapter 4) of the
specification says the following:
"All of the formats described below include the ability to add more
attribute-value fields to objects for backward-compatible format
changes. Implementers who encounter undefined attribute-value pairs in
the format must include the data when calculating hashes or verifying
signatures and must preserve the data when re-serializing."
I initially thought it's applicable only to the SIGNED fields as
"undefined attribute-value pairs in the format must include the data
when calculating hashes or verifying signatures"
This doesn't mean that the sentence before that excludes "Metadata" as a
possible place for additional fields.
The other maintainers agreed with me and we are going to add support for
'unrecognized_fields" inside "Metadata".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
There is no need to copy "case_dict" inside serialization test
functions in test_metadata_serialization.py when we are testing
invalid arguments.
These dictionaries are not be used after calling "from_dict" and
it doesn't matter if they are empty afterward.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Move the duplicating signatures tests from test_metadata_base function
in test_api.py into test_metadata_serialization.py.
This is a more logical place to store this test case as
test_metadata_base is actually focused on testing
Metadata.signed.is_expired.
That also is the reason why I renamed test_metadata_base to
test_metadata_signed_is_expired.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
If a securesystemslib.FormatError is raised inside
Key.from_securesystemslib_key() then reraise ValueError.
This is done so that our users don't have to import securesystemslib
in order to handle the error and because the securesystemslib error
itself is securesystemslib implementation-specific.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
I've not supported many renames but I'm suggesting this one:
FetcherHTTPError was created because we needed to signal 403/404
from the fetcher to updater. At that time the download error hierarchy
in general was not thought out.
Now we have a couple of different errors all derived from
DownloadError. I believe it does not make sense to point out "Fetcher"
in one of their names: DownloadHTTPError makes it clearer this is a
specific type of DownloadError.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Fetcher interface should only raise DownloadErrors,
regardless of the implementation.
* Make sure fetch() wraps non-DownloadError errors in a DownloadError
* Make the abstract function private _fetch()
* Try to be more consistent in doscstrings
This now makes the example client more sensible (when server does not
respond):
$ ./client_example.py download qwerty
...
Failed to download target qwerty: Failed to download url http://127.0.0.1:8000/metadata/2.root.json
(here the latter part of the error string comes from DownloadError
raised by FetcherInterface.fetch())
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
- Update linter config to no longer distinguish between legacy
and new implementation. This requires addressing a linter warning
in an until now not linted module (tuf/__init__.py).
- Remove obsolete rules in MANIFEST.in (source distribution) and
tests/.coveragerc (test coverage).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove tests for legacy client, repository/developer tool and
command line tools, which will be removed in subsequent commits.
This commits also removes obsolete test tooling:
- Regarding simple_https_server + test certificates -- http/https
is no longer handled by tuf client directly but transparently by
the underlying requests module used by the default fetcher
implementation.
- For details about unittest_toolbox see #1792
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Old tests are not touched as they will be removed in subsequent
commits, along with the custom log module.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove metadata generation scripts based on legacy
repository/developer tools, which will be removed in subsequent
commits.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Most of the test_invalid_signed_serialization subtests are currently
failing because "_type": "signed" and then the test tries to deserialize
them as Snapshot (which fails a type check).
Correct the type to "snapshot" so that we can fail in the correct places
during serialization instead.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
All TUF implementations used to use "1.0" as the spec version and most
of them have never modified that value since.
Accept two-part spec_version for legacy compatibility: it is strictly
speaking against the current spec (which requires semver) but there
should be no harm in doing this and it allows us to deserialize
metadata generated by e.g. go-tuf.
Fixes#1751
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This commit tries to deal with two interests:
* metadata is highly repetitive and compressible: allowing compression
would be good
* there may be broken web servers (see
404838abcc/src/pip/_internal/download.py (L842))
that have problems with compression on already compressed target files
We can make things better for that first interest while we have no real
data for the second interest -- our current workarounds to avoid
compression are based on hearsay, not testing.
Now that individual fetchers are possible I suggest we simplify
ngclient and allow compression. As an example the pip Fetcher
could still use the pip response chunking code with all their
workarounds -- pip certainly has better capability to maintain
a mountain of workarounds and also has endless amounts of real-world
testing compared to python-tuf.
Details:
* Stop modifying Accept-Encoding (Requests default includes gzip)
* Don't use response.raw in RequestsFetcher as there is no need:
This was a workaround for false "Content-encoding: gzip" inserted by
a broken server -- and the workaround was only possible because we
knew we never asked for compression
* Fix issue in test_session_get_timeout(): it's not mocking the error
that requests really raises in this case
Fixes#1251
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Update new test modules to stop using unittest_toolbox, in
preparation for its removal in #1790.
The tools provided by unittest_toolbox can easily (in a more
obvious way) be replaced by using the standard library modules
`tempfile` and `random` (no more used) directly.
In the case of tempdir and -file creation/removal, skipping the use
of unittest_toolbox, which does this by default, also uncovers some
test cleanup failures, which would occur when temporary test
directories were removed while a test server hadn't released them.
(see `except OSError: pass` in unittest_toolbox's `tearDown`
method)
**Change details**
**test_fetcher_ng.py:**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs. -Move now manual creation of an exemplary targets file to
setUpClass, as the same file is used by all tests. And remove it
explicitly in tearDownClass after killing the server (see note
about failure above). - Trigger URL parsing error with a hardcoded
invalid URL string instead of a random string.
**test_updater_ng.py**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs.
- Explicitly create tmp test dirs in setUp, but don't remove
them in tearDown to avoid above mentioned failures. They will be
removed all at once when removing the tmp root test dir in
tearDownClass
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Define TESTS_DIR constant in tests/util.py as full path to the
parent directory of the util module. This may be used to reliably
read other files in tests dir, such es "repository_data" or
"simple_server", regardless of cwd.
This commit also replaces a couple of `getcwd() + "filename"` with
`TESTS_DIR + filename`, so that in the future (post #1790) we
should be able to invoke the tests from anywhere, not only from
within the tests directory as is now the case.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
After we drop support for python3.6 we can relly that dictionaries
preserve the insertion order:
https://docs.python.org/3.7/whatsnew/3.7.html
This means we can replace the usage of OrderedDict with a standard
dictionaries.
Something we have to keep in mind is that even thought the insertion
order is preserved the equality comparison for normal dicts is
insensitive for normal dicts compared to OrderedDict
For example:
>>> OrderedDict([(1,1), (2,2)]) == OrderedDict([(2,2), (1,1)])
False
>>> dict([(1,1), (2,2)]) == dict([(2,2), (1,1)])
True
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
LengthOrHashMismatchError is a thrown when there are problems with
metadata verification or problems from the repository side when looking
it from the user's perspective.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
UnsupportedAlgorithmError is a detailed securesystemslib exception
and there is no need for TUF to redefine it.
Moreover which hash "algorithms" are allowed is work for
securesystemslib not for TUF.
It's only used once inside "Targetfile.from_data()" and there it's used
to denote that there is a problem with the given argument.
That's why this error can be just replaced with "ValueError".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
ReplayedMetadataError is a subset of
BadVersionNumberError and in a discussion with
Jussi we realized that ReplayedMetadataError can
be replaced by BadVersionNumberError with a
good message.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
URLParsingError is a specific download error and
is not clear what benefit it provides.
It's used only once in the new code and the
message says everything you need to know about
the exception.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add tuf/api/exceptions.py for exceptions in the new code.
I copied the exceptions from tuf/exceptions.py with a few important
decisions:
1. I only added the exceptions that are used in the new code
2. I removed the general "Error" class as we can directly inherit
Exceptions
3. I tried grouping the exceptions by relevance
4. I removed the second argument "UnsignedMetadataError" as it's only
kept for backward compatibility and is not used
5. I tried following the new code style guidelines and linted the file
with our linters.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit explicitly encodes role names. Mostly this encoding is already
happening in ``requests`` for what is not a URL.
The "/" in a role name will now be encoded.
Also, a slight change in the RepositorySimulator will align with the tests.
This commit partially covers issue #1634
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Add a method to rotate roles keys into RepositorySimulator (only
top-level roles are supported for now). Rotation is used in four
places already and this refactoring makes the tests easier to
understand.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
When calling updater._persist_metadata() there is a possibility that
writing the temporary file to storage can succeed, but moving it with
os.replace could fail with OSError.
Make sure we are removing the newly created temporary file in that case.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Remove `bump_version()` method, which is just an alias for "+= 1"
on the version attribute. For a slim low-level API it seems okay to
just directly access/modify the attribute.
The extra level of abstraction of "bumping a version" is more
appropriate for a repository library (see #1136).
This patch also removes a related unit test and updates another one
to directly do `(...).version +=`.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
New pylint warnings appeared related to changes
in urlib3:
- tests/test_fetcher_ng.py:128: error: Argument 1 to "ReadTimeoutError"
has incompatible type "None"; expected "ConnectionPool" [arg-type]
- tests/test_fetcher_ng.py:128: error: Argument 2 to "ReadTimeoutError"
has incompatible type "None"; expected "str" [arg-type]
I noticed these error in this CI run:
https://github.com/theupdateframework/python-tuf/runs/4764931441?check_suite_focus=true
I fixed them by creating a urllib3.HTTPConnectionPool() instance as
the first argument and replaced the second argument with an empty
string.
This seems to do the job.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
After making a successful update of valid metadata which stores it
in cache and performing a second update with a new updater while
the metadata is already stored in cache, this test verifies that
timestamp, snaphot and targets are loaded from cache and not
downloaded
Fixes#1681
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
keyids are ordered in the data we deserialize: Not preserving that order
breaks canonicalization. Set does not preserve order.
Change Role.keyids type from Set to List. This is strictly speaking
an API change but a minor one: keyids are supposed to be changed
via add_key()/remove_key().
Add tests for this for both Role and DelegatedRole. Shorten a related
exception message.
Fix#1752
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This test simulates the targets fast-forward attack recovery.
It simulates that the targets keys were compromised, the attacker
generated a new high version of the targets.
The repository generates new key for snapshot to rollback the
targets version to the initial version.
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Move the remaining
test_snapshot_rollback_with_local_snapshot_hash_mismatch
to test_updater_top_level_update.py and remove the file.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Remove TestUpdater.test_refresh from test_updater_with_simulator.
Testing refresh() is now extensively covered in the newly added
test_updater_top_level_update.py.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
