Fetcher interface should only raise DownloadErrors,
regardless of the implementation.
* Make sure fetch() wraps non-DownloadError errors in a DownloadError
* Make the abstract function private _fetch()
* Try to be more consistent in doscstrings
This now makes the example client more sensible (when server does not
respond):
$ ./client_example.py download qwerty
...
Failed to download target qwerty: Failed to download url http://127.0.0.1:8000/metadata/2.root.json
(here the latter part of the error string comes from DownloadError
raised by FetcherInterface.fetch())
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This commit tries to deal with two interests:
* metadata is highly repetitive and compressible: allowing compression
would be good
* there may be broken web servers (see
404838abcc/src/pip/_internal/download.py (L842))
that have problems with compression on already compressed target files
We can make things better for that first interest while we have no real
data for the second interest -- our current workarounds to avoid
compression are based on hearsay, not testing.
Now that individual fetchers are possible I suggest we simplify
ngclient and allow compression. As an example the pip Fetcher
could still use the pip response chunking code with all their
workarounds -- pip certainly has better capability to maintain
a mountain of workarounds and also has endless amounts of real-world
testing compared to python-tuf.
Details:
* Stop modifying Accept-Encoding (Requests default includes gzip)
* Don't use response.raw in RequestsFetcher as there is no need:
This was a workaround for false "Content-encoding: gzip" inserted by
a broken server -- and the workaround was only possible because we
knew we never asked for compression
* Fix issue in test_session_get_timeout(): it's not mocking the error
that requests really raises in this case
Fixes#1251
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Update new test modules to stop using unittest_toolbox, in
preparation for its removal in #1790.
The tools provided by unittest_toolbox can easily (in a more
obvious way) be replaced by using the standard library modules
`tempfile` and `random` (no more used) directly.
In the case of tempdir and -file creation/removal, skipping the use
of unittest_toolbox, which does this by default, also uncovers some
test cleanup failures, which would occur when temporary test
directories were removed while a test server hadn't released them.
(see `except OSError: pass` in unittest_toolbox's `tearDown`
method)
**Change details**
**test_fetcher_ng.py:**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs. -Move now manual creation of an exemplary targets file to
setUpClass, as the same file is used by all tests. And remove it
explicitly in tearDownClass after killing the server (see note
about failure above). - Trigger URL parsing error with a hardcoded
invalid URL string instead of a random string.
**test_updater_ng.py**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs.
- Explicitly create tmp test dirs in setUp, but don't remove
them in tearDown to avoid above mentioned failures. They will be
removed all at once when removing the tmp root test dir in
tearDownClass
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Define TESTS_DIR constant in tests/util.py as full path to the
parent directory of the util module. This may be used to reliably
read other files in tests dir, such es "repository_data" or
"simple_server", regardless of cwd.
This commit also replaces a couple of `getcwd() + "filename"` with
`TESTS_DIR + filename`, so that in the future (post #1790) we
should be able to invoke the tests from anywhere, not only from
within the tests directory as is now the case.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
After we drop support for python3.6 we can relly that dictionaries
preserve the insertion order:
https://docs.python.org/3.7/whatsnew/3.7.html
This means we can replace the usage of OrderedDict with a standard
dictionaries.
Something we have to keep in mind is that even thought the insertion
order is preserved the equality comparison for normal dicts is
insensitive for normal dicts compared to OrderedDict
For example:
>>> OrderedDict([(1,1), (2,2)]) == OrderedDict([(2,2), (1,1)])
False
>>> dict([(1,1), (2,2)]) == dict([(2,2), (1,1)])
True
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
LengthOrHashMismatchError is a thrown when there are problems with
metadata verification or problems from the repository side when looking
it from the user's perspective.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
UnsupportedAlgorithmError is a detailed securesystemslib exception
and there is no need for TUF to redefine it.
Moreover which hash "algorithms" are allowed is work for
securesystemslib not for TUF.
It's only used once inside "Targetfile.from_data()" and there it's used
to denote that there is a problem with the given argument.
That's why this error can be just replaced with "ValueError".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
ReplayedMetadataError is a subset of
BadVersionNumberError and in a discussion with
Jussi we realized that ReplayedMetadataError can
be replaced by BadVersionNumberError with a
good message.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
URLParsingError is a specific download error and
is not clear what benefit it provides.
It's used only once in the new code and the
message says everything you need to know about
the exception.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add tuf/api/exceptions.py for exceptions in the new code.
I copied the exceptions from tuf/exceptions.py with a few important
decisions:
1. I only added the exceptions that are used in the new code
2. I removed the general "Error" class as we can directly inherit
Exceptions
3. I tried grouping the exceptions by relevance
4. I removed the second argument "UnsignedMetadataError" as it's only
kept for backward compatibility and is not used
5. I tried following the new code style guidelines and linted the file
with our linters.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit explicitly encodes role names. Mostly this encoding is already
happening in ``requests`` for what is not a URL.
The "/" in a role name will now be encoded.
Also, a slight change in the RepositorySimulator will align with the tests.
This commit partially covers issue #1634
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Add a method to rotate roles keys into RepositorySimulator (only
top-level roles are supported for now). Rotation is used in four
places already and this refactoring makes the tests easier to
understand.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
When calling updater._persist_metadata() there is a possibility that
writing the temporary file to storage can succeed, but moving it with
os.replace could fail with OSError.
Make sure we are removing the newly created temporary file in that case.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Remove `bump_version()` method, which is just an alias for "+= 1"
on the version attribute. For a slim low-level API it seems okay to
just directly access/modify the attribute.
The extra level of abstraction of "bumping a version" is more
appropriate for a repository library (see #1136).
This patch also removes a related unit test and updates another one
to directly do `(...).version +=`.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
New pylint warnings appeared related to changes
in urlib3:
- tests/test_fetcher_ng.py:128: error: Argument 1 to "ReadTimeoutError"
has incompatible type "None"; expected "ConnectionPool" [arg-type]
- tests/test_fetcher_ng.py:128: error: Argument 2 to "ReadTimeoutError"
has incompatible type "None"; expected "str" [arg-type]
I noticed these error in this CI run:
https://github.com/theupdateframework/python-tuf/runs/4764931441?check_suite_focus=true
I fixed them by creating a urllib3.HTTPConnectionPool() instance as
the first argument and replaced the second argument with an empty
string.
This seems to do the job.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
After making a successful update of valid metadata which stores it
in cache and performing a second update with a new updater while
the metadata is already stored in cache, this test verifies that
timestamp, snaphot and targets are loaded from cache and not
downloaded
Fixes#1681
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
keyids are ordered in the data we deserialize: Not preserving that order
breaks canonicalization. Set does not preserve order.
Change Role.keyids type from Set to List. This is strictly speaking
an API change but a minor one: keyids are supposed to be changed
via add_key()/remove_key().
Add tests for this for both Role and DelegatedRole. Shorten a related
exception message.
Fix#1752
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This test simulates the targets fast-forward attack recovery.
It simulates that the targets keys were compromised, the attacker
generated a new high version of the targets.
The repository generates new key for snapshot to rollback the
targets version to the initial version.
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Move the remaining
test_snapshot_rollback_with_local_snapshot_hash_mismatch
to test_updater_top_level_update.py and remove the file.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Remove TestUpdater.test_refresh from test_updater_with_simulator.
Testing refresh() is now extensively covered in the newly added
test_updater_top_level_update.py.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Remove `bump_expiration()` method, which is unlikely to be used as
is, i.e. bump to "current expiration date plus delta". A more
realistic use case is to bump to "now plus delta" (see #1727 for
details).
Moreover, bump_expiration can either way easily be replaced by a
one-liner expression using the 'datetime' module. A corresponding
code snippet is added to the `expires` property's docstring. Note:
`expires` became a property with a millisec-removing setter (for
spec conformance) in #1712, which further reduces the need for a
convenience bump_expiration method.
This patch also removes a related unit test and updates another
one.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add test cases covering downloading and loading from cache
targets with non-matching hash and length.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Remove parts of the test case which are covered in other
tests, this way making its purpose clearer.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Add a new test file and class for testing target files
fetching.
Move test_targets from test_updater_with_simulator.py to
tests/test_updater_fetch_targets.py.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Remove ambiguous, unspecific, opinionated and trivial 'update'
methods, which can be replaced by feasible one-liners that assign
values directly to the object attribute to be *updated*. (see #1627
for details).
Reasons to have these methods would be increased usability in terms of
- reduced work
- immediate feedback on invalid assignments
However, given above described issues, the reasons against the
methods as they are now seem to outweigh the reasons for them.
Furthermore, it seems easier to re-add similar methods, which
addressed these issues, after the upcoming 1.0.0 release than to
remove or modify them.
This patch also removes the corresponding tests as they become
irrelevant (there is no need to test object assignment). In the
case of the timestamp test, the removal also includes redundant
test logic, which is already tested in `test_metadata_base`.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This test simulates the snapshot fast-forward attack recovery.
It simulates that the snapshot keys were compromised, the attacker
generated a new high version of the snapshot.
The repository generates new keys for snapshot and timestamp and
rollbacks the snapshot version to the initial version.
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Using a raw string allows the use of backslashes
in the docstring comment whithout them being interpreted
as an escape character.
It also silences pylint W1401: anomalous-backslash-in-string.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
