Catch Metadata.sign() securesystemslib exceptions and instead throw
a more general UnsignedMetadataError exception.
We don't want to expose securesystemslib exceptions and it's better
to replace them with a more general exception that could be easily
handled.
As the signer is an argument implementing securesystemslib.signer.Signer
interface we don't know what exception will it throw.
That's why we need to catch all possible exceptions during signing and
raise UnsignedMetadataError.
That is the same reason why we should move the serialization outside
the "try" block, so a tuf.api.serialization.SerializationError can
propagate and warn the user that 'signed' cannot be serialized.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Reexport securesystemslib StorageError, so that our users can catch
it without importing securesystemslib.
The securesystemslib StorageError makes sense in the context of
metadata API, because it supports different storage interfaces and
this exception is denoting all possible errors that could arrise
from using any kind of storage interface.
Additionally, I changed the places where we mention that StorageError
is thrown, so that our users will know they can directly import it
from tuf/api/exceptions.py instead of importing securesystemslib.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Replace the usage of securesystemslib.util.persist_temp_file() with
shutil.copyfileobj() as file system abstraction is not used in the
client.
This way we prevent securesystemslib.exception.StorageError from
leaking through client API calls.
Note: with those changes we are no longer do fsync.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
I made a review on all files inside tuf/ngclient to see which of them
needs additions or changes in their function docstrings regarding
exceptions.
I didn't find any changes required inside the request_fetcher.py
and of course inside the config module.
Other than that multiple additions had to be made.
For trusted_metadata_set we had a discussion with Jussi that there is
no need to list each of the specific RepositoryErrors one by one as
this is an internal module and this will only create a bigger
maintenance burden.
For updater.py we had discussions with Jussi and Lukas that we want to
document only those exceptions that could be potentially handled.
This means there is no point in documenting each of the RepositoryErrors
or DownloadErrors separately.
Finally, I added a little documentation for download_bytes() inside
fetcher.py, as it's naming, suggests it's not an internal function.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit tries to deal with two interests:
* metadata is highly repetitive and compressible: allowing compression
would be good
* there may be broken web servers (see
404838abcc/src/pip/_internal/download.py (L842))
that have problems with compression on already compressed target files
We can make things better for that first interest while we have no real
data for the second interest -- our current workarounds to avoid
compression are based on hearsay, not testing.
Now that individual fetchers are possible I suggest we simplify
ngclient and allow compression. As an example the pip Fetcher
could still use the pip response chunking code with all their
workarounds -- pip certainly has better capability to maintain
a mountain of workarounds and also has endless amounts of real-world
testing compared to python-tuf.
Details:
* Stop modifying Accept-Encoding (Requests default includes gzip)
* Don't use response.raw in RequestsFetcher as there is no need:
This was a workaround for false "Content-encoding: gzip" inserted by
a broken server -- and the workaround was only possible because we
knew we never asked for compression
* Fix issue in test_session_get_timeout(): it's not mocking the error
that requests really raises in this case
Fixes#1251
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This commit simplifies the contributor's doc.
It adds the reference to the Secure Systems Lab Development Guidelines,
gives more evidence to the tox usage, shares information about the tests,
linting, and coverage, and creates a session about submitting
the contributions highlighted by the DCO.
Fixes#1709
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Update new test modules to stop using unittest_toolbox, in
preparation for its removal in #1790.
The tools provided by unittest_toolbox can easily (in a more
obvious way) be replaced by using the standard library modules
`tempfile` and `random` (no more used) directly.
In the case of tempdir and -file creation/removal, skipping the use
of unittest_toolbox, which does this by default, also uncovers some
test cleanup failures, which would occur when temporary test
directories were removed while a test server hadn't released them.
(see `except OSError: pass` in unittest_toolbox's `tearDown`
method)
**Change details**
**test_fetcher_ng.py:**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs. -Move now manual creation of an exemplary targets file to
setUpClass, as the same file is used by all tests. And remove it
explicitly in tearDownClass after killing the server (see note
about failure above). - Trigger URL parsing error with a hardcoded
invalid URL string instead of a random string.
**test_updater_ng.py**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs.
- Explicitly create tmp test dirs in setUp, but don't remove
them in tearDown to avoid above mentioned failures. They will be
removed all at once when removing the tmp root test dir in
tearDownClass
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Define TESTS_DIR constant in tests/util.py as full path to the
parent directory of the util module. This may be used to reliably
read other files in tests dir, such es "repository_data" or
"simple_server", regardless of cwd.
This commit also replaces a couple of `getcwd() + "filename"` with
`TESTS_DIR + filename`, so that in the future (post #1790) we
should be able to invoke the tests from anywhere, not only from
within the tests directory as is now the case.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
We no longer need to catch LengthOrHashMismatchError and reraise a
RepositoryError as LengthOrHashMismatchError is changed to inherit
RepositoryError.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
After we drop support for python3.6 we can relly that dictionaries
preserve the insertion order:
https://docs.python.org/3.7/whatsnew/3.7.html
This means we can replace the usage of OrderedDict with a standard
dictionaries.
Something we have to keep in mind is that even thought the insertion
order is preserved the equality comparison for normal dicts is
insensitive for normal dicts compared to OrderedDict
For example:
>>> OrderedDict([(1,1), (2,2)]) == OrderedDict([(2,2), (1,1)])
False
>>> dict([(1,1), (2,2)]) == dict([(2,2), (1,1)])
True
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
SerializationError and DeserializationError are both errors coming
from the repository side looking from the clients point of view.
That's why it makes sense to make them repository errors.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
LengthOrHashMismatchError is a thrown when there are problems with
metadata verification or problems from the repository side when looking
it from the user's perspective.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
UnsupportedAlgorithmError is a detailed securesystemslib exception
and there is no need for TUF to redefine it.
Moreover which hash "algorithms" are allowed is work for
securesystemslib not for TUF.
It's only used once inside "Targetfile.from_data()" and there it's used
to denote that there is a problem with the given argument.
That's why this error can be just replaced with "ValueError".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
ReplayedMetadataError is a subset of
BadVersionNumberError and in a discussion with
Jussi we realized that ReplayedMetadataError can
be replaced by BadVersionNumberError with a
good message.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
URLParsingError is a specific download error and
is not clear what benefit it provides.
It's used only once in the new code and the
message says everything you need to know about
the exception.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Make FetcherHTTPError a DownloadError as the
error itself denotes an error happening during
the download process.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Stop linting tuf/exceptions.py with mypy as we are going to use
tuf/api/exceptions.py for exceptions in the new code.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add tuf/api/exceptions.py for exceptions in the new code.
I copied the exceptions from tuf/exceptions.py with a few important
decisions:
1. I only added the exceptions that are used in the new code
2. I removed the general "Error" class as we can directly inherit
Exceptions
3. I tried grouping the exceptions by relevance
4. I removed the second argument "UnsignedMetadataError" as it's only
kept for backward compatibility and is not used
5. I tried following the new code style guidelines and linted the file
with our linters.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The code examples now have a simple README.md. This commit simplifies
the README.md examples to the example folder instead of listing each
example.
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
This commit adds to the RTD the links references to source code
examples.
The examples are added to TUF ngclient Updater, Metadata and API
reference.
includes a seed for examples/README.md
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
This commit fixes the relatives links on README.md, adding absolute
links.
The PyPI renders the README.md as an HTML page on the landing page
of the python-tuf package. If the links are not absolute links to
the files in git repository, they will fail.
Fixes: #1739
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
This commit explicitly encodes role names. Mostly this encoding is already
happening in ``requests`` for what is not a URL.
The "/" in a role name will now be encoded.
Also, a slight change in the RepositorySimulator will align with the tests.
This commit partially covers issue #1634
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Remove old doc/tuf-spec* documents, which are merely pointers to
the theupdateframework/specification repo (created in late 2017).
They were likely kept in place to avoid 404s of old links, but the
up-to-date TUF specification location should be discoverable enough
to get rid of the pointers.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
