This is useful for those who want to use the default fetcher
but modify some attributes
The file itself could be moved to tuf/ngclient/ but this is not done yet
as sigstore-python is using this internal module. Move can be done once
sigstore-python 1.0 is no longer relevant.
Fixes#2268
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
The properties in TrustedMetadataSet are a bit difficult to use
with static typing since they return Optional but in many cases
we know the "None"-case is impossible.
Remove None from annotation: the idea is that calling the property
getter too early is a programming error: it will result in KeyError
which is consistent:
* trusted_set["timestamp"] raises KeyError if timestamp is not set
* trusted_set.timestamp raises KeyError if timestamp is not set
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
If role is delegated but missing from snapshot, we currently raise a
undocumented KeyError: a generic RepositoryError seems better as callers
are expected to handle it (and adding a more specific error seems
useless as this is a repository software bug, not just expired metadata or
something).
The same check is also done later in TrustedMetadataSet but I think
keeping the check in both is clearest.
Fixes#2195
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
There may be ways to unsafely use the client library but situation
should be significantly better now with ngclient:
* metadata writing is safer, more atomic
* non-root cached metadata is never trusted (so inconsistent
cached repository is not a security issue)
* the cache locations are now clearly application
decisions (they are required Updater constructor args)
Move the notice to Updater module documentation.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
The typeshed annotations for requests say that the hostname could be None:
I think this is untrue but let's keep mypy happy.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
In the spec version 1.0.30, a new change has been added considering what
should happen if there is a new timestamp with the same version.
It says the following:
"In case they [versions] are equal, discard the new
timestamp metadata and abort the update cycle.
This is normal and it shouldn't raise any error."
In other words, if there is a new timestamp with the same version, then
stop the update process and use the old timestamp.
Those changes reflect these latest specification modifications.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit contains 2 API changes in "Delegations" class from
tuf/api/metadata.py:
1. roles argment is made optional
2. unrecognized_fields argument becomes the 4-th rather than the 3-rd
as it used to be
In this commit, I add support for succinct_roles roles inside
Delegations class. This change is related to TAP 15 proposal.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The class docstring for FetcherInterface needed to clearly state that
only _fetch() had to be implemented in it's implementation. This is
because the public API of the interface is implemented already.
Signed-off-by: Abhisman Sarkar <abhisman.sarkar@gmail.com>
Annotating local temp_file_name variable is simple than
to annotate temp_file.
Fixes#1999
Signed-off-by: Dhaval Shah <30974879+dhavalgshah@users.noreply.github.com>
This change fixes the expired metadata tests to mock `datetime`
as previously they mocked `time` incorrectly, which did not affect
update methods, as they use `datetime.datetime.utcnow()` to
calculate now
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change shortens line lengths that exceed the requiremets and
adds more clarification on methods where the short message is not
complete enough
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change unifies common wording in the docstrings library of
ngclient, like "Args" vs. "Arguments"
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change updates the docstrings library of ngclient with no
article for all Args in order to be unified amongst all python-tuf
docstrings
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change updates the docstrings library of ngclient with
unified double backtick quoting for better readability
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
I've not supported many renames but I'm suggesting this one:
FetcherHTTPError was created because we needed to signal 403/404
from the fetcher to updater. At that time the download error hierarchy
in general was not thought out.
Now we have a couple of different errors all derived from
DownloadError. I believe it does not make sense to point out "Fetcher"
in one of their names: DownloadHTTPError makes it clearer this is a
specific type of DownloadError.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Fetcher interface should only raise DownloadErrors,
regardless of the implementation.
* Make sure fetch() wraps non-DownloadError errors in a DownloadError
* Make the abstract function private _fetch()
* Try to be more consistent in doscstrings
This now makes the example client more sensible (when server does not
respond):
$ ./client_example.py download qwerty
...
Failed to download target qwerty: Failed to download url http://127.0.0.1:8000/metadata/2.root.json
(here the latter part of the error string comes from DownloadError
raised by FetcherInterface.fetch())
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Replace the usage of securesystemslib.util.persist_temp_file() with
shutil.copyfileobj() as file system abstraction is not used in the
client.
This way we prevent securesystemslib.exception.StorageError from
leaking through client API calls.
Note: with those changes we are no longer do fsync.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
I made a review on all files inside tuf/ngclient to see which of them
needs additions or changes in their function docstrings regarding
exceptions.
I didn't find any changes required inside the request_fetcher.py
and of course inside the config module.
Other than that multiple additions had to be made.
For trusted_metadata_set we had a discussion with Jussi that there is
no need to list each of the specific RepositoryErrors one by one as
this is an internal module and this will only create a bigger
maintenance burden.
For updater.py we had discussions with Jussi and Lukas that we want to
document only those exceptions that could be potentially handled.
This means there is no point in documenting each of the RepositoryErrors
or DownloadErrors separately.
Finally, I added a little documentation for download_bytes() inside
fetcher.py, as it's naming, suggests it's not an internal function.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit tries to deal with two interests:
* metadata is highly repetitive and compressible: allowing compression
would be good
* there may be broken web servers (see
404838abcc/src/pip/_internal/download.py (L842))
that have problems with compression on already compressed target files
We can make things better for that first interest while we have no real
data for the second interest -- our current workarounds to avoid
compression are based on hearsay, not testing.
Now that individual fetchers are possible I suggest we simplify
ngclient and allow compression. As an example the pip Fetcher
could still use the pip response chunking code with all their
workarounds -- pip certainly has better capability to maintain
a mountain of workarounds and also has endless amounts of real-world
testing compared to python-tuf.
Details:
* Stop modifying Accept-Encoding (Requests default includes gzip)
* Don't use response.raw in RequestsFetcher as there is no need:
This was a workaround for false "Content-encoding: gzip" inserted by
a broken server -- and the workaround was only possible because we
knew we never asked for compression
* Fix issue in test_session_get_timeout(): it's not mocking the error
that requests really raises in this case
Fixes#1251
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
