I'm not sure I agree with not using the parens in
raise SomeError
but being consistent is definitely better than not being consistent.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
* API changes covered:
* keys and interface modules removed
* SSlibSigner removed
* CryptoSigner added: this replaces the removed functionality
* DSSE "signatures" container type changed
* Currently pins a securesystemslib main branch commit:
this shoudl be reverted before merging, when securesystemslib
has made a release
* tests/generated_data/generate_md.py was simplified
* Encrypted test keys in tests/repository_data/keystore were replaced
with the unencrypted PEM versions of the same keys
* The public test keys in tests/repository_data/keystore were removed
as they were not used anymore
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
* application user-agent can be set with UpdaterConfig object
* Setting will affect the default fetcher only
* the application user-agent will be prefixed to the ngclient
default user-agent
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Minor fixes were needed, the only possibly interesting one is
the one in RequestsFetcher (use "yield from").
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
In RequestsFetcher it makes sense to raise "from e" but in
updater the error we raise is not related to the original
error: use "from None".
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
The internal wrapping interface to case handle deserialization and
verification of traditional metadata vs. simple envelopes inside
trusted metadata set might be a more complicated solution than
necessary.
This removes the interface and instead adds the methods of the interface
implementations as helpers to trusted metadata set, and updates it to
to call one or the other function based on the envelope type
configuration flag.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The flag allows adding other envelope types in the future (unlikely),
or parallel support (`METADATA & SIMPLE`) without breaking the API.
Internally, the flag is now just passed on to TrustedMetadataSet as
mandatory parameter. (Optional parameters make less sense when we
control all the invocations.)
This change requires updating all invocations of TrustedMetadataSet,
including the duplication of a test function.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
* Rename Envelope to SimpleEnvelope:
Envelope should be the generic term in this context for something that
contains a payload and signatures. SimpleEnvelope is the specific
DSSE implementation (just like Metadata is the specific traditional
canonical JSON -based TUF envelope implementation).
* Add SimpleEnvelope class and method docstrings.
* Add convenience alias for ``self.signatures`` mapped to keyids for
compatibility with Metadata.signatures.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
* Add `use_dsse` updater config parameter which indicates, if an updater
instance expects metadata to come in a DSSE envelope.
* Update TrustedMetadataSet to take an Unwrapper instance.
* Update Updater, to pass an EnvelopeUnwrapper to TrustedMetadataSet, if
configured with `use_dsse`.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add Unwrapper implementation for DSSE Envelope. The order of
deserialization and signature verification differs from traditional
Metadata.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Change TrustedMetadataSet to load and verify metadata using a
MetadataUnwrapper instance.
IMPORTANT NOTES:
* Requires changing the TrustedMetadataSet to store payloads only, which
is okay, because signatures are no longer needed, after being verified
at load time.
* Includes a minor re-ordering of validation steps. That is, the
version increment for root metadata is now checked after signature
verification. Preserving the order would require including the check in
the Unwrapper interface, which is feasible but does not seem correct wrt
separation of responsibility.
Changes are adopted in updater, tests and _localrepo.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add internal payload unwrapper interface and implementation for payloads
wrapped in Metadata.
This is an abstraction over behavior -- load signature wrapper, verify
signatures over payload, return deserialized payload -- which is common
for relevant signature wrappers (Metadata, DSSE Envelope), but performed
differently / in different order.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- adpot changes in dependabot.yml and remove --diff from ruff check.
- select pydocstyle, isort, pyflakes, pep8-naming, pycodestyle for ruff and ignore some small issues / add inline comments.
- adjust docstring length to 80 in various files
Signed-off-by: E3E <ntanzill@purdue.edu>
* Increase default network timeout
* trying to defend against slow retrieval attacks in a generic library is impossible
but too low timeouts mean failures in high latency systems (like tests running
on CI).
Signed-off-by: E3E <ntanzill@purdue.edu>
Change new _Delegator.verify_delegate to take payload bytes and
signatures instead of a Metadata object and a payload serializer.
This allows using verify_delegate for payloads that do not come in
a Metadata container, but e.g. in a DSSE envelope (see #2385).
Usage becomes a bit more cumbersome, but still feels reasonable with the
recently added shortcut for default canonical bytes representation of
Metadata.signed.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Avoid Metadata.verify_delegate() now that it's deprecated.
Note that this commit does not try to make any code cleanups
that are now possible: this is the minimal change to use the new
API.
Future improvements can make code in TrustedMetadataSet and
Updater slightly easier to read: as an example there's no need for
TrustedMetadataSet to actually store or expose actual Metadata in its
cache -- Signed is all that's needed.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
This is useful for those who want to use the default fetcher
but modify some attributes
The file itself could be moved to tuf/ngclient/ but this is not done yet
as sigstore-python is using this internal module. Move can be done once
sigstore-python 1.0 is no longer relevant.
Fixes#2268
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
The properties in TrustedMetadataSet are a bit difficult to use
with static typing since they return Optional but in many cases
we know the "None"-case is impossible.
Remove None from annotation: the idea is that calling the property
getter too early is a programming error: it will result in KeyError
which is consistent:
* trusted_set["timestamp"] raises KeyError if timestamp is not set
* trusted_set.timestamp raises KeyError if timestamp is not set
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
The typeshed annotations for requests say that the hostname could be None:
I think this is untrue but let's keep mypy happy.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
In the spec version 1.0.30, a new change has been added considering what
should happen if there is a new timestamp with the same version.
It says the following:
"In case they [versions] are equal, discard the new
timestamp metadata and abort the update cycle.
This is normal and it shouldn't raise any error."
In other words, if there is a new timestamp with the same version, then
stop the update process and use the old timestamp.
Those changes reflect these latest specification modifications.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This change fixes the expired metadata tests to mock `datetime`
as previously they mocked `time` incorrectly, which did not affect
update methods, as they use `datetime.datetime.utcnow()` to
calculate now
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change shortens line lengths that exceed the requiremets and
adds more clarification on methods where the short message is not
complete enough
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change unifies common wording in the docstrings library of
ngclient, like "Args" vs. "Arguments"
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change updates the docstrings library of ngclient with no
article for all Args in order to be unified amongst all python-tuf
docstrings
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
This change updates the docstrings library of ngclient with
unified double backtick quoting for better readability
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
I've not supported many renames but I'm suggesting this one:
FetcherHTTPError was created because we needed to signal 403/404
from the fetcher to updater. At that time the download error hierarchy
in general was not thought out.
Now we have a couple of different errors all derived from
DownloadError. I believe it does not make sense to point out "Fetcher"
in one of their names: DownloadHTTPError makes it clearer this is a
specific type of DownloadError.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Fetcher interface should only raise DownloadErrors,
regardless of the implementation.
* Make sure fetch() wraps non-DownloadError errors in a DownloadError
* Make the abstract function private _fetch()
* Try to be more consistent in doscstrings
This now makes the example client more sensible (when server does not
respond):
$ ./client_example.py download qwerty
...
Failed to download target qwerty: Failed to download url http://127.0.0.1:8000/metadata/2.root.json
(here the latter part of the error string comes from DownloadError
raised by FetcherInterface.fetch())
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
I made a review on all files inside tuf/ngclient to see which of them
needs additions or changes in their function docstrings regarding
exceptions.
I didn't find any changes required inside the request_fetcher.py
and of course inside the config module.
Other than that multiple additions had to be made.
For trusted_metadata_set we had a discussion with Jussi that there is
no need to list each of the specific RepositoryErrors one by one as
this is an internal module and this will only create a bigger
maintenance burden.
For updater.py we had discussions with Jussi and Lukas that we want to
document only those exceptions that could be potentially handled.
This means there is no point in documenting each of the RepositoryErrors
or DownloadErrors separately.
Finally, I added a little documentation for download_bytes() inside
fetcher.py, as it's naming, suggests it's not an internal function.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit tries to deal with two interests:
* metadata is highly repetitive and compressible: allowing compression
would be good
* there may be broken web servers (see
404838abcc/src/pip/_internal/download.py (L842))
that have problems with compression on already compressed target files
We can make things better for that first interest while we have no real
data for the second interest -- our current workarounds to avoid
compression are based on hearsay, not testing.
Now that individual fetchers are possible I suggest we simplify
ngclient and allow compression. As an example the pip Fetcher
could still use the pip response chunking code with all their
workarounds -- pip certainly has better capability to maintain
a mountain of workarounds and also has endless amounts of real-world
testing compared to python-tuf.
Details:
* Stop modifying Accept-Encoding (Requests default includes gzip)
* Don't use response.raw in RequestsFetcher as there is no need:
This was a workaround for false "Content-encoding: gzip" inserted by
a broken server -- and the workaround was only possible because we
knew we never asked for compression
* Fix issue in test_session_get_timeout(): it's not mocking the error
that requests really raises in this case
Fixes#1251
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
