* Remove exectuable flag from a couple of files
* Half of the test files have a shebang (but are
still not executable): remove the shebang
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
* API changes covered:
* keys and interface modules removed
* SSlibSigner removed
* CryptoSigner added: this replaces the removed functionality
* DSSE "signatures" container type changed
* Currently pins a securesystemslib main branch commit:
this shoudl be reverted before merging, when securesystemslib
has made a release
* tests/generated_data/generate_md.py was simplified
* Encrypted test keys in tests/repository_data/keystore were replaced
with the unencrypted PEM versions of the same keys
* The public test keys in tests/repository_data/keystore were removed
as they were not used anymore
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Change TrustedMetadataSet to load and verify metadata using a
MetadataUnwrapper instance.
IMPORTANT NOTES:
* Requires changing the TrustedMetadataSet to store payloads only, which
is okay, because signatures are no longer needed, after being verified
at load time.
* Includes a minor re-ordering of validation steps. That is, the
version increment for root metadata is now checked after signature
verification. Preserving the order would require including the check in
the Unwrapper interface, which is feasible but does not seem correct wrt
separation of responsibility.
Changes are adopted in updater, tests and _localrepo.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This test covers `targetinfo`, `target_path`, `target_base_url`,
`metadata_dir` and `filepath` input validation of the `Updater`
methods
Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
Update new test modules to stop using unittest_toolbox, in
preparation for its removal in #1790.
The tools provided by unittest_toolbox can easily (in a more
obvious way) be replaced by using the standard library modules
`tempfile` and `random` (no more used) directly.
In the case of tempdir and -file creation/removal, skipping the use
of unittest_toolbox, which does this by default, also uncovers some
test cleanup failures, which would occur when temporary test
directories were removed while a test server hadn't released them.
(see `except OSError: pass` in unittest_toolbox's `tearDown`
method)
**Change details**
**test_fetcher_ng.py:**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs. -Move now manual creation of an exemplary targets file to
setUpClass, as the same file is used by all tests. And remove it
explicitly in tearDownClass after killing the server (see note
about failure above). - Trigger URL parsing error with a hardcoded
invalid URL string instead of a random string.
**test_updater_ng.py**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs.
- Explicitly create tmp test dirs in setUp, but don't remove
them in tearDown to avoid above mentioned failures. They will be
removed all at once when removing the tmp root test dir in
tearDownClass
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Define TESTS_DIR constant in tests/util.py as full path to the
parent directory of the util module. This may be used to reliably
read other files in tests dir, such es "repository_data" or
"simple_server", regardless of cwd.
This commit also replaces a couple of `getcwd() + "filename"` with
`TESTS_DIR + filename`, so that in the future (post #1790) we
should be able to invoke the tests from anywhere, not only from
within the tests directory as is now the case.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add tuf/api/exceptions.py for exceptions in the new code.
I copied the exceptions from tuf/exceptions.py with a few important
decisions:
1. I only added the exceptions that are used in the new code
2. I removed the general "Error" class as we can directly inherit
Exceptions
3. I tried grouping the exceptions by relevance
4. I removed the second argument "UnsignedMetadataError" as it's only
kept for backward compatibility and is not used
5. I tried following the new code style guidelines and linted the file
with our linters.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
When calling updater._persist_metadata() there is a possibility that
writing the temporary file to storage can succeed, but moving it with
os.replace could fail with OSError.
Make sure we are removing the newly created temporary file in that case.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Remove `bump_version()` method, which is just an alias for "+= 1"
on the version attribute. For a slim low-level API it seems okay to
just directly access/modify the attribute.
The extra level of abstraction of "bumping a version" is more
appropriate for a repository library (see #1136).
This patch also removes a related unit test and updates another one
to directly do `(...).version +=`.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
metadata_dir matches metadata_base_url better.
This is an API break for anyone using named arguments.
Fixes#1638
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
The changes are automatic linting fixes from black.
The target files are only those who test the new code.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This applies the use of constants of top-level rolenames in the
tests instead of the previously hardcoded strings.
Fixes#1648
Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
As a target path is a URL path it's not correct to consider it as
interchangeable with a filepath within every operation system. The
unquote is also removed as the ngclient cannot assume correctly
which encoding is intended and which not
Fixes#1483
Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
This commit includes manual fixes for a lot of mypy warnings.
When there were warnings that we are calling non-annotated function
in annotated context I decided to add annotations instead of ignoring
those warnings.
That's how I end up adding annotations in the whole tests/utils.py
module.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Consistent snapshot and consistent targets are now
extensively tested in test_updater_consistent_snapshot.py.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Address or disable pylint warnings raised on all test files inside
the "tests/" directory testing the code of the new implementation.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This is a minimal test case to show the basic functionality:
I'd like to update most of the tests to use the implicit method
but would like to do that when there's less churn in these tests.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This is slightly cosmetic but rename get_one_valid_targetinfo to
get_targetinfo:
* The function name is long without any reason: "one" and "valid" are
always implicit
* shortening makes code (incl. our examples and tests) easier to read
* We're also already changing updater API (compared to legacy) so this
alone does not break things -- it's also not a difficult "port".
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Remove updated_targets() as it doesn't fit the rest of the API.
In its stead add find_cached_target() which has a similar signature
as download_target(): both accept an optional local filepath as
argument and return full local filepath. In the
find_cached_target() case None is returned if the local file is not the
correct target file.
Updater constructor gets a new optional target_dir argument: This means
client can avoid giving a local filepath as an argument to
find_cached_target()/download_target() -- Updater will then generate a
filename within targets_dir.
A reasonable use pattern (when targets_dir is set in constructor):
info = updater.get_one_valid_targetinfo("targetname")
path = updater.find_cached_target(info)
if path is None:
path = updater.download_target(info)
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Do not add similar assertions for test_refresh_on_consistent_targets():
The test is broken and can't actually update metadata (#1573).
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Add tests covering missing branches of the Updater
code. Inlcude ngclient in the total coverage report.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Currently, TargetFile instances do not contain the path relative URL of
the file they represent. The API itself does not need it but it could be
useful for users of the API.
As an example, the current client returns a dict for
get_one_valid_targetinfo(): that dict contains a filepath field and
a targetinfo field (essentially TargetFile).
We would like to keep a similar API, but avoid hand-crafted dicts.
It would be much nicer to return a TargetFile that would contain the
full "metadata" of the targetfile.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add support for prefixing targets with their hashes when downloading or
using HASH.FILENAME.EXT as target names.
The introduction of prefix_targets_with_hash was necessary, because
there are use cases like Warehouse where you could use
consistent_snapshot, but without adding a hash prefix to your targets.
When prefix_targets_with_hash is set to True, target files conforming
the format HASH.FILENAME.EXT will be downloaded from the server, but
they will be saved on the client side without their hash prefixes or
FILENAME.EXT.
This makes sure the client won't understand the usage of
prefix_targets_with_hash.
Still, if you want to use HASH.FILENAME.EXT as target names when
downloading, then additionally you need to provide consistent_snapshot
set to True in your root.json. The reason is that the specification uses
consistent_snapshot for the same purpose:
"If consistent snapshots are not used (see § 6.2 Consistent snapshots),
then the filename used to download the target file is of the fixed form
FILENAME.EXT (e.g., foobar.tar.gz). Otherwise, the filename is of the
form HASH.FILENAME.EXT
(e.g., c14aeb4ac9f4a8fc0d83d12482b9197452f6adf3eb710e3b1e2b79e8d14cb681.foobar.tar.gz),
where HASH is one of the hashes of the targets file listed in the
targets metadata file found earlier in step § 5.6 Update the targets role.
In either case, the client MUST write the file to non-volatile
storage as FILENAME.EXT."
The same behavior of using two flags is used in the legacy code when
calling tuf.client.updater.download_target() in a repository using
prefix_targets_with_hash and consistent_snapshot.
See chapter 5.7.3:
https://theupdateframework.github.io/specification/latest/index.html#fetch-target
By default, prefix_targets_with_hash is set to true to make it easier
to the user to provide uniquely identifiable targets file names by
using consistent_snapshot set to True.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
