Our sdist has typically included everything from git apart from the CI
related files (.github/*, .fossa.yml, .readthedocs.yaml). Update our
MANIFEST.in and the check-manifest section of setup.cfg to be explicit
about this.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add a 'Documentation' entry to project_urls pointing to our stable docs
on readthedocs.io. This will result in a 'Documentation' entry under the
'Project links' section on PyPI.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Invoking setup.py directly is deprecated, see:
https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html
Therefore:
* remove the executable bit from setup.py's permissions
* remove the shebang entry from setup.py
* update the comments in setup.py to recommend using build to create dists
and pip to install them
Signed-off-by: Joshua Lock <jlock@vmware.com>
Update the MANIFEST.in to be explicit about what we choose to ship in our
sdist. This _does not_ result in any additional files being included in
our sdist, but does remove warnings from build.
Signed-off-by: Joshua Lock <jlock@vmware.com>
* List license files in a new metadata section
* Remove .travis.yml from check-manifest section's ignore entry
Signed-off-by: Joshua Lock <jlock@vmware.com>
build, twine and wheel packages should all be installed in order to be
able to build and release python-tuf -- add those dependencies to
requirements-dev.txt
Signed-off-by: Joshua Lock <jlock@vmware.com>
Python 3.10 is released on October 4-th 2021 and it seems
logical to add support for it as it doesn't require any major effort
from the project.
For reference read:
https://www.python.org/downloads/release/python-3100/
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
When I tried adding support for Python3.10 we had CI errors due to
test failures: https://github.com/theupdateframework/python-tuf/pull/1610/checks?check_run_id=3861875325
The problem comes from the fact that we start a subprocess
executing simple_https_server.py, but then we fail to communicate the
message we expect from the server process to the main process actually
running the test. We expect our custom message to be the first line
printed from the server process, but instead, a deprecation warning is
printed first about the usage of ssl.wrap_socket(). Our custom message
is printed second.
As of Python 3.7 this function has been deprecated:
https://docs.python.org/3/library/ssl.html#ssl.wrap_socket and for
whatever the reason we didn't get a warning when using it before.
My fix does what is suggested in the warning and replaces the usage of
ssl.wrap_socket() by instantiating a ssl.SSLContext object and then
calling SSLContext.wrap_socket().
This removes the warning.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
For users of legacy client (tuf/client/) this is purely a security fix
release with no API or functionality changes. For ngclient and Metadata
API, some API changes are included.
All users are advised to upgrade.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
_fileinfo_has_changed() and _update_fileinfo() have been unused internal
methods since 2016. Remove them.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
The original commit 051b8229 handled the loading and saving metadata
cases but the legacy client actually checks for the files existence
in various other places:
* _update_versioninfo() never reads the file but operates differently
depending on whether the file exists or not
* _move_current_to_previous() that copies files around
* MultiRepoUpdater initialization: this only handle root.json so
is still correct
* _update_fileinfo() which is dead code
Fix the first two of these cases.
Make sure rolenames like "../a" won't trick ngclient into creating the
metadata file outside the metadata cache.
The test data was semi-manually created with RepositorySimulator:
this test code could use RepositorySimulator directly instead (like the
ngclient tests do) but that would require some more infrastructural
work.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
If an attacker manages to create arbitrary rolenames they could trick
the client into writing metadata files into unexpected locations:
To avoid directory traversal and writing files into unexpected
locations, encode the rolename before using it as filename.
If a client has delegated targets metadata with rolenames that have
percent-encoded characters in them, these metadata will now not be
found in local metadata cache and must be re-downloaded.
Note that this does not mean using rolenames that get encoded is
advisable (as forming the download URLs still has issues with them),
this just means the client will not do unsafe writes when it encounters
rolenames like this.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Add support for adding delegated targets into RepositorySimulator.
Make the metadata URL parsing in RepositorySimulator more robust.
Add a test to make sure "../a" won't trick ngclient into creating the
metadata file outside the metadata cache.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
If an attacker manages to create arbitrary rolenames they could trick
the client into writing metadata files into unexpected locations:
To avoid directory traversal and writing files into unexpected
locations, encode the rolename before using it as filename.
If a client has delegated targets metadata with rolenames that have
percent-encoded characters in them, these metadata will now not be
found in local metadata cache and must be re-downloaded.
Note that this does not mean using rolenames that get encoded is
advisable (as forming the download URLs still has issues with them),
this just means the client will not do unsafe writes when it encounters
rolenames like this.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
urljoin considers the second URL to override the base URL if the second
one contains e.g. hostname: this could lead to ngclient downloading
from the wrong host entirely. Doing that would not compromise the
security of the system as the metadata would still need to be verified,
but would definitely be unexpected and a bug.
Note that we're still not encoding the rolename, it's just inserted into
the URL as is.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Modify RepositorySimulator function delegates() to all_targets(), so
that all targets can be traversed and updated with one cycle when
calling update_snapshot() (which is the only use case for now for
delegates()).
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add an option to calculate the hashes and length for timestamp/snapshot
meta.
This will help to cover more use cases with the repository simulator.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
If you do the following steps:
1. call Updater.refresh() and load, verify and cache all metadata files
2. modify timestamp snapshot meta information:
(One or more of hashes or length for snapshot changes here)
3. call Updater.refresh() again
4. root and timestamp will be updated to their latest versions
5. local snapshot will be loaded, but hashes/length will be different
than the ones in timestamp.snapshot_meta and that will prevent loading
6. remote snapshot is loaded and verification starts
then when executing step 6 the rollback checks will not be done because
the old snapshot was not loaded on step 5.
In order to resolve this issue, we are introducing the idea of trusted and
untrusted snapshot.
Trusted snapshot is the locally available cached version. This version has
been verified at least once meaning hashes and length were already checked
against timestamp.snapshot_meta hashes and length.
That's why we can allow loading a trusted snapshot version even if there is a
mismatch between the current timestamp.snapshot_meta hashes/length and
hashes/length inside the trusted snapshot.
Untrusted snapshot is the one downloaded from the web. It hasn't been verified
before and that's why we mandate that timestamp.snapshot_meta hashes and length
should match the hashes and legth calculated on this untrusted version of
snapshot.
As the TrustedMetadataSet doesn't have information which snapshot is trusted or
not, so possibly the best solution is to add a new argument "trusted"
to update_snapshot.
Even though this is ugly as the rest of the update functions doesn't
have such an argument, it seems the best solution as it seems to work
in all cases:
- when loading a local snapshot, we know the data has at some point been
trusted (signatures have been checked): it doesn't need to match hashes
now
- if there is no local snapshot and we're updating from remote, the
remote data must match meta hashes in timestamp
- if there is a local snapshot and we're updating from remote, the remote
data must match meta hashes in timestamp
Lastly, I want to point out that hash checks for metadata files are not
essential to TUF security guarantees: they are just an additional layer of
security that allows us to avoid even parsing json that could be malicious -
we already know the malicious metadata would be stopped at metadata
verification after the parsing.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The purpose of this config was to ensure blocking
inside the download loop and releasing CPU resources.
To our best knowledge the network stack currently used
in RequestsFetcher will always block inside the loop
and the issue cannot be reproduced.
'chunk_size' and 'socket_timeout' are currently the
settings provided by RequestsFetcher to tweak
CPU usage and download granularity.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Reuse the decorator defined in tests/utils.py in order
to receive more helpful messages when an assertion
fails in test_tragets().
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
