A few test conditions in test_updater.py incorrectly used self.assertTrue() instead of self.assertEqual().
Fix updater.remove_obsolete_targets(), where targets in the destination directory were not being removed because target paths were treated as absolute paths by os.path.join().
Fix test conditions for updated_targets(), which incorrectly verified the expected number of updated targets with assertTrue().
Update comment for PR fix to clarify os.path.join() behavior and the expected file paths.
os.path.join ignore previous parameters if one starts with '/'. All
targets start with '/', making updated_targets to try to open a file in
the root folder in case of a unix system.
1. Raise an exception for clients that do not wish to automatically fetch a Root file when an expired version is loaded from disk.
2. Properly 'install' compressed metadata downloaded. The compressed version of the rolename was added to the fileinfo store, which prevented detection of changed metadata and would unintentionally cause compressed metadata to always refresh (only the fileinfo of uncompressed metadata is stored and compared.)
3. Rename unsafely_verify_compressed_metadata_file() variable names, so that they match the other verify functions.
Reject downloaded metadata as early as possible. The top-level roles were all downloaded as a group and then their
expiration inspected. All metadata provided by a mirror that has already expired is discarded immediately and the next mirror tried. The update process stops if a requested role cannot be successfully validated, or one of its parents.
[2014-04-29 02:00:32,308 UTC] [tuf.download] [INFO] [_download_file:745@download.py]
Downloading: http://localhost:8001/metadata/timestamp.json
[2014-04-29 02:00:32,324 UTC] [tuf.download] [INFO] [_check_downloaded_length:676@download.py]
Downloaded 544 bytes out of an upper limit of 16384 bytes.
[2014-04-29 02:00:32,324 UTC] [tuf.client.updater] [INFO] [_get_file:1189@updater.py]
Not decompressing http://localhost:8001/metadata/timestamp.json
[2014-04-29 02:00:32,331 UTC] [tuf.download] [INFO] [_download_file:745@download.py]
Downloading: http://localhost:8001/metadata/snapshot.json
[2014-04-29 02:00:32,333 UTC] [tuf.download] [INFO] [_check_downloaded_length:654@download.py]
Downloaded 1003 bytes out of the expected 1003 bytes.
[2014-04-29 02:00:32,334 UTC] [tuf.client.updater] [INFO] [_get_file:1189@updater.py]
Not decompressing http://localhost:8001/metadata/snapshot.json
[2014-04-29 02:00:32,334 UTC] [tuf.client.updater] [INFO] [_check_hashes:696@updater.py]
The file's sha256 hash is correct: 5b3aec7cf295a25e4b39d875c7474511da9645bc6d27f9e86fb7e439c82e0ec7
[2014-04-29 02:00:32,335 UTC] [tuf.client.updater] [ERROR] [_ensure_not_expired:1789@updater.py]
Metadata 'snapshot' expired on Tue Apr 29 01:59:01 2014 (UTC).
Do not request, download, and install top-level roles if the root of trust has already expired after the inital load. If requested, update an expired root role:
[2014-04-29 01:18:02,457 UTC] [tuf.client.updater] [ERROR] [_ensure_not_expired:1789@updater.py]
Metadata 'root' expired on Mon Apr 28 23:23:57 2014 (UTC).
[2014-04-29 01:18:02,458 UTC] [tuf.client.updater] [INFO] [refresh:628@updater.py]
Expired Root metadata was loaded from disk. Try to update it now.
[2014-04-29 01:18:02,458 UTC] [tuf.download] [INFO] [_download_file:745@download.py]
Downloading: http://localhost:8001/metadata/root.json
[2014-04-29 01:18:02,461 UTC] [tuf.download] [INFO] [_check_downloaded_length:676@download.py]
Downloaded 1198 bytes out of an upper limit of 512000 bytes.
[2014-04-29 01:18:02,461 UTC] [tuf.client.updater] [INFO] [_get_file:1189@updater.py]
Not decompressing http://localhost:8001/metadata/root.json
[2014-04-29 01:18:02,462 UTC] [tuf.client.updater] [ERROR] [_ensure_not_expired:1789@updater.py]
Metadata 'root' expired on Mon Apr 28 23:23:57 2014 (UTC).
Note: An expired 'root' was provided by the server. The requested root must also be signed by keys trusted by the client.
Continue refactoring the test cases of test_updater.py.
Fix issue where repository_tool.py was not writing new compressed metadata.
Minor edits to TUF modules.
Add generate.py script and pre-generated repository files used by Unit tests (replacing repository_setup.py).
Refactor test_updater.py (10 test cases remaining).
Support the following installation setup:
# Minimal install intended for clients (pure Python, only ed25519 signatures).
pip install tuf
# Optional installation required by the TUF repository tools (RSA keys and rsassa-pss
# signatures, faster ed25519 cryptographic computations, general-purpose cryptography, etc.)
pip install tuf[tools]
Re-implement the fix for issue #167.
Update comments and docstrings related to the fixes above.
Update the remaining unit tests following the many design changes. The unit tests should next be factored to use repository_tool.py, avoid pseudo repositories and repository objects, monkey patching, etc.
add_key() -> add_verification_key()
remove_key() -> remove_verification_key()
release.json -> snapshot.json
Update repository_tool-diagram.png
Update README following the renamed functions and release role changes.
Minor edits, such as removing lint left over from a previous merge conflict.
repository_tool.py verified that valid metadata was written, properly signed, hashes all matched, and required roles created. The only check missing was verification of delegated target paths according the parent's delegations. Now, target paths are validated before write() is called for both 'path' and 'path_hash_prefixes'. updater.py has always properly verified paths, only repository_tool.py needed the support. If any of the delegated paths are not allowed, write()
raises the following exception:
tuf.ForbiddenTargetError: Role 'targets/unclaimed' specifies target '/simple/bad_file.txt', which is not an allowed path according to the delegations set by its parent role.
Allow leading os.sep for target and restricted paths listed in metadata. Previously, '{repository directory}/targets/file.txt' was listed as 'file.txt' and unsuccessfully compared against '/', or the root-most targets directory. Now, '/file.txt' & '/' and similar examples may be compared and verified.
Fix issue where a hashed bin with empty targets was flagged as invalid because its targets did not produce a path hash prefix that matched any of those listed in the parent role. Uncovered with _ensure_all_targets_allowed().
Minor edits to code that compared paths and and used path separators.
Add the add_restricted_paths() method.
Rename libtuf.py
Update README.
Update delegate_hashed_bins() docstring.
More testing of hashed bins and consistent snapshots.
Remove old scripts from setup.py.
Support multiple hash algorithms, where the generated digests of metadata and
target files is included in metadata (and filenames if 'consistent_snapshots'
is True). Previously, only a single hash algorithm was supported, and it was
set by default to 'sha256' in code. Repository maintainers may now choose any,
and/or multiple, hash algorithms from those supported by TUF. By default,
'sha256' is used when generating digests.
Support the recent change to the TUF specification, where writing consistent
snapshots may include N versions of identical metadata and targets, if N hash
algorithms is used by the repository when generating metadata.
Update code affected by the recent changes to the specification, such as
targets that may include digests in their filename.
Support consistent snapshots of compressed metadata, including repositories
that provide multiple versions of metadata with different digests included
in their filenames.
The repository tools can now load repositories that include consistent snapshots
of metadata and targets, including those with multiple (i.e., multiple digests
prepended to filenames) consistent snapshots of files.
The client code may now read repositories with 'consistent_snapshots': true in
Root metadata, and properly request and update files with digests included.
Move 'tuf.client.updater.Updater.refresh()' call in interposition to its updater's __init__(). The refresh() was previously executed on every target request.
Add tuf.interposition.refresh() for integrations that may require a manual refresh of top-level metadata (rare).
Improve targets_of_role(rolename) behavior. It now updates the minimum metadata needed for 'rolename'.
Ensure all_targets() adds targets in the expected order, where parent role targets come before children.
Minor updates to comments and docstrings.
Issue 147: Finalize conversion of all written metadata behavior. This commit ensures that compressed and uncompressed metadata is also written as outlined in the issue.
Issue 158: As requested, updater.refresh() may now unsafely fetch (i.e., unknown file size and hash) Root metadata if valid top-level metadata cannot be downloaded successfully (e.g., top-level keys may have been revoked). The repository must also sign the new Root file (at least until all clients have updated) with any revoked keys so that clients may successfully update. After unsafely updating Root, the top-level metadata is updated again as normal (and only once to avoid an infinite loop). By default, refresh() unsafely updates Root if only invalid top-level metadata can be downloaded, although this behavior may be overriden by the caller if they wish. Changed default behavior: refresh(self, unsafely_update_root_if_necessary=True)
Issue 165: Delegated roles are no longer added as attributes of a Targets object by libtuf.py (e.g., repository.targets.delegated_role). The previous bahavior restricted rolenames to Python identifiers (i.e., can only include letters, numbers, the underscore character, and must start with a nonnumeric character). Now, delegated roles may be referenced as strings (e.g., repository.targets('recently-claimed')) and include characters other than '_'. In addition, methods have been added to return all the delegated rolesnames of a target (e.g., repository.targets.get_delegated_rolenames()) and the immediate delegated Target objects of a role. Previous behavior: repository.targets.unclaimed.django.version = 8
Current behavior: repository.targets('unclaimed')('django').version = 8.
