Add some additional checks to test_add_target_to_bin to ensure the code
to add a target passing a fileinfo is tested.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Factor out code to calculate the prefix length, total number of prefixes
and number of prefixes to be allocated to each bin into a helper
function _get_bin_numbers() so that we a) do not duplicate logic,
b) can document the logic in one place and c) ensure the logic is
consistent.
Signed-off-by: Joshua Lock <jlock@vmware.com>
We intend to deprecate the custom parameter of add_target() in favour
of using the fileinfo parameter with the custom value populated,
therefore it does not make sense to _add_ the custom parameter to
add_target_to_bin()
Signed-off-by: Joshua Lock <jlock@vmware.com>
When testing delegate_hashed_bins to ensure that hash_path_prefixes
map to the generated name of the bin, also check to ensure that at least
one of the delegations contains one or more path_hash_prefixes.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Test the newly added functionality to:
* add a target to the repository without access to the target file on disk
* write targets metadata without access to target files on disk, by using
the existing fileinfo data from the roledb
Signed-off-by: Joshua Lock <jlock@vmware.com>
Merge the logger calls reporting information about the hashed bin
delegations into a single logger.info() call to ensure the messages
will be grouped together even when integrated into a logging system
with multiple parallel sources.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Previously, at the time of writing targets metadata, we accessed all
targets files and ensured they exist on disk before generating a
filinfo dict containing hashes of the target file.
This change enables targets metadata to be generated using the
fileinfo data that is already stored in the roledb.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add an additional optional parameter to add_target() and
add_target_to_bin() which is a fileinfo object matching
tuf.formats.FILEINFO_OBJECT
This parameter and the custom parameter are mutually exclusive and
thus cannot be passed at the same time.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Match the pattern in add_target() where if the filepath already exists
in roleinfo['paths'] it is updated to replace the existing entry with
the new fileinfo.
Signed-off-by: Joshua Lock <jlock@vmware.com>
The file isn't strictly needed on-disk at the time add_target() and
add_targets() are called and this duplicates the check for the file's
presence in write[_all]()
By removing this check we allow extra versatility in adding targets.
Signed-off-by: Joshua Lock <jlock@vmware.com>
In delegate_to_hashed_bins() duplicate the code which determines the
target's file path for writing to the metadata in repository_lib's
generate_targets_metadata to ensure that the target path hashed by
delegate_hashed_bins() matches the target path in the metadata file.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Vastly simplify the implementation, using the _get_hash() and
_find_bin_for_hash() helpers added in earlier commits.
Furthermore, enable passing of the custom parameter to
add_target_to_bin() to better match add_target()
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add a helper function to determine the name of a bin that a hashed
targetfile will be delegated to.
Based sketches by Lukas Puehringer in issues #994 & #995
Signed-off-by: Joshua Lock <jlock@vmware.com>
This helper will generate a hash of the passed target_filepath using the
algorithm defined by the repository_tool's HASH_FUNCTION variable.
Signed-off-by: Joshua Lock <jlock@vmware.com>
As we are adding and removing items from the hashed bins and checking
for their presence/absence it's simplest if we being with the hashed
bins initially empty.
If we pass a list of targets when we call delgate_hashed_bins() the
delegated roles have an initial set of targets delegated to them,
which complicates testing of adding then removing a target to a
delegated bin.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add test to ensure delegated bin names are consistent with the hash
prefixes that are delegated to the role.
This is an implicit assumption of the current implementation, the
testing of which will enable us to modify the code with greater
confidence.
Signed-off-by: Joshua Lock <jlock@vmware.com>
An up-to-date version of the minimal client usage instructions
from client_setup_and_repository_example.md can be found in
the main TUF tutorial.
More comprehensive documentation of the tuf client can be found in
tuf/client/README.md, which also needs a revision.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove link to incomplete and severely outdated
client_setup_and_repository_example.md in client section of
TUTORIAL.md.
Instead we should link (or move the entire client tutorial part) to
tuf/client/README.md, which is more comprehensive and less outdated
than above document (see #808).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove logic for handling of root metadata in _update_metadata_if_changed()
as root metadata is no longer fetched with this function, instead
_update_root_metadata() serves this purpose.
Additionally remove redundant mention of root metadata in a TODO comment.
Signed-off-by: Joshua Lock <jlock@vmware.com>
One of the created target files has its file permissions encoded in the
targets metadata via the custom attribute of the add_target() function.
On Linux-based OS the umask value of the environment the script is run
in can result in different octal permissions for the created file, i.e.
on Fedora the default umask is 0002 (default permissions 664) whereas
on Debian/Ubuntu the default umask is 0022 (default permissions 644).
Explicitly chown 'file1' to octal permissions 644 so that the generated
data has the same custom attributes for targets regardless of which
Linux host they are generated on.
Signed-off-by: Joshua Lock <jlock@vmware.com>
* Fix the path referenced in the Purpose
* Change add_target() calls to pass file paths relative to targets dir
Signed-off-by: Joshua Lock <jlock@vmware.com>
The workflow for downloading metadata for top-level roles has changed.
Root is now updated and verified by stepping through a chain of trust
based on the currently available root metadata. For that reason
root.json is no longer needed in snapshot and has been dropped from
there per theupdateframework/specification#40
Update docstrings and comments in the Updater object to reflect the
correct flow of metadata updates:
root (if necessary) -> timestamp -> snapshot -> targets
Signed-off-by: Joshua Lock <jlock@vmware.com>
In PR #40 aginst the specification "root.json" has been removed from
the meta dictionary in "snapshot.json".
Update generate_snapshot_metadata() to no longer add an entry for
root.json to root.json
Signed-off-by: Joshua Lock <jlock@vmware.com>
In tutorial code snippet fix a call to repository.mark_dirty that
passes a non-existing bin role name raising:
"tuf.exceptions.UnknownRoleError: Role name does not exist: 90-7"
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Timestamp.json includes a METAFILES entry for snapshot.json. METAFILES
includes HASHES:
"HASHES is the dictionary that specifies one or more hashes, including the
cryptographic hash function. For example: { "sha256": HASH, ... }."
We've been hard-coding this to a single sha256 hash, as that's the default
algorithms argument of securesystemlib.util.get_file_details() -- this
feels wrong. Change to using the new tuf.settings.FILE_HASH_ALGORITHMS
setting.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Using securesystemslib.settings.HASH_ALGORITHMS is undersirable, because
it binds tuf to an implementation detail of the underlying library.
In this specific instance of file hashing algorithms it's even more
undesirable because it's overloading the intended use of the setting
which is "algorithm(s) [...] used to generate key IDs".
Add a new setting tuf.settings.FILE_HASH_ALGORITHMS, with a default
value of ['sha256', 'sha512'] (that matches the current value of
securesystemslib.settings.HASH_ALGORITHMS), to be used for file
hashing operations in tuf.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Replace hard-coded logger names with __name__. For the most part this just uses
the standard conventions to create the same logger hierarchy as existed before.
The only real difference is that loggers created for printing during tests are
no longer part of the 'tuf' hierarchy.
Signed-off-by: Joshua Lock <jlock@vmware.com>
tests/simple_server.py was copied to tuf/scripts/ to "make testing
easier" (cf84d3f51f), although with
the current test setup the original (and recently patched to fix an
Windows/Py2 test issue) test simple_server.py can be used just as
well.
This commit:
- removes tuf/scripts/simple_server.py
Note: that version slightly differed from the original test
server, probably due to demands by the linter that is only executed
on the tuf core code and not on the tests. However, for the testing
purposes of simple_server.py these changes (i.e., `SystemRandom()`,
`if __name__ =='__main__':`) are not necessary.
- updates the tests that used tuf.scripts.simple_server to instead
use tests.simple_server,
- updates setup.py to not install the simple_server module as
script, when installing tuf, as it is only a testing script and
not meant for end-user usage.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Since #885 the tests in TestUpdater and TestKeyRevocation fail on
Appveyor Python 2.7 builds. After some live debugging, it turns out
that the tests fail due to the extra amount of http requests to
the simple http server (see tests/simple_server.py) that were
added in #885.
The simple server runs in a subprocess and is re-used for the
entire TestCase. After a certain amount of requests it becomes
unresponsive. Note that neither the subprocess exits (ps -W), nor
does the port get closed (netstat -a). It just doesn't serve the
request, making it time out and fail the test.
The following script can be used to reproduce the issue (run in
tests directory):
```python
import subprocess
import requests
import random
counter = 0
port = random.randint(30000, 45000)
command = ['python', 'simple_server.py', str(port)]
server_process = subprocess.Popen(command, stderr=subprocess.PIPE)
url = 'http://localhost:'+str(port) + '/'
sess = requests.Session()
try:
while True:
sess.get(url, timeout=3)
counter +=1
finally:
print(counter)
server_process.kill()
```
It fails repeatedly on the 69th request, but only if
`stderr=subprocess.PIPE` is passed to Popen. Given that for each
request the simple server writes about ~60 characters to stderr,
e.g. ...
```
127.0.0.1 - - [24/Feb/2020 12:01:23] "GET / HTTP/1.1" 200 -
```
... it looks a lot like a full pipe buffer of size 4096. Note that the
`bufsize` argument to Popen does not change anything.
As a simple work around we silence the test server on
Windows/Python2 to not fill the buffer.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
