An up-to-date version of the minimal client usage instructions
from client_setup_and_repository_example.md can be found in
the main TUF tutorial.
More comprehensive documentation of the tuf client can be found in
tuf/client/README.md, which also needs a revision.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove link to incomplete and severely outdated
client_setup_and_repository_example.md in client section of
TUTORIAL.md.
Instead we should link (or move the entire client tutorial part) to
tuf/client/README.md, which is more comprehensive and less outdated
than above document (see #808).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove logic for handling of root metadata in _update_metadata_if_changed()
as root metadata is no longer fetched with this function, instead
_update_root_metadata() serves this purpose.
Additionally remove redundant mention of root metadata in a TODO comment.
Signed-off-by: Joshua Lock <jlock@vmware.com>
One of the created target files has its file permissions encoded in the
targets metadata via the custom attribute of the add_target() function.
On Linux-based OS the umask value of the environment the script is run
in can result in different octal permissions for the created file, i.e.
on Fedora the default umask is 0002 (default permissions 664) whereas
on Debian/Ubuntu the default umask is 0022 (default permissions 644).
Explicitly chown 'file1' to octal permissions 644 so that the generated
data has the same custom attributes for targets regardless of which
Linux host they are generated on.
Signed-off-by: Joshua Lock <jlock@vmware.com>
* Fix the path referenced in the Purpose
* Change add_target() calls to pass file paths relative to targets dir
Signed-off-by: Joshua Lock <jlock@vmware.com>
The workflow for downloading metadata for top-level roles has changed.
Root is now updated and verified by stepping through a chain of trust
based on the currently available root metadata. For that reason
root.json is no longer needed in snapshot and has been dropped from
there per theupdateframework/specification#40
Update docstrings and comments in the Updater object to reflect the
correct flow of metadata updates:
root (if necessary) -> timestamp -> snapshot -> targets
Signed-off-by: Joshua Lock <jlock@vmware.com>
In PR #40 aginst the specification "root.json" has been removed from
the meta dictionary in "snapshot.json".
Update generate_snapshot_metadata() to no longer add an entry for
root.json to root.json
Signed-off-by: Joshua Lock <jlock@vmware.com>
In tutorial code snippet fix a call to repository.mark_dirty that
passes a non-existing bin role name raising:
"tuf.exceptions.UnknownRoleError: Role name does not exist: 90-7"
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Timestamp.json includes a METAFILES entry for snapshot.json. METAFILES
includes HASHES:
"HASHES is the dictionary that specifies one or more hashes, including the
cryptographic hash function. For example: { "sha256": HASH, ... }."
We've been hard-coding this to a single sha256 hash, as that's the default
algorithms argument of securesystemlib.util.get_file_details() -- this
feels wrong. Change to using the new tuf.settings.FILE_HASH_ALGORITHMS
setting.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Using securesystemslib.settings.HASH_ALGORITHMS is undersirable, because
it binds tuf to an implementation detail of the underlying library.
In this specific instance of file hashing algorithms it's even more
undesirable because it's overloading the intended use of the setting
which is "algorithm(s) [...] used to generate key IDs".
Add a new setting tuf.settings.FILE_HASH_ALGORITHMS, with a default
value of ['sha256', 'sha512'] (that matches the current value of
securesystemslib.settings.HASH_ALGORITHMS), to be used for file
hashing operations in tuf.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Replace hard-coded logger names with __name__. For the most part this just uses
the standard conventions to create the same logger hierarchy as existed before.
The only real difference is that loggers created for printing during tests are
no longer part of the 'tuf' hierarchy.
Signed-off-by: Joshua Lock <jlock@vmware.com>
tests/simple_server.py was copied to tuf/scripts/ to "make testing
easier" (cf84d3f51f), although with
the current test setup the original (and recently patched to fix an
Windows/Py2 test issue) test simple_server.py can be used just as
well.
This commit:
- removes tuf/scripts/simple_server.py
Note: that version slightly differed from the original test
server, probably due to demands by the linter that is only executed
on the tuf core code and not on the tests. However, for the testing
purposes of simple_server.py these changes (i.e., `SystemRandom()`,
`if __name__ =='__main__':`) are not necessary.
- updates the tests that used tuf.scripts.simple_server to instead
use tests.simple_server,
- updates setup.py to not install the simple_server module as
script, when installing tuf, as it is only a testing script and
not meant for end-user usage.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Since #885 the tests in TestUpdater and TestKeyRevocation fail on
Appveyor Python 2.7 builds. After some live debugging, it turns out
that the tests fail due to the extra amount of http requests to
the simple http server (see tests/simple_server.py) that were
added in #885.
The simple server runs in a subprocess and is re-used for the
entire TestCase. After a certain amount of requests it becomes
unresponsive. Note that neither the subprocess exits (ps -W), nor
does the port get closed (netstat -a). It just doesn't serve the
request, making it time out and fail the test.
The following script can be used to reproduce the issue (run in
tests directory):
```python
import subprocess
import requests
import random
counter = 0
port = random.randint(30000, 45000)
command = ['python', 'simple_server.py', str(port)]
server_process = subprocess.Popen(command, stderr=subprocess.PIPE)
url = 'http://localhost:'+str(port) + '/'
sess = requests.Session()
try:
while True:
sess.get(url, timeout=3)
counter +=1
finally:
print(counter)
server_process.kill()
```
It fails repeatedly on the 69th request, but only if
`stderr=subprocess.PIPE` is passed to Popen. Given that for each
request the simple server writes about ~60 characters to stderr,
e.g. ...
```
127.0.0.1 - - [24/Feb/2020 12:01:23] "GET / HTTP/1.1" 200 -
```
... it looks a lot like a full pipe buffer of size 4096. Note that the
`bufsize` argument to Popen does not change anything.
As a simple work around we silence the test server on
Windows/Python2 to not fill the buffer.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Follows up on #978, which had the following problems:
- too many requirements files (cc @trishankatdatadog ;)
- used extra tooling around pip-compile that
- didn't take into account requirement markers (see comments
in requirements.txt in this commit), and
- confused Dependabot, which expects the hashed requirements
file in a certain format, as pip-compile would generate it
without custom tooling (see #979).
This commit restructures the requirements files as follows:
- Merges requirements-tox.txt and requirements-test.txt. The
separation was semantically correct but operationally irrelevant.
- Removes the hashed requirements file, which doesn't add much
security, especially with PEP 458 on the way (see python/peps#1306),
but extra maintenance (see notes about requirements.txt in #978
and about Dependabot above)
- Manually adds environment markers to requirements-pinned.txt (see
comments in requirements.txt in this commit).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
colorama is now an optional dependency for securesystemslib, and so isn't
installed by default. However, the repo script uses colorama and doesn't
handle its absence - add securesystemslib[colors] to ensure colorama is
installed.
Signed-off-by: Joshua Lock <jlock@vmware.com>
The PyUp GitHub integration for TUF stopped working a few days
ago. Instead of troubleshooting, I'm seizing the opportunity to
replace it with Dependabot, which has shown to work well in the
in-toto org.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
* Use suffixed instead of prefixed sub-requirements files to group
them alphabetically in the file tree.
* Layer requirements files akin to the in-toto project
(see in-toto/in-toto#294). The hierarchy is:
- *requirements.in*
tuf runtime requirements, including optional requirements
(pynacl and cyrptography)
- *requirements-pinned.txt*
pinned tuf runtime requirements, including optional
and transitive (1 level deep) requirements and their hashes.
The file is generated semi-automatically using pip-compile
and a bash script (see document header), based off of
requirements.in, combining requirements from all supported
Python versions.
This file should be auto-updated, by e.g. dependabot, and be used
for ci/cd tests, to catch issues with new dependencies.
- *requirements-test.txt*
additional test runtime requirements
- *requirements-tox.txt*
combines requirements.txt, requirements-test.txt and additional
test tools (for linting and coverage), i.e. everything that is
needed in each tox environment to run the tests.
- *requirements-dev.txt*
lists tox for local development and testing, and also
requirements-tox.txt and tuf in editable mode to run
the test suite or individual tests directly.
- *requirements.txt*
requirements-pinned.txt with the hashes of the dependencies
as reported by pip at the time of creating the file.
NOTE: this is not used for testing or dev-install because pip
doesn't allow mixed (with and without hashes) installations.
This file should also be auto-updated, by e.g. dependabot.
* Removes an obsolete version constraint on coverage
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Prior to this commit metadadata signature verification as provided
by `tuf.sig.verify()` and used e.g. in `tuf.client.updater` counted
multiple signatures with identical authorized keyids each
separately towards the threshold. This behavior practically
subverts the signature thresholds check.
This commit fixes the issue by counting identical authorized keyids
only once towards the threshold.
The commit further clarifies the behavior of the relevant functions
in the `sig` module, i.e. `get_signature_status` and `verify` in
their respective docstrings. And adds tests for those functions and
also for the client updater.
---
NOTE: With this commit signatures with different authorized keyids
still each count separately towards the threshold, even if the
keyids identify the same key. If this behavior is not desired, I
propose the following fix instead. It verifies uniqueness of keys
(and not keyids):
```
diff --git a/tuf/sig.py b/tuf/sig.py
index ae9bae15..5392e596 100755
--- a/tuf/sig.py
+++ b/tuf/sig.py
@@ -303,7 +303,14 @@ def verify(signable, role, repository_name='default', threshold=None,
if threshold is None or threshold <= 0: #pragma: no cover
raise securesystemslib.exceptions.Error("Invalid threshold: " + repr(threshold))
- return len(good_sigs) >= threshold
+ # Different keyids might point to the same key
+ # To be safe, check against unique public key values
+ unique_good_sig_keys = set()
+ for keyid in good_sigs:
+ key = tuf.keydb.get_key(keyid, repository_name)
+ unique_good_sig_keys.add(key["keyval"]["public"])
+
+ return len(unique_good_sig_keys) >= threshold
```
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
