Using securesystemslib.settings.HASH_ALGORITHMS is undersirable, because
it binds tuf to an implementation detail of the underlying library.
In this specific instance of file hashing algorithms it's even more
undesirable because it's overloading the intended use of the setting
which is "algorithm(s) [...] used to generate key IDs".
Add a new setting tuf.settings.FILE_HASH_ALGORITHMS, with a default
value of ['sha256', 'sha512'] (that matches the current value of
securesystemslib.settings.HASH_ALGORITHMS), to be used for file
hashing operations in tuf.
Signed-off-by: Joshua Lock <jlock@vmware.com>
tests/simple_server.py was copied to tuf/scripts/ to "make testing
easier" (cf84d3f51f), although with
the current test setup the original (and recently patched to fix an
Windows/Py2 test issue) test simple_server.py can be used just as
well.
This commit:
- removes tuf/scripts/simple_server.py
Note: that version slightly differed from the original test
server, probably due to demands by the linter that is only executed
on the tuf core code and not on the tests. However, for the testing
purposes of simple_server.py these changes (i.e., `SystemRandom()`,
`if __name__ =='__main__':`) are not necessary.
- updates the tests that used tuf.scripts.simple_server to instead
use tests.simple_server,
- updates setup.py to not install the simple_server module as
script, when installing tuf, as it is only a testing script and
not meant for end-user usage.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Since #885 the tests in TestUpdater and TestKeyRevocation fail on
Appveyor Python 2.7 builds. After some live debugging, it turns out
that the tests fail due to the extra amount of http requests to
the simple http server (see tests/simple_server.py) that were
added in #885.
The simple server runs in a subprocess and is re-used for the
entire TestCase. After a certain amount of requests it becomes
unresponsive. Note that neither the subprocess exits (ps -W), nor
does the port get closed (netstat -a). It just doesn't serve the
request, making it time out and fail the test.
The following script can be used to reproduce the issue (run in
tests directory):
```python
import subprocess
import requests
import random
counter = 0
port = random.randint(30000, 45000)
command = ['python', 'simple_server.py', str(port)]
server_process = subprocess.Popen(command, stderr=subprocess.PIPE)
url = 'http://localhost:'+str(port) + '/'
sess = requests.Session()
try:
while True:
sess.get(url, timeout=3)
counter +=1
finally:
print(counter)
server_process.kill()
```
It fails repeatedly on the 69th request, but only if
`stderr=subprocess.PIPE` is passed to Popen. Given that for each
request the simple server writes about ~60 characters to stderr,
e.g. ...
```
127.0.0.1 - - [24/Feb/2020 12:01:23] "GET / HTTP/1.1" 200 -
```
... it looks a lot like a full pipe buffer of size 4096. Note that the
`bufsize` argument to Popen does not change anything.
As a simple work around we silence the test server on
Windows/Python2 to not fill the buffer.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Follows up on #978, which had the following problems:
- too many requirements files (cc @trishankatdatadog ;)
- used extra tooling around pip-compile that
- didn't take into account requirement markers (see comments
in requirements.txt in this commit), and
- confused Dependabot, which expects the hashed requirements
file in a certain format, as pip-compile would generate it
without custom tooling (see #979).
This commit restructures the requirements files as follows:
- Merges requirements-tox.txt and requirements-test.txt. The
separation was semantically correct but operationally irrelevant.
- Removes the hashed requirements file, which doesn't add much
security, especially with PEP 458 on the way (see python/peps#1306),
but extra maintenance (see notes about requirements.txt in #978
and about Dependabot above)
- Manually adds environment markers to requirements-pinned.txt (see
comments in requirements.txt in this commit).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
colorama is now an optional dependency for securesystemslib, and so isn't
installed by default. However, the repo script uses colorama and doesn't
handle its absence - add securesystemslib[colors] to ensure colorama is
installed.
Signed-off-by: Joshua Lock <jlock@vmware.com>
The PyUp GitHub integration for TUF stopped working a few days
ago. Instead of troubleshooting, I'm seizing the opportunity to
replace it with Dependabot, which has shown to work well in the
in-toto org.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
* Use suffixed instead of prefixed sub-requirements files to group
them alphabetically in the file tree.
* Layer requirements files akin to the in-toto project
(see in-toto/in-toto#294). The hierarchy is:
- *requirements.in*
tuf runtime requirements, including optional requirements
(pynacl and cyrptography)
- *requirements-pinned.txt*
pinned tuf runtime requirements, including optional
and transitive (1 level deep) requirements and their hashes.
The file is generated semi-automatically using pip-compile
and a bash script (see document header), based off of
requirements.in, combining requirements from all supported
Python versions.
This file should be auto-updated, by e.g. dependabot, and be used
for ci/cd tests, to catch issues with new dependencies.
- *requirements-test.txt*
additional test runtime requirements
- *requirements-tox.txt*
combines requirements.txt, requirements-test.txt and additional
test tools (for linting and coverage), i.e. everything that is
needed in each tox environment to run the tests.
- *requirements-dev.txt*
lists tox for local development and testing, and also
requirements-tox.txt and tuf in editable mode to run
the test suite or individual tests directly.
- *requirements.txt*
requirements-pinned.txt with the hashes of the dependencies
as reported by pip at the time of creating the file.
NOTE: this is not used for testing or dev-install because pip
doesn't allow mixed (with and without hashes) installations.
This file should also be auto-updated, by e.g. dependabot.
* Removes an obsolete version constraint on coverage
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Prior to this commit metadadata signature verification as provided
by `tuf.sig.verify()` and used e.g. in `tuf.client.updater` counted
multiple signatures with identical authorized keyids each
separately towards the threshold. This behavior practically
subverts the signature thresholds check.
This commit fixes the issue by counting identical authorized keyids
only once towards the threshold.
The commit further clarifies the behavior of the relevant functions
in the `sig` module, i.e. `get_signature_status` and `verify` in
their respective docstrings. And adds tests for those functions and
also for the client updater.
---
NOTE: With this commit signatures with different authorized keyids
still each count separately towards the threshold, even if the
keyids identify the same key. If this behavior is not desired, I
propose the following fix instead. It verifies uniqueness of keys
(and not keyids):
```
diff --git a/tuf/sig.py b/tuf/sig.py
index ae9bae15..5392e596 100755
--- a/tuf/sig.py
+++ b/tuf/sig.py
@@ -303,7 +303,14 @@ def verify(signable, role, repository_name='default', threshold=None,
if threshold is None or threshold <= 0: #pragma: no cover
raise securesystemslib.exceptions.Error("Invalid threshold: " + repr(threshold))
- return len(good_sigs) >= threshold
+ # Different keyids might point to the same key
+ # To be safe, check against unique public key values
+ unique_good_sig_keys = set()
+ for keyid in good_sigs:
+ key = tuf.keydb.get_key(keyid, repository_name)
+ unique_good_sig_keys.add(key["keyval"]["public"])
+
+ return len(unique_good_sig_keys) >= threshold
```
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
TUF does not reliably mark roles as dirty whose metadata needs
to be re-generated.
Only roles that have changed are marked as dirty, but sometimes
roles metadata needs to be updated, although the role wasn't
changed directly (see #958).
Furthermore, the tutorial assumes at one point that the reader
leaves and re-enter the interpreter session, being forced to reload the
signing keys, roles that later need to be re-written, are marked as
dirty. If the reader does not leave the interpreter, the roles are
not marked as dirty (see #964).
To not confuse the reader with flawed state-keeping, and to never
write an inconsistent repository to disk, the tutorial lets the
reader explicitly mark all roles that need to be re-written as
"dirty".
This can be changed once above issues are fixed.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Explain and show output of delegate_hashed_bins() function call in
tutorial snippet.
Also update the subsequent comment for better continuity.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Ask the reader to ignore a misleading warning about the location of
a delegation path pattern.
The comment may be removed when fixing the warning in
theupdateframework/tuf#963.
- Comment out text that has become obsolete when commenting out
the "Revoke Delegated Role" section (in an earlier commit).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This should fix the following build error:
error in tuf setup command: 'tests_require' must be a string or
list of strings containing valid project/version requirement
specifiers; Expected version spec in mock; python_version < "3.3"
at ; python_version < "3.3"
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add, remove and update function calls to match code snippets in
tutorial.
This commit also adds tests for outputs of `repo.status()` and
`repo.dirty_roles()` functions.
Note that the compare-to strings need to be constructed
programatically, akin to how they are constructed in the relevant
functions, in order to avoid issues with unicode prefixes in
Python2/3, e.g.
"Dirty roles: ['root']"
vs "Dirty roles: [u'root']"
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
roledb.get_dirty_roles(repo_name) returns the list representation
of the global _dirty_roles[repo_name] set. To make the return value
deterministic this commit sorts the list before returning it.
The commit also removes calls to sorted on the return value of
get_dirty_roles in test_roledb.py and test_repository_tool.py.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Instead of keeping track of files created during the tutorial and
removing them afterwards, this commit updates the test case to
create and change into a temporary directory in setUp and
change back and remove the tempdir in tearDown.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The text above the snippet explains the basic idea of "consistent
snapshots" and how to generate them with `write` and `writeall`.
The commands in the snippet just leave the repo in an inconsistent
state (see comment).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Fix expected output
- Update comments
- Add a few additional calls, to help the reader understand the
repo state
- Also see theupdateframework/tuf#958
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
In "Targets" section:
- Remove `repository.targets('<delegated rolename>').add_target(...)`
command, because it is not copy-pastable and delegations
have not yet been covered at that point.
- Update the "remove targets" snippet to remove the previously added
"myproject/file4.txt" instead of "file3.txt", because we will add
"myproject/file4.txt" to the delegated "unclaimed" role in the
Delegation section.
In "Delegation" section:
- Change "unclaimed" delegation pattern from 'foo*.tgz', for which
no file exists in the tutorial, to 'myproject/*.txt'.
- Add "myproject/file4.txt" to the delegated unclaimed targets role
- Remove the command that updates the version of the "unclaimed"
role, because this should not be done manually, and the add_target
call shows just as well how to access delegated roles.
- Comment out the revoke delgated role section, leaving a TODO note
for required updates (should be ticketized).
In "Delegate to Hashed Bins":
- Add call to remove target "myproject/file4.txt" from "unclaimed",
because it is further delgated to hashed bins
- Add dirty_roles() call to show all the newly created bins
- Add mark_dirty() and writeall() calls to create a consistent
state of the repo
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
