Add some additional checks to test_add_target_to_bin to ensure the code
to add a target passing a fileinfo is tested.
Signed-off-by: Joshua Lock <jlock@vmware.com>
When testing delegate_hashed_bins to ensure that hash_path_prefixes
map to the generated name of the bin, also check to ensure that at least
one of the delegations contains one or more path_hash_prefixes.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Test the newly added functionality to:
* add a target to the repository without access to the target file on disk
* write targets metadata without access to target files on disk, by using
the existing fileinfo data from the roledb
Signed-off-by: Joshua Lock <jlock@vmware.com>
Merge the logger calls reporting information about the hashed bin
delegations into a single logger.info() call to ensure the messages
will be grouped together even when integrated into a logging system
with multiple parallel sources.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add an additional optional parameter to add_target() and
add_target_to_bin() which is a fileinfo object matching
tuf.formats.FILEINFO_OBJECT
This parameter and the custom parameter are mutually exclusive and
thus cannot be passed at the same time.
Signed-off-by: Joshua Lock <jlock@vmware.com>
The file isn't strictly needed on-disk at the time add_target() and
add_targets() are called and this duplicates the check for the file's
presence in write[_all]()
By removing this check we allow extra versatility in adding targets.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Vastly simplify the implementation, using the _get_hash() and
_find_bin_for_hash() helpers added in earlier commits.
Furthermore, enable passing of the custom parameter to
add_target_to_bin() to better match add_target()
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add a helper function to determine the name of a bin that a hashed
targetfile will be delegated to.
Based sketches by Lukas Puehringer in issues #994 & #995
Signed-off-by: Joshua Lock <jlock@vmware.com>
As we are adding and removing items from the hashed bins and checking
for their presence/absence it's simplest if we being with the hashed
bins initially empty.
If we pass a list of targets when we call delgate_hashed_bins() the
delegated roles have an initial set of targets delegated to them,
which complicates testing of adding then removing a target to a
delegated bin.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add test to ensure delegated bin names are consistent with the hash
prefixes that are delegated to the role.
This is an implicit assumption of the current implementation, the
testing of which will enable us to modify the code with greater
confidence.
Signed-off-by: Joshua Lock <jlock@vmware.com>
One of the created target files has its file permissions encoded in the
targets metadata via the custom attribute of the add_target() function.
On Linux-based OS the umask value of the environment the script is run
in can result in different octal permissions for the created file, i.e.
on Fedora the default umask is 0002 (default permissions 664) whereas
on Debian/Ubuntu the default umask is 0022 (default permissions 644).
Explicitly chown 'file1' to octal permissions 644 so that the generated
data has the same custom attributes for targets regardless of which
Linux host they are generated on.
Signed-off-by: Joshua Lock <jlock@vmware.com>
* Fix the path referenced in the Purpose
* Change add_target() calls to pass file paths relative to targets dir
Signed-off-by: Joshua Lock <jlock@vmware.com>
Replace hard-coded logger names with __name__. For the most part this just uses
the standard conventions to create the same logger hierarchy as existed before.
The only real difference is that loggers created for printing during tests are
no longer part of the 'tuf' hierarchy.
Signed-off-by: Joshua Lock <jlock@vmware.com>
tests/simple_server.py was copied to tuf/scripts/ to "make testing
easier" (cf84d3f51f), although with
the current test setup the original (and recently patched to fix an
Windows/Py2 test issue) test simple_server.py can be used just as
well.
This commit:
- removes tuf/scripts/simple_server.py
Note: that version slightly differed from the original test
server, probably due to demands by the linter that is only executed
on the tuf core code and not on the tests. However, for the testing
purposes of simple_server.py these changes (i.e., `SystemRandom()`,
`if __name__ =='__main__':`) are not necessary.
- updates the tests that used tuf.scripts.simple_server to instead
use tests.simple_server,
- updates setup.py to not install the simple_server module as
script, when installing tuf, as it is only a testing script and
not meant for end-user usage.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Since #885 the tests in TestUpdater and TestKeyRevocation fail on
Appveyor Python 2.7 builds. After some live debugging, it turns out
that the tests fail due to the extra amount of http requests to
the simple http server (see tests/simple_server.py) that were
added in #885.
The simple server runs in a subprocess and is re-used for the
entire TestCase. After a certain amount of requests it becomes
unresponsive. Note that neither the subprocess exits (ps -W), nor
does the port get closed (netstat -a). It just doesn't serve the
request, making it time out and fail the test.
The following script can be used to reproduce the issue (run in
tests directory):
```python
import subprocess
import requests
import random
counter = 0
port = random.randint(30000, 45000)
command = ['python', 'simple_server.py', str(port)]
server_process = subprocess.Popen(command, stderr=subprocess.PIPE)
url = 'http://localhost:'+str(port) + '/'
sess = requests.Session()
try:
while True:
sess.get(url, timeout=3)
counter +=1
finally:
print(counter)
server_process.kill()
```
It fails repeatedly on the 69th request, but only if
`stderr=subprocess.PIPE` is passed to Popen. Given that for each
request the simple server writes about ~60 characters to stderr,
e.g. ...
```
127.0.0.1 - - [24/Feb/2020 12:01:23] "GET / HTTP/1.1" 200 -
```
... it looks a lot like a full pipe buffer of size 4096. Note that the
`bufsize` argument to Popen does not change anything.
As a simple work around we silence the test server on
Windows/Python2 to not fill the buffer.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Prior to this commit metadadata signature verification as provided
by `tuf.sig.verify()` and used e.g. in `tuf.client.updater` counted
multiple signatures with identical authorized keyids each
separately towards the threshold. This behavior practically
subverts the signature thresholds check.
This commit fixes the issue by counting identical authorized keyids
only once towards the threshold.
The commit further clarifies the behavior of the relevant functions
in the `sig` module, i.e. `get_signature_status` and `verify` in
their respective docstrings. And adds tests for those functions and
also for the client updater.
---
NOTE: With this commit signatures with different authorized keyids
still each count separately towards the threshold, even if the
keyids identify the same key. If this behavior is not desired, I
propose the following fix instead. It verifies uniqueness of keys
(and not keyids):
```
diff --git a/tuf/sig.py b/tuf/sig.py
index ae9bae15..5392e596 100755
--- a/tuf/sig.py
+++ b/tuf/sig.py
@@ -303,7 +303,14 @@ def verify(signable, role, repository_name='default', threshold=None,
if threshold is None or threshold <= 0: #pragma: no cover
raise securesystemslib.exceptions.Error("Invalid threshold: " + repr(threshold))
- return len(good_sigs) >= threshold
+ # Different keyids might point to the same key
+ # To be safe, check against unique public key values
+ unique_good_sig_keys = set()
+ for keyid in good_sigs:
+ key = tuf.keydb.get_key(keyid, repository_name)
+ unique_good_sig_keys.add(key["keyval"]["public"])
+
+ return len(unique_good_sig_keys) >= threshold
```
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add, remove and update function calls to match code snippets in
tutorial.
This commit also adds tests for outputs of `repo.status()` and
`repo.dirty_roles()` functions.
Note that the compare-to strings need to be constructed
programatically, akin to how they are constructed in the relevant
functions, in order to avoid issues with unicode prefixes in
Python2/3, e.g.
"Dirty roles: ['root']"
vs "Dirty roles: [u'root']"
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
roledb.get_dirty_roles(repo_name) returns the list representation
of the global _dirty_roles[repo_name] set. To make the return value
deterministic this commit sorts the list before returning it.
The commit also removes calls to sorted on the return value of
get_dirty_roles in test_roledb.py and test_repository_tool.py.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Instead of keeping track of files created during the tutorial and
removing them afterwards, this commit updates the test case to
create and change into a temporary directory in setUp and
change back and remove the tempdir in tearDown.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Note that target filepaths specified in the repo use '/' even on
Windows.
(That property is important to make sure that metadata is platform-
independent.)
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
(that is, despite currently existing issue to be remedied in #774)
Currently, repository_tool.get_filepaths_in_directory yields
relative paths, not the absolute paths it promises in its docstring.
This test will now function despite this and continue to function
after #774 is merged.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
Because leading '/' is no longer allowed in target filenames in
target addition or delegation.
See https://github.com/theupdateframework/tuf/issues/639
While we're at it, remove some other unnecessary '/' characters
in repository creation and loading in the tutorial and tutorial
test.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
add_restricted_paths was renamed to add_path; however, this
function represents a problematic element of TUF that assumes
that roles are have a single delegator and delegatee, and that
one can refer to a role's expected keys without being concerned
about any delegation metadata....
So this is being removed from the tutorial. In time, add_paths
will either be removed or changed (to expect a delegator role
and a delegatee role, not just a delegatee role).
This comment does not do justice to the issue: please see TUF
GitHub Issue #660:
https://github.com/theupdateframework/tuf/issues/660
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
While it may be nice, the use of a keystore/ directory for the
temporary keys created in the tutorial complicates the code a
good bit when it's done in a portable way (tons of os.path.join()
calls), and that's not worth it. It also is a slight complication
in a tutorial that profits from being as simple as possible.
Tests will be run in multiple environments (including non-Linux
environments) and to leave so many extra subdirectory uses in the
tutorial means that the tutorial test will deviate over a large
number of lines from the TUTORIAL.md content it is intended to
test, which would be bad.
This commit adjusts both the tutorial doc and the regression test
for the tutorial.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
running test_tutorial.py attempts the commands replicated from
TUTORIAL.md. This should help us avoid breaking the tutorial with
future changes without noticing by having automated testing run
the tutorial and produce helpful output.
NOTE that this test currently fails because the tutorial is
currently broken!
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
The class has been removed from securesystemslib as its desirable
features are no longer required and its behavioural inconsistencies
with a standard Python file object are confusing. Therefore remove
uses of the class from TUF.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
securesystemslib PR #162 removed implicit encoding of data to bytes
in securesystemslib.keys.[create_signature|verify_signature]
Update to encode data where required.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Test that client does not rotate beyond a configured upper bound,
i.e. `current_version + MAX_NUMBER_ROOT_ROTATIONS`
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Test that a client whose root is outdated by multiple versions and
who has none of the latest nor next-to-latest root keys can still
update and does so by incrementally verifying all roots until the
most recent one.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This commit also updates the key loader helper in
test_updater_root_rotation_integration.py to load
the new keys too.
The keys were created (at the root of the repository) like
so:
```
from tuf import repository_tool
repository_tool.generate_and_write_ed25519_keypair(
"tests/repository_data/keystore/root_key2", "password")
repository_tool.generate_and_write_ed25519_keypair(
"tests/repository_data/keystore/root_key3", "password")
```
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The interposition sub-package was removed in #537.
This commits removes obsolete mentions of 'interposition' from code
comments and documentation.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove only wrappers and corresponding tests that don't add any
new functionality, but blindly forward the caller to sslib, where
the same function exists and is tested.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Locally merging #880 to fix code style issue (missing whitespace).
Note: the PR had an unwanted documentation patch that is excluded
from this merge.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
