Use Google style guide with refinements, because the Google style
guide is a comprehensive, well-established style guide that is
mostly based on PEP-8 and was accepted by everyone on the TUF team.
There is no need to replicate these recommendations. However, we do
provide a very slim document with additional refinements, in order
to emphasize on items the we consider especially important, want to
be handled differently, or in one specific way, where the Google
guide would allow multiple.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Add MADR that justifies why we want to add custom classes for
complex tuf metadata attributes.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Installing securesystemslib in editable mode leads to a problem
in sys.path where we have two "tests" packages.
By not installing securesystemslib in an editable mode we are not
adding the securesystemslib tests to sys.path.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Describe pros of developing TUF 1.0.0 in a subdirectory
of the current implementation against the rest of the options.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Document the outcome of #1126 to develop TUF 1.0.0
in a subdirectory of the current TUF implementation.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
If during updater.download_target() the download succeeds but a later
check fails (e.g. BadHashError), remember to close the tempfile.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Simplify the loop exit logic in _get_target_file() to simply return a
verified file_object, once we have it, rather than breaking from the loop
and then returning the file_object.
This converts a use of a try/except/else to a try/except and is a little
easier to read.
Signed-off-by: Joshua Lock <jlock@vmware.com>
When verifying newly downloaded root metadata with the keys listed in the
root metadata being verified, multiple signatures with the same keyid
should not be counted towards the threshold. A keyid should only count
once towards the threshold.
This fixes the _verify_root_self_signed() method introduced in PR #1101 to
ensure that keyids are only counted once when verifying a threshold of new
root signatures.
Signed-off-by: Joshua Lock <jlock@vmware.com>
When the updater is verifying that the new root metadata is signed by a
threshold of keys defined by the new root metadata itself, multiple
signatures with the same keyid should not be counted more than once
towards the threshold.
Implement a test for this, which currently fails.
Reported-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>
Signed-off-by: Joshua Lock <jlock@vmware.com>
Currently, we are importing the "utils" module in tests/utils
with "import utils".
This could become a problem when there is another module with
the same general name "utils" and could lead to import mistakes.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This quote from the Google Python style guide made me realize
why empty list as a default value for an argument could be
dangerous:
"Default arguments are evaluated once at module load time.
This may cause problems if the argument is a mutable object
such as a list or a dictionary. If the function modifies the object
(e.g., by appending an item to a list), the default value is modified."
Read more here:
https://google.github.io/styleguide/pyguide.html#2123-cons
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We don't want to read the whole file in memory as it can be huge. Use
digest_fileobject() instead: This way Securesystemslib will read the
file in chunks.
Securesystemslib already takes care of seeking to beginning of file.
Fixes#1215
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
In 'repository_lib._generate_and_write_metadata' sort the set of
signing key keyids alphabetically before passing them on to signing
functions, to make the order in which signatures are added
deterministic.
This is above all beneficial for testing.
This commit also adds an exemplary test for signatures on root
metadata using the repository_tool interface to setup all the state
that required to test _generate_and_write_metadata.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Update the comments to not mention the usage of temp file
for logging regarding the instances of the TestServerProcess class.
Also, remove one unused import.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The current implementation for server startup in TestServerProcess
relies on the fact that "bind successful..." is the first message
sent by the server process.
Make sure that this is true and leave a comment about this.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We want to make sure that server are successfully started in
the common use cases and that the new port generation works.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Now, after we can use wait_for_server and the retry mechanism
of TestServerProcess in utils.py we no longer need to use
sleep in this test file.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
These changes can be summarized with the following bullets:
- Delegate generation of ports used for the tests to the OS
- Use thread-safe Queue for processes communication
instead of temporary files
- Remove all instances of port generation or hardcoded ports
- Make test_slow_retrieval.py fully conform with TestServerProcess
Delegate generation of ports used for the tests to the OS is much
better than if we manually generate them, because there is always
a chance that the port we have randomly pick turns out to be taken.
By giving 0 to the port argument we ask the OS to give us
an arbitrary unused port.
Use thread-safe Queue for processes communication instead of temporary
files became a necessity because of findings made by Jussi Kukkonen.
With the latest changes made in pr 1192 we were rapidly reading
from the temporary files and Jussi found that it happened rarely
the successful message "bind succeded..." to be corrupted.
It seems, this is a thread issue related to the thread redirecting
the subprocess stdout to the temp file and our thread rapidly
reading from the file.
By using a thread-safe Queue we eliminate this possibility.
For reference read:
https://github.com/theupdateframework/tuf/issues/1196
Lastly, test_slow_retrieval.py and slow_retrieval.py were refactored.
Until now, slow_retrieval.py couldn't use the TestServerProcess class
from utils.py for a port generation because of a bug related to
httpd.handle_request().
Now, when we use httpd.serve_forever() we can refactor both of those
files and fully conform with TestServerProcess.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Remove the test with mode 2 ('mode_2': During the download process,
the server blocks the download by sending just several characters
every few seconds.) from test_slow_retrieval.
This test is marked as "expected failure" with the purpose of
rewriting it one day, but slow retrievals have been removed from
the specification and soon it will be removed from the tuf
reference implementation as a whole.
That means that the chances of making this test useful are close
to 0 if not none.
The other test (with mode 1) in test_slow_retrieval is not removed.
For reference:
- https://github.com/theupdateframework/specification/pull/111
- https://github.com/theupdateframework/tuf/pull/1156
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Update badge URL in readme after migrating from travis-ci.org to
travis-ci.com, due to brownout on the former.
Migration was performed via Travis Web UI:
https://docs.travis-ci.com/user/migrate/open-source-repository-migration
NOTE: This is a quick fix to speed up Travis builds until we switch
to GitHub Actions (#1195)
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Rather than read to the end of the file in order to determin its size, use
the whence value of seek() to move the file object's position to the end
of the file, then the tell() method of the file object to read the current
position in bytes.
Co-authored-by: Jussi Kukkonen <jkukkonen@vmware.com>
Signed-off-by: Joshua Lock <jlock@vmware.com>
