The current implementation for server startup in TestServerProcess
relies on the fact that "bind successful..." is the first message
sent by the server process.
Make sure that this is true and leave a comment about this.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We want to make sure that server are successfully started in
the common use cases and that the new port generation works.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Now, after we can use wait_for_server and the retry mechanism
of TestServerProcess in utils.py we no longer need to use
sleep in this test file.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
These changes can be summarized with the following bullets:
- Delegate generation of ports used for the tests to the OS
- Use thread-safe Queue for processes communication
instead of temporary files
- Remove all instances of port generation or hardcoded ports
- Make test_slow_retrieval.py fully conform with TestServerProcess
Delegate generation of ports used for the tests to the OS is much
better than if we manually generate them, because there is always
a chance that the port we have randomly pick turns out to be taken.
By giving 0 to the port argument we ask the OS to give us
an arbitrary unused port.
Use thread-safe Queue for processes communication instead of temporary
files became a necessity because of findings made by Jussi Kukkonen.
With the latest changes made in pr 1192 we were rapidly reading
from the temporary files and Jussi found that it happened rarely
the successful message "bind succeded..." to be corrupted.
It seems, this is a thread issue related to the thread redirecting
the subprocess stdout to the temp file and our thread rapidly
reading from the file.
By using a thread-safe Queue we eliminate this possibility.
For reference read:
https://github.com/theupdateframework/tuf/issues/1196
Lastly, test_slow_retrieval.py and slow_retrieval.py were refactored.
Until now, slow_retrieval.py couldn't use the TestServerProcess class
from utils.py for a port generation because of a bug related to
httpd.handle_request().
Now, when we use httpd.serve_forever() we can refactor both of those
files and fully conform with TestServerProcess.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Remove the test with mode 2 ('mode_2': During the download process,
the server blocks the download by sending just several characters
every few seconds.) from test_slow_retrieval.
This test is marked as "expected failure" with the purpose of
rewriting it one day, but slow retrievals have been removed from
the specification and soon it will be removed from the tuf
reference implementation as a whole.
That means that the chances of making this test useful are close
to 0 if not none.
The other test (with mode 1) in test_slow_retrieval is not removed.
For reference:
- https://github.com/theupdateframework/specification/pull/111
- https://github.com/theupdateframework/tuf/pull/1156
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Update badge URL in readme after migrating from travis-ci.org to
travis-ci.com, due to brownout on the former.
Migration was performed via Travis Web UI:
https://docs.travis-ci.com/user/migrate/open-source-repository-migration
NOTE: This is a quick fix to speed up Travis builds until we switch
to GitHub Actions (#1195)
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Rather than read to the end of the file in order to determin its size, use
the whence value of seek() to move the file object's position to the end
of the file, then the tell() method of the file object to read the current
position in bytes.
Co-authored-by: Jussi Kukkonen <jkukkonen@vmware.com>
Signed-off-by: Joshua Lock <jlock@vmware.com>
secure-systems-lab/securesystemslib#288 changes the key generation
interface functions in such a way that it is clear if a call opens
a blocking prompt, or writes the key unencrypted. To do this two
functions are added per key type:
- `generate_and_write_*_keypair_with_prompt`
- `generate_and_write_unencrypted_*_keypair`
The default `generate_and_write_*_keypair` function now only allows
encrypted keys and only using a passed password. This respects the
principle of secure defaults and least surprise.
sslib#288 furthermore adds a protected
`_generate_and_write_*_keypair`, which is not exposed publicly
because it does not encrypt by default, but is more flexible and
thus convenient e.g. to consume all arguments from a key generation
command line tool such as 'repo.py'.
This commit adds the new public functions to the tuf namespace and
adopts their usage accordingly.
NOTE regarding repo.py:
This commit does not fix any problematic password behavior of
'repo.py' like default passwords, etc. (see #881). It only adopts
the sslib#288 changes to maintain the current behvior, plus
removing one glaringly obsolete password prompt.
NOTE regarding key import:
The securesystemslib private key import functions were also changed
to no longer auto-prompt for decryption passwords , TUF, however,
only exposes custom wrappers (see repository_lib) that do
auto-prompt. sslib#288 changes to the prompt texts are nevertheless
propagated to tuf and reflected in this commit.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The call stack and code for download_target() is more complex than
required:
* download_target() : builds target destination filepath, gets length
and hashes
* _get_target_file() : fixes filenames if consistent snapshots enabled,
defines verification callback
* _get_file() : iterates mirrors, tries to download files, verifies them
Remove the verification callback and collapse the call stack by a single
level to make the code easier to follow.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Use deepcopy to ensure that the dictionaries with expected data
are not referencing the same memory as the tested ones.
Add a check asserting that metadata is not equal prior to its
update.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Add root metadata class to tuf.api.metadata module and implement
(de)serialisation and modification methods.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Python 3.9 is released on October 5-th 2020 and it seems
logical to add support for it.
For reference read:
https://docs.python.org/3/whatsnew/3.9.html
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
There is a simpler way to skip modules or particular tests
built-in into the unittest module.
That's why it doesn't make sense for us to manually filter
modules based on the python version we are running.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Added ExpiredMetadataError to function documentation where it seems to
be missing.
Corrected the refresh() documentation: ExpiredMetadataError can only
happen when top level metadata does not need to be updated but is
expired. If the metadata gets updated and is expired, the result will
be a NoWorkingMirror with ExpiredMetadata inside it.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Provide additional context to clarify where we expect Python 3.6+ to be used
exclusively (new modules) and link to other discussions around the future of
Python 2.7 supporting code.
Signed-off-by: Joshua Lock <jlock@vmware.com>
In order to make decisions about the code and the design explicit and easier
to reference in future we want to record significant architectural decisions.
This commit introduces docs/adr with a template Architectural Decision Record
and index using the [MADR](https://adr.github.io/madr/) format.
It also adds ADR 0000 to document the decisions to use MADR.
Fixes#1141
Signed-off-by: Joshua Lock <jlock@vmware.com>
Commit eb00d14 modified requirements-pinned.txt so that sslib specifiers
are now "[crypto,pynacl]". This happens to match the exact specifiers
used for the sslib git master dependency in tox.ini. This triggers pip
to say:
ERROR: Double requirement given: securesystemslib[crypto,pynacl]==0.16.0
(from -r /home/jku/src/tuf/requirements-pinned.txt (line 12)) (already
in securesystemslib[crypto,pynacl] from
git+http://github.com/secure-systems-lab/securesystemslib.git@master#egg=securesystemslib[crypto,pynacl],
name='securesystemslib')
Avoid this by not setting any specifiers for the sslib git master
dependency in tox.ini: This makes pip happy and we get the git master
version installed. pynacl and crypto are still installed because they
are in requirements-pinned.txt.
Fixes#1184.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
The repo script was the only user and can now do the right thing when
colorama isn't available in the environment.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Instead of using colorama directly for terminal colours, use the
constants in securesystemslib.interface which map to colorama colours
IFF colorama is installed.
This change results in a red password prompt when colorama is installed
and a standard terminal output coloured prompt when colorama is not
installed.
Signed-off-by: Joshua Lock <jlock@vmware.com>
