We could just stop testing with 3.9... but I think this will lead to
unintentionally breaking 3.9 anyway sooner or later.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
This allows using some more nice annotations from 3.10
while still being compatible with even Python 3.8.
These are all annotation changes, should not modify any functionality.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
The goal here is to have ruff enable new rulesets when new releases are
made without us having to o anything: we can then decide if we disable
or not.
* Enable a couple more rulesets (ERA, INP, T )
* Add a few individual ignores to tests and examples
* Default to enable all, disable the rulesets we don't want
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
* Remove exectuable flag from a couple of files
* Half of the test files have a shebang (but are
still not executable): remove the shebang
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Minor fixes were needed, the only possibly interesting one is
the one in RequestsFetcher (use "yield from").
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Old tests are not touched as they will be removed in subsequent
commits, along with the custom log module.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Define TESTS_DIR constant in tests/util.py as full path to the
parent directory of the util module. This may be used to reliably
read other files in tests dir, such es "repository_data" or
"simple_server", regardless of cwd.
This commit also replaces a couple of `getcwd() + "filename"` with
`TESTS_DIR + filename`, so that in the future (post #1790) we
should be able to invoke the tests from anywhere, not only from
within the tests directory as is now the case.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Apply all 4 linters on tests/utils.py, so we can lint it in future and
not rename and exclude it.
It's expected that most if not all code in tests/utils.py will be
useful even when we remove the tests file on the old code.
Keep in mind when reviewing that type annotations where already added
in e2deff3148
Black and isort changes where automatically made.
The only manual changes are:
- pylint disables
- function docstrings
- initializations of attributes of "TestServerProcess" __init__()
- additional asserts of the attributes types
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Save the case name in the "unittest.TestCase" object when executing
test cases through the "run_sub_tests_with_dataset" decorator.
This allows dumping directories with names specific to the test case
that can be used for debugging as showed in
test_updater_key_rotations.py
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit includes manual fixes for a lot of mypy warnings.
When there were warnings that we are calling non-annotated function
in annotated context I decided to add annotations instead of ignoring
those warnings.
That's how I end up adding annotations in the whole tests/utils.py
module.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Generalize the decorator used in test_metadata_serialization.py and
move it inside tests/utils.py, so it can be reused in other similar
situations.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Use a common test level constant for defining
the host address forming the download URL on
the client side.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
This quote from the Google Python style guide made me realize
why empty list as a default value for an argument could be
dangerous:
"Default arguments are evaluated once at module load time.
This may cause problems if the argument is a mutable object
such as a list or a dictionary. If the function modifies the object
(e.g., by appending an item to a list), the default value is modified."
Read more here:
https://google.github.io/styleguide/pyguide.html#2123-cons
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The current implementation for server startup in TestServerProcess
relies on the fact that "bind successful..." is the first message
sent by the server process.
Make sure that this is true and leave a comment about this.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
These changes can be summarized with the following bullets:
- Delegate generation of ports used for the tests to the OS
- Use thread-safe Queue for processes communication
instead of temporary files
- Remove all instances of port generation or hardcoded ports
- Make test_slow_retrieval.py fully conform with TestServerProcess
Delegate generation of ports used for the tests to the OS is much
better than if we manually generate them, because there is always
a chance that the port we have randomly pick turns out to be taken.
By giving 0 to the port argument we ask the OS to give us
an arbitrary unused port.
Use thread-safe Queue for processes communication instead of temporary
files became a necessity because of findings made by Jussi Kukkonen.
With the latest changes made in pr 1192 we were rapidly reading
from the temporary files and Jussi found that it happened rarely
the successful message "bind succeded..." to be corrupted.
It seems, this is a thread issue related to the thread redirecting
the subprocess stdout to the temp file and our thread rapidly
reading from the file.
By using a thread-safe Queue we eliminate this possibility.
For reference read:
https://github.com/theupdateframework/tuf/issues/1196
Lastly, test_slow_retrieval.py and slow_retrieval.py were refactored.
Until now, slow_retrieval.py couldn't use the TestServerProcess class
from utils.py for a port generation because of a bug related to
httpd.handle_request().
Now, when we use httpd.serve_forever() we can refactor both of those
files and fully conform with TestServerProcess.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Filter out:
* DeprecationWarnings for updater module when we are on purpose
testing deprecated methods from updater
* SubjectAltNameWarning for connections to our test server
These warnings are visible with e.g.
python3 test_updater.py
The large change in test_download.py is just indentation into with-block.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Logging the stdout and stderr from the test subprocesses into
temporary files clean the console from unnecessary messages from
the server-side such as "code 404, message File not found" or
"GET" queries.
I have decided to create TestServerProcess class that will handle
the server subprocess creation and redirection to a temporary file
object. That way that code can be reused in more than 10 files.
Also, I have cleaned some parts of the unit test to make them more
readable and efficient with the new abstraction.
The unit tests are executed in sequential order and that's why
we can reuse one temporary file object for multiple tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We want the tests own log output visible as well, and they are not
under the "tuf" logger. Set root level to the same value as "tuf".
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
all test_*.py files now accept zero or more '-v' to increase tuf
logging level. The default is now ERROR.
default: ERROR
"-v": ERROR, but unittest prints test names
"-vv": WARNING
"-vvv": INFO
"-vvvv": DEBUG
Example to run a single test with DEBUG level:
python3 test_updater.py -vvvv TestUpdater.test_4_refresh
Also make test_log.py restore the log level it modifies during test.
Fixes#1093
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Travis managed to still timeout with 5 seconds: increasing the timeout
(now that it's not a sleep) doesn't really hurt normal non-VM use cases
so let's bump it to 10 seconds.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
* Add utility function to wait on a socket until it responds
* Use the function instead of sleeping in tests that need to wait for
the server to start
* Increase the max timeout to 5 seconds by default (as appveyor builds
still seem to hit the 3 second mark sometimes)
wait_for_server() functions quite differently depending on OS: Windows
can take 2 seconds to respond with ECONNREFUSED whereas Linux is almost
instant. There might be tricks to be faster on Windows (like setting
a shorter socket timeout) but this was not done here.
This makes a full Linux test run almost 40% faster and should be more
reproducible on every platform.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
