Make sure test server processes are killed before the temporary
directories are removed.
Let Modified_Testcase handle the top-level temporary directory.
Don't let Modified_testcase handle any subdirectories because:
* teardown will try to remove them in the wrong order
* removing the top level is enough
Fixes#1344
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Remove use of six
Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
Replace the use of dict.items(mydict) with mydict.items(), dict.keys(mydict) with mydict.keys() and dict.values(mydict) with mydict.values()
Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
Replace 'import urllib' and 'import urllib.x' with 'from urllib import x' for vendor compatibility
Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
Use a common test level constant for defining
the host address forming the download URL on
the client side.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
On Windows (Github Actions) the lookup for 'localhost' takes 1 second.
This is because:
- Windows retries connect() with a timeout
- the machine has IPv6 and IPv4 but Testserver only binds the port on IPv4
- the test clients connect to 'localhost'
Since socketserver.TCPServer does not seem to support IPv6 before 3.8,
just replace 'localhost' with '127.0.0.1' in client-side URLs.
See #1257
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Currently, we are importing the "utils" module in tests/utils
with "import utils".
This could become a problem when there is another module with
the same general name "utils" and could lead to import mistakes.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Update the comments to not mention the usage of temp file
for logging regarding the instances of the TestServerProcess class.
Also, remove one unused import.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Now, after we can use wait_for_server and the retry mechanism
of TestServerProcess in utils.py we no longer need to use
sleep in this test file.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
These changes can be summarized with the following bullets:
- Delegate generation of ports used for the tests to the OS
- Use thread-safe Queue for processes communication
instead of temporary files
- Remove all instances of port generation or hardcoded ports
- Make test_slow_retrieval.py fully conform with TestServerProcess
Delegate generation of ports used for the tests to the OS is much
better than if we manually generate them, because there is always
a chance that the port we have randomly pick turns out to be taken.
By giving 0 to the port argument we ask the OS to give us
an arbitrary unused port.
Use thread-safe Queue for processes communication instead of temporary
files became a necessity because of findings made by Jussi Kukkonen.
With the latest changes made in pr 1192 we were rapidly reading
from the temporary files and Jussi found that it happened rarely
the successful message "bind succeded..." to be corrupted.
It seems, this is a thread issue related to the thread redirecting
the subprocess stdout to the temp file and our thread rapidly
reading from the file.
By using a thread-safe Queue we eliminate this possibility.
For reference read:
https://github.com/theupdateframework/tuf/issues/1196
Lastly, test_slow_retrieval.py and slow_retrieval.py were refactored.
Until now, slow_retrieval.py couldn't use the TestServerProcess class
from utils.py for a port generation because of a bug related to
httpd.handle_request().
Now, when we use httpd.serve_forever() we can refactor both of those
files and fully conform with TestServerProcess.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Remove the test with mode 2 ('mode_2': During the download process,
the server blocks the download by sending just several characters
every few seconds.) from test_slow_retrieval.
This test is marked as "expected failure" with the purpose of
rewriting it one day, but slow retrievals have been removed from
the specification and soon it will be removed from the tuf
reference implementation as a whole.
That means that the chances of making this test useful are close
to 0 if not none.
The other test (with mode 1) in test_slow_retrieval is not removed.
For reference:
- https://github.com/theupdateframework/specification/pull/111
- https://github.com/theupdateframework/tuf/pull/1156
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The field confined_target_dirs from the MIRROR_SCHEMA is
a list of strings. Those strings define the accessible target
paths for that mirror. For one target to be available for that mirror,
its path should have as a prefix at least one of the strings defined
in confined_target_dirs.
That's why when confined_target_dirs is a list with one element empty
string (e.g. ['']) this means all targets files on that mirror are
available and if confined_target_dirs is empty list (e.g. []) this
would be interpreted as none of the target files is available.
This is a confusing API that could easily lead to mistakes.
That's why it's better we promote to not set confined_target_dirs
at all if a user wants targets to be available.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
I don't see a need to leave a comment about what setupClass,
tearDownClass, setup and tearDown functions do.
There is documentation that describes that.
Additionally, the links referenced in the comments are from
Python 2 is deprecated.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Logging the stdout and stderr from the test subprocesses into
temporary files clean the console from unnecessary messages from
the server-side such as "code 404, message File not found" or
"GET" queries.
I have decided to create TestServerProcess class that will handle
the server subprocess creation and redirection to a temporary file
object. That way that code can be reused in more than 10 files.
Also, I have cleaned some parts of the unit test to make them more
readable and efficient with the new abstraction.
The unit tests are executed in sequential order and that's why
we can reuse one temporary file object for multiple tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
all test_*.py files now accept zero or more '-v' to increase tuf
logging level. The default is now ERROR.
default: ERROR
"-v": ERROR, but unittest prints test names
"-vv": WARNING
"-vvv": INFO
"-vvvv": DEBUG
Example to run a single test with DEBUG level:
python3 test_updater.py -vvvv TestUpdater.test_4_refresh
Also make test_log.py restore the log level it modifies during test.
Fixes#1093
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This avoids the need to sleep() before removing the temporary
directories used, and makes sure we don't get
ResourceWarning: subprocess N is still running
messages. Use subprocess.communicate() instead of wait() if the process
has a pipe (currently the return values are just dropped though).
Practical results should be more reliability and a slightly reduced
test runtime.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Passing a pipe to the subprocess, but not reading from it
conceals helpful error messages.
As the code redirects all of the stderr from the subprocess
to nowhere, the error output of the process is never read.
If we remove the PIPEs from the tests we should see some
error messages on the console/logger that can
help us understand what went wrong.
On another hand, when we stop passing stderr=subprocess.PIPE arg
to the subprocess.Popen function call there are a lot of
HTTP messages together with the helpful error messages.
One decision is to make QuietHTTPRequestHandler
the default. That way we receive the helpful error messages
without the HTTP messages.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Replace hard-coded logger names with __name__. For the most part this just uses
the standard conventions to create the same logger hierarchy as existed before.
The only real difference is that loggers created for printing during tests are
no longer part of the 'tuf' hierarchy.
Signed-off-by: Joshua Lock <jlock@vmware.com>
This adds to PR #799; please see that for details. In short,
tests sometimes fail on slow test systems (primarily on AppVeyor)
if we don't have long enough delays.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
and also clarify setup workflow in test_slow_retrieval_attack.
Because the test was written incorrectly, the test was failing
with a bad hash error after 31 seconds, when it should instead
have been failing because the slow retrieval attack was not
averted.
In particular, a target file was updated, but metadata was not
correctly updated on the repository and, further, the client's
copy of the metadata was never updated. (The client continued
to expect 31 bytes of target file instead of 400 or 800.)
The way the test used to run, the target file change
previously had no purpose.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
Also halves the duration of a failed test (from 800s to 400s).
Otherwise, this is a code-style-only change.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
Potential slow retrieval errors are raised for safe downloads (strict check on file size) and not for unsafe (a file size less than an upper limit). Slow retrieval errors should always be prevented
