Currently, we are importing the "utils" module in tests/utils
with "import utils".
This could become a problem when there is another module with
the same general name "utils" and could lead to import mistakes.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
These changes can be summarized with the following bullets:
- Delegate generation of ports used for the tests to the OS
- Use thread-safe Queue for processes communication
instead of temporary files
- Remove all instances of port generation or hardcoded ports
- Make test_slow_retrieval.py fully conform with TestServerProcess
Delegate generation of ports used for the tests to the OS is much
better than if we manually generate them, because there is always
a chance that the port we have randomly pick turns out to be taken.
By giving 0 to the port argument we ask the OS to give us
an arbitrary unused port.
Use thread-safe Queue for processes communication instead of temporary
files became a necessity because of findings made by Jussi Kukkonen.
With the latest changes made in pr 1192 we were rapidly reading
from the temporary files and Jussi found that it happened rarely
the successful message "bind succeded..." to be corrupted.
It seems, this is a thread issue related to the thread redirecting
the subprocess stdout to the temp file and our thread rapidly
reading from the file.
By using a thread-safe Queue we eliminate this possibility.
For reference read:
https://github.com/theupdateframework/tuf/issues/1196
Lastly, test_slow_retrieval.py and slow_retrieval.py were refactored.
Until now, slow_retrieval.py couldn't use the TestServerProcess class
from utils.py for a port generation because of a bug related to
httpd.handle_request().
Now, when we use httpd.serve_forever() we can refactor both of those
files and fully conform with TestServerProcess.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
There is a simpler way to skip modules or particular tests
built-in into the unittest module.
That's why it doesn't make sense for us to manually filter
modules based on the python version we are running.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
It's convenient to be able to run unit test scripts directly, rather than
having to pass them as arguments to Python. This is already possible for
several of our unit tests, make it possible for all by setting the execute
bit.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Logging the stdout and stderr from the test subprocesses into
temporary files clean the console from unnecessary messages from
the server-side such as "code 404, message File not found" or
"GET" queries.
I have decided to create TestServerProcess class that will handle
the server subprocess creation and redirection to a temporary file
object. That way that code can be reused in more than 10 files.
Also, I have cleaned some parts of the unit test to make them more
readable and efficient with the new abstraction.
The unit tests are executed in sequential order and that's why
we can reuse one temporary file object for multiple tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
all test_*.py files now accept zero or more '-v' to increase tuf
logging level. The default is now ERROR.
default: ERROR
"-v": ERROR, but unittest prints test names
"-vv": WARNING
"-vvv": INFO
"-vvvv": DEBUG
Example to run a single test with DEBUG level:
python3 test_updater.py -vvvv TestUpdater.test_4_refresh
Also make test_log.py restore the log level it modifies during test.
Fixes#1093
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
proxy_server.py is python2 only, don't try to setup the class as it
leads to a confusing TimeoutError.
This may prevent me from debugging this apparent test failure a third
time.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Our tests already expect localhost lookup to work to find test servers:
use it consistently instead of sometimes using 127.0.0.1
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
* Add utility function to wait on a socket until it responds
* Use the function instead of sleeping in tests that need to wait for
the server to start
* Increase the max timeout to 5 seconds by default (as appveyor builds
still seem to hit the 3 second mark sometimes)
wait_for_server() functions quite differently depending on OS: Windows
can take 2 seconds to respond with ECONNREFUSED whereas Linux is almost
instant. There might be tricks to be faster on Windows (like setting
a shorter socket timeout) but this was not done here.
This makes a full Linux test run almost 40% faster and should be more
reproducible on every platform.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This avoids the need to sleep() before removing the temporary
directories used, and makes sure we don't get
ResourceWarning: subprocess N is still running
messages. Use subprocess.communicate() instead of wait() if the process
has a pipe (currently the return values are just dropped though).
Practical results should be more reliability and a slightly reduced
test runtime.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Replace hard-coded logger names with __name__. For the most part this just uses
the standard conventions to create the same logger hierarchy as existed before.
The only real difference is that loggers created for printing during tests are
no longer part of the 'tuf' hierarchy.
Signed-off-by: Joshua Lock <jlock@vmware.com>
on AppVeyor during automatic testing. Also explains a bit better
the reason for those delays, in the comments adjacent to them.
AppVeyor was occasionally laggy enough that spawning a separate
server process didn't happen fast enough for the included delays,
so connection attempts in the tests occasionally failed.
This lengthens a few 0.3s delays that I've seen pop up in test
failures to 2s delays, along with a few others for good measure.
Sadly, this slows testing a bit.... I'll keep an eye out for more
of these.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
After seeing some AppVeyor failures, I've increased the wait after
starting test HTTP, HTTPS, and proxy servers from 0.5s to 1s, to make
it less likely that tests will fail because the servers weren't done
starting up yet.
After some review comments by @aaaaalbert, I've tightened the logic
in aggregate_tests.py around which tests to skip unless a certain
Python version is running, and added some consistency checks.
This also involved a bit of clarification of comments and variable
names.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
to make sure that the test uses the intended certificate. (There's some
indirect indication that the updated environment variable might not always
have been used.)
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
that draws from sys.executable (the currently running Python interpreter)
instead of assuming 'python' is correct. Use this function instead of having
many individual subprocess calls written out. Slightly simplifies code, too.
This should eventually be moved to a common test module instead of appearing
in two places in the test code.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
- two reversions to unnecessary changes
- some typo fixes
- capitalization of HTTP/S where reasonable
- commenting out code section with ''' rather than #
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
add more explanatory comments in test_proxy.use,
and prepare for rename of proxy2.py to proxy_server.py in next
commit (separate so that the rename can be seen as such).
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
- client makes HTTPS connection to proxy; proxy makes HTTP
connection to target server
- client makes HTTPS connection to proxy; proxy makes HTTPS
connection to target server
Added functionality to the proxy2.py script to allow it to
take and use a certificate to use to validate the target
server.
Also added clarifying comments in test_proxy_use.py.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
Fixes a typo in arg processing for test script proxy2.py.
Also removes an outdated comment and clarifies another.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
for proxy testing. Also update the test scripts to point to
the new location of ssl certificates and ssl keys.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
in proxy2. And use it to run both relaying and intercepting proxies.
True: normal HTTP proxy. Support HTTP & HTTPS connections to target server
False: intercepting MITM transparent HTTPS proxy. Makes own TLS connections
and has its own cert; must be trusted by the client and is able to
modify requests.
Also perform some cleanup of test_proxy_use.py
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
- adds inaz2/proxy2.py, copied code.
- adds dev dependency on twisted for a simple proxy test
- adds a new test module, test_proxy_use, and runs those tests only in
Python2.7 (as proxy2 only supports Python2.7) using new logic in
aggregate_tests.py.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
