python-tuf/tests/simple_server.py

74 lines
2.1 KiB
Python
Raw Permalink Normal View History

#!/usr/bin/env python
# Copyright 2012 - 2017, New York University and the TUF contributors
# SPDX-License-Identifier: MIT OR Apache-2.0
"""
<Program>
simple_server.py
<Author>
Konstantin Andrianov.
<Started>
February 15, 2012.
<Copyright>
See LICENSE-MIT or LICENSE for licensing information.
<Purpose>
This is a basic server that was designed to be used in conjunction with
test_download.py to test download.py module.
<Reference>
SimpleHTTPServer:
http://docs.python.org/library/simplehttpserver.html#module-SimpleHTTPServer
"""
# Help with Python 3 compatibility, where the print statement is a function, an
# implicit relative import is invalid, and the '/' operator performs true
# division. Example: print 'hello world' raises a 'SyntaxError' exception.
from __future__ import print_function
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
import sys
import random
import six
Fix failing AppVeyor Python2.7 tests Since #885 the tests in TestUpdater and TestKeyRevocation fail on Appveyor Python 2.7 builds. After some live debugging, it turns out that the tests fail due to the extra amount of http requests to the simple http server (see tests/simple_server.py) that were added in #885. The simple server runs in a subprocess and is re-used for the entire TestCase. After a certain amount of requests it becomes unresponsive. Note that neither the subprocess exits (ps -W), nor does the port get closed (netstat -a). It just doesn't serve the request, making it time out and fail the test. The following script can be used to reproduce the issue (run in tests directory): ```python import subprocess import requests import random counter = 0 port = random.randint(30000, 45000) command = ['python', 'simple_server.py', str(port)] server_process = subprocess.Popen(command, stderr=subprocess.PIPE) url = 'http://localhost:'+str(port) + '/' sess = requests.Session() try: while True: sess.get(url, timeout=3) counter +=1 finally: print(counter) server_process.kill() ``` It fails repeatedly on the 69th request, but only if `stderr=subprocess.PIPE` is passed to Popen. Given that for each request the simple server writes about ~60 characters to stderr, e.g. ... ``` 127.0.0.1 - - [24/Feb/2020 12:01:23] "GET / HTTP/1.1" 200 - ``` ... it looks a lot like a full pipe buffer of size 4096. Note that the `bufsize` argument to Popen does not change anything. As a simple work around we silence the test server on Windows/Python2 to not fill the buffer. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2020-02-24 13:51:28 +00:00
from six.moves.SimpleHTTPServer import SimpleHTTPRequestHandler
Fix failing AppVeyor Python2.7 tests Since #885 the tests in TestUpdater and TestKeyRevocation fail on Appveyor Python 2.7 builds. After some live debugging, it turns out that the tests fail due to the extra amount of http requests to the simple http server (see tests/simple_server.py) that were added in #885. The simple server runs in a subprocess and is re-used for the entire TestCase. After a certain amount of requests it becomes unresponsive. Note that neither the subprocess exits (ps -W), nor does the port get closed (netstat -a). It just doesn't serve the request, making it time out and fail the test. The following script can be used to reproduce the issue (run in tests directory): ```python import subprocess import requests import random counter = 0 port = random.randint(30000, 45000) command = ['python', 'simple_server.py', str(port)] server_process = subprocess.Popen(command, stderr=subprocess.PIPE) url = 'http://localhost:'+str(port) + '/' sess = requests.Session() try: while True: sess.get(url, timeout=3) counter +=1 finally: print(counter) server_process.kill() ``` It fails repeatedly on the 69th request, but only if `stderr=subprocess.PIPE` is passed to Popen. Given that for each request the simple server writes about ~60 characters to stderr, e.g. ... ``` 127.0.0.1 - - [24/Feb/2020 12:01:23] "GET / HTTP/1.1" 200 - ``` ... it looks a lot like a full pipe buffer of size 4096. Note that the `bufsize` argument to Popen does not change anything. As a simple work around we silence the test server on Windows/Python2 to not fill the buffer. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2020-02-24 13:51:28 +00:00
class QuietHTTPRequestHandler(SimpleHTTPRequestHandler):
"""A SimpleHTTPRequestHandler that does not write incoming requests to
stderr. """
def log_request(self, code='-', size='-'):
pass
# NOTE: On Windows/Python2 tests that use this simple_server.py in a
# subprocesses hang after a certain amount of requests (~68), if a PIPE is
# passed as Popen's stderr argument. This problem doesn't emerge if
# we silence the HTTP messages.
# If you decide to receive the HTTP messages, then this bug
# could reappear.
use_quiet_http_request_handler = True
if len(sys.argv) > 2:
use_quiet_http_request_handler = sys.argv[2]
if use_quiet_http_request_handler:
handler = QuietHTTPRequestHandler
else:
handler = SimpleHTTPRequestHandler
Fix failing AppVeyor Python2.7 tests Since #885 the tests in TestUpdater and TestKeyRevocation fail on Appveyor Python 2.7 builds. After some live debugging, it turns out that the tests fail due to the extra amount of http requests to the simple http server (see tests/simple_server.py) that were added in #885. The simple server runs in a subprocess and is re-used for the entire TestCase. After a certain amount of requests it becomes unresponsive. Note that neither the subprocess exits (ps -W), nor does the port get closed (netstat -a). It just doesn't serve the request, making it time out and fail the test. The following script can be used to reproduce the issue (run in tests directory): ```python import subprocess import requests import random counter = 0 port = random.randint(30000, 45000) command = ['python', 'simple_server.py', str(port)] server_process = subprocess.Popen(command, stderr=subprocess.PIPE) url = 'http://localhost:'+str(port) + '/' sess = requests.Session() try: while True: sess.get(url, timeout=3) counter +=1 finally: print(counter) server_process.kill() ``` It fails repeatedly on the 69th request, but only if `stderr=subprocess.PIPE` is passed to Popen. Given that for each request the simple server writes about ~60 characters to stderr, e.g. ... ``` 127.0.0.1 - - [24/Feb/2020 12:01:23] "GET / HTTP/1.1" 200 - ``` ... it looks a lot like a full pipe buffer of size 4096. Note that the `bufsize` argument to Popen does not change anything. As a simple work around we silence the test server on Windows/Python2 to not fill the buffer. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2020-02-24 13:51:28 +00:00
# Allow re-use so you can re-run tests as often as you want even if the
# tests re-use ports. Otherwise TCP TIME-WAIT prevents reuse for ~1 minute
six.moves.socketserver.TCPServer.allow_reuse_address = True
Tests: Queue replace tmp files, OS port creation These changes can be summarized with the following bullets: - Delegate generation of ports used for the tests to the OS - Use thread-safe Queue for processes communication instead of temporary files - Remove all instances of port generation or hardcoded ports - Make test_slow_retrieval.py fully conform with TestServerProcess Delegate generation of ports used for the tests to the OS is much better than if we manually generate them, because there is always a chance that the port we have randomly pick turns out to be taken. By giving 0 to the port argument we ask the OS to give us an arbitrary unused port. Use thread-safe Queue for processes communication instead of temporary files became a necessity because of findings made by Jussi Kukkonen. With the latest changes made in pr 1192 we were rapidly reading from the temporary files and Jussi found that it happened rarely the successful message "bind succeded..." to be corrupted. It seems, this is a thread issue related to the thread redirecting the subprocess stdout to the temp file and our thread rapidly reading from the file. By using a thread-safe Queue we eliminate this possibility. For reference read: https://github.com/theupdateframework/tuf/issues/1196 Lastly, test_slow_retrieval.py and slow_retrieval.py were refactored. Until now, slow_retrieval.py couldn't use the TestServerProcess class from utils.py for a port generation because of a bug related to httpd.handle_request(). Now, when we use httpd.serve_forever() we can refactor both of those files and fully conform with TestServerProcess. Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2020-10-27 15:28:01 +00:00
httpd = six.moves.socketserver.TCPServer(('localhost', 0), handler)
port_message = 'bind succeeded, server port is: ' \
+ str(httpd.server_address[1])
print(port_message)
httpd.serve_forever()