* Remove exectuable flag from a couple of files
* Half of the test files have a shebang (but are
still not executable): remove the shebang
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
This is useful for those who want to use the default fetcher
but modify some attributes
The file itself could be moved to tuf/ngclient/ but this is not done yet
as sigstore-python is using this internal module. Move can be done once
sigstore-python 1.0 is no longer relevant.
Fixes#2268
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
I've not supported many renames but I'm suggesting this one:
FetcherHTTPError was created because we needed to signal 403/404
from the fetcher to updater. At that time the download error hierarchy
in general was not thought out.
Now we have a couple of different errors all derived from
DownloadError. I believe it does not make sense to point out "Fetcher"
in one of their names: DownloadHTTPError makes it clearer this is a
specific type of DownloadError.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This commit tries to deal with two interests:
* metadata is highly repetitive and compressible: allowing compression
would be good
* there may be broken web servers (see
404838abcc/src/pip/_internal/download.py (L842))
that have problems with compression on already compressed target files
We can make things better for that first interest while we have no real
data for the second interest -- our current workarounds to avoid
compression are based on hearsay, not testing.
Now that individual fetchers are possible I suggest we simplify
ngclient and allow compression. As an example the pip Fetcher
could still use the pip response chunking code with all their
workarounds -- pip certainly has better capability to maintain
a mountain of workarounds and also has endless amounts of real-world
testing compared to python-tuf.
Details:
* Stop modifying Accept-Encoding (Requests default includes gzip)
* Don't use response.raw in RequestsFetcher as there is no need:
This was a workaround for false "Content-encoding: gzip" inserted by
a broken server -- and the workaround was only possible because we
knew we never asked for compression
* Fix issue in test_session_get_timeout(): it's not mocking the error
that requests really raises in this case
Fixes#1251
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Update new test modules to stop using unittest_toolbox, in
preparation for its removal in #1790.
The tools provided by unittest_toolbox can easily (in a more
obvious way) be replaced by using the standard library modules
`tempfile` and `random` (no more used) directly.
In the case of tempdir and -file creation/removal, skipping the use
of unittest_toolbox, which does this by default, also uncovers some
test cleanup failures, which would occur when temporary test
directories were removed while a test server hadn't released them.
(see `except OSError: pass` in unittest_toolbox's `tearDown`
method)
**Change details**
**test_fetcher_ng.py:**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs. -Move now manual creation of an exemplary targets file to
setUpClass, as the same file is used by all tests. And remove it
explicitly in tearDownClass after killing the server (see note
about failure above). - Trigger URL parsing error with a hardcoded
invalid URL string instead of a random string.
**test_updater_ng.py**
- Stop implicitly creating (setUp) and removing (tearDown) tmp test
dirs.
- Explicitly create tmp test dirs in setUp, but don't remove
them in tearDown to avoid above mentioned failures. They will be
removed all at once when removing the tmp root test dir in
tearDownClass
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
URLParsingError is a specific download error and
is not clear what benefit it provides.
It's used only once in the new code and the
message says everything you need to know about
the exception.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add tuf/api/exceptions.py for exceptions in the new code.
I copied the exceptions from tuf/exceptions.py with a few important
decisions:
1. I only added the exceptions that are used in the new code
2. I removed the general "Error" class as we can directly inherit
Exceptions
3. I tried grouping the exceptions by relevance
4. I removed the second argument "UnsignedMetadataError" as it's only
kept for backward compatibility and is not used
5. I tried following the new code style guidelines and linted the file
with our linters.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
New pylint warnings appeared related to changes
in urlib3:
- tests/test_fetcher_ng.py:128: error: Argument 1 to "ReadTimeoutError"
has incompatible type "None"; expected "ConnectionPool" [arg-type]
- tests/test_fetcher_ng.py:128: error: Argument 2 to "ReadTimeoutError"
has incompatible type "None"; expected "str" [arg-type]
I noticed these error in this CI run:
https://github.com/theupdateframework/python-tuf/runs/4764931441?check_suite_focus=true
I fixed them by creating a urllib3.HTTPConnectionPool() instance as
the first argument and replaced the second argument with an empty
string.
This seems to do the job.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This commit includes manual fixes for a lot of mypy warnings.
When there were warnings that we are calling non-annotated function
in annotated context I decided to add annotations instead of ignoring
those warnings.
That's how I end up adding annotations in the whole tests/utils.py
module.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Address or disable pylint warnings raised on all test files inside
the "tests/" directory testing the code of the new implementation.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Instead of starting a dedicated slow_retrieval_server
to test for read timeout in RequestsFetcher, use
unittest.mock to mock the response.raw.read call.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
RequestsFetcher now handles connect/read timeout
errors on session.get() as it does when reading data.
Adding a test which uses unittest.mock to patch the
session.get method instead of simulating the timeout
with a server.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
