Currently, if the repository is consistent_snapshot,
Updater will prefix the target filename with the hash
when constructing the download URL.
For some adopters of TUF (like Warehouse) this is not wanted
(warehouse target file paths are "consistent",
even if the filenames are not).
For example, Warehouse doesn't follow what tuf
(the reference implementation and specification) advice for naming
consistent filenames, which is to prefix the filename with the hash
of the files contents.
However, the target filenames it does use are consistent,
only the hash is part of the target's file path
not the target's file name.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
proxy_server.py is python2 only, don't try to setup the class as it
leads to a confusing TimeoutError.
This may prevent me from debugging this apparent test failure a third
time.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Our tests already expect localhost lookup to work to find test servers:
use it consistently instead of sometimes using 127.0.0.1
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Travis managed to still timeout with 5 seconds: increasing the timeout
(now that it's not a sleep) doesn't really hurt normal non-VM use cases
so let's bump it to 10 seconds.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
* Add utility function to wait on a socket until it responds
* Use the function instead of sleeping in tests that need to wait for
the server to start
* Increase the max timeout to 5 seconds by default (as appveyor builds
still seem to hit the 3 second mark sometimes)
wait_for_server() functions quite differently depending on OS: Windows
can take 2 seconds to respond with ECONNREFUSED whereas Linux is almost
instant. There might be tricks to be faster on Windows (like setting
a shorter socket timeout) but this was not done here.
This makes a full Linux test run almost 40% faster and should be more
reproducible on every platform.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Ensure someone following the release process pushes the tag they create
and include a tag message matching the common form, rather than requiring
somebody following the steps to figure out what to enter into their editor
Signed-off-by: Joshua Lock <jlock@vmware.com>
For some reason the first level 3 heading 'Added' was not rendering
correctly. The level 2 heading for 'Fixed' should be level 3.
Signed-off-by: Joshua Lock <jlock@vmware.com>
This avoids the need to sleep() before removing the temporary
directories used, and makes sure we don't get
ResourceWarning: subprocess N is still running
messages. Use subprocess.communicate() instead of wait() if the process
has a pipe (currently the return values are just dropped though).
Practical results should be more reliability and a slightly reduced
test runtime.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This seems to have only worked because someone else usually imports
formats.
This fixes e.g.
python3 test_download.py \
TestDownload.test_download_url_to_tempfileobj
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
If a test happens to use the same port as a previous test run
(either by bad luck or hardcoding like TestMultiRepoUpdater) that
happened within a minute, the second run will fail because TCP by
default keeps sockets open for a while.
Avoid this by explicitly saying re-use is fine in this case.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Using logging.exception logs messages to the root logger and
calls basicConfig() to add a console handler with a pre-defined
format which breaks the current logging configuration.
Replacing logging.exception with logger.exception which is the
logger for the updater module.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Adding a do-nothing handler to the top-level 'tuf' logger
in case no other handlers exist (in tests for example).
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
This commit ensures that each key will only count toward the signature
threshold once, even if the keys have different keyids.
Signed-off-by: marinamoore <mmoore32@calpoly.edu>
Add optional parameters in repository_tool.create_new_repository()
and repository_tool.load_repository() so that our users
could control if they want to calculate length and hashes
for snapshot and timestamp roles or not.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We want to make sure we are calculating length and hashes only when
at least one of them is needed.
Otherwise, for adoptors of tuf with lots of delegations,
this will cause unnecessary overhead.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
First in the generate_timestamp_metadata both "snapshot_filename"
and the constant SNAPSHOT_FILENAME are used which is redundant
and possibly confusing. There should be only one input
for the snapshot file name.
Second, when calling the generate_timestamp_metadata there are
cases when "snapshot_filename" is in reality "snapshot_file_path".
That's what led to the need for the addition of SNAPSHOT_FILENAME
when populating the "meta" field from the TIMESTAMP_SCHEMA.
For the same reason, it seems logical to me to rename snapshot_filename
to snapshot_file_path and explicitly take the snapshot file name
from it.
Third, in the _generate_and_write_metadata function the argument
"filenames" is by default None, but at the same time without check
it's considered that filenames is a dictionary which has a key
"snapshot". This is could be okay if the default "filenames" value
was not None, but in the current situation it's easy to call
"_generate_and_write_metadata" with rolename = timestamp
and forget to populate the filenames dictionary.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This separation and refactoring is part of the change to
make length and hashes optional for timestamp and snapshot roles.
It separates FILEINFO_SCHEMA into two separate schemas:
TARGETS_FILEINFO_SCHEMA and METADATA_FILEINFO_SCHEMA.
The distinction is needed because as of version 1.0.1 of the tuf
spec targets role has mandatory length and hashes, and
snapshot and timestamp roles have a mandatory version, and optional
length and hashes.
That's why targets can't share the same schemas
as timestamp and snapshot.
Because of that schema distinction, make_fileinfo had to be too
separated into make_targets_fileinfo and make_metadata_fileinfo.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The argument targets_filename in the generate_snapshot_metadata
is redundant because the places where we are calling
generate_snapshot_metadata is by using the constant
TARGETS_FILENAME or by creating a variable with the same value
of "targets.json".
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Right now the targets_filename variable in the
_generate_and_write_metadata and in
generate_snapshot_metadata functions
was used without the.json suffix which is a little misleading.
This wasn't a big issue before because this variable wasn't
actually used as a file name until now.
Now, we need to use it with it's 'json' suffix so we can
calculate the hashes and length for the targets metadata.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
As per the specification (v1.0.1) length and hashes fields
in snapshot metadata are optional.
The reference implementation should reflect this.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
As per the specification (v1.0.1) length and hashes fields
in timestamp metadata are optional.
The reference implementation should reflect this.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
