secure-systems-lab/securesystemslib#288 changes the key generation
interface functions in such a way that it is clear if a call opens
a blocking prompt, or writes the key unencrypted. To do this two
functions are added per key type:
- `generate_and_write_*_keypair_with_prompt`
- `generate_and_write_unencrypted_*_keypair`
The default `generate_and_write_*_keypair` function now only allows
encrypted keys and only using a passed password. This respects the
principle of secure defaults and least surprise.
sslib#288 furthermore adds a protected
`_generate_and_write_*_keypair`, which is not exposed publicly
because it does not encrypt by default, but is more flexible and
thus convenient e.g. to consume all arguments from a key generation
command line tool such as 'repo.py'.
This commit adds the new public functions to the tuf namespace and
adopts their usage accordingly.
NOTE regarding repo.py:
This commit does not fix any problematic password behavior of
'repo.py' like default passwords, etc. (see #881). It only adopts
the sslib#288 changes to maintain the current behvior, plus
removing one glaringly obsolete password prompt.
NOTE regarding key import:
The securesystemslib private key import functions were also changed
to no longer auto-prompt for decryption passwords , TUF, however,
only exposes custom wrappers (see repository_lib) that do
auto-prompt. sslib#288 changes to the prompt texts are nevertheless
propagated to tuf and reflected in this commit.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
There is a simpler way to skip modules or particular tests
built-in into the unittest module.
That's why it doesn't make sense for us to manually filter
modules based on the python version we are running.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Even though we don't want to promote the usage of [''] as a value
for confined_target_dirs, it's good to test against because we
don't want to introduce a breaking change for our users.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The field confined_target_dirs from the MIRROR_SCHEMA is
a list of strings. Those strings define the accessible target
paths for that mirror. For one target to be available for that mirror,
its path should have as a prefix at least one of the strings defined
in confined_target_dirs.
That's why when confined_target_dirs is a list with one element empty
string (e.g. ['']) this means all targets files on that mirror are
available and if confined_target_dirs is empty list (e.g. []) this
would be interpreted as none of the target files is available.
This is a confusing API that could easily lead to mistakes.
That's why it's better we promote to not set confined_target_dirs
at all if a user wants targets to be available.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Our 'expires' strings are constrained by the ISO8601_DATETIME_SCHEMA
which matches regex '\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z'. This can be
parsed with just a datetime.strptime(): iso8601 module is not needed.
* Add formats.expiry_string_to_datetime() helper function
* Modify the 3 locations that used iso8601 and the api/metadata.py usage
of datetime.strptime()
* Remove related unnecessary logger setup
* Add the missing exception documentation to relevant functions (in many
cases the exception is rather unlikely as the schema has been verified
many times before this though...)
Fixes#1065
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This allows clients to separate
a) missing local repository and
b) error while loading local repository
This is fully backwards-compliant: MissingLocalRepositoryError derives
from RepositoryError and every situation that now results in
MissingLocalRepositoryError used to result in a RepositoryError.
Fixes#1063
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Filter out:
* DeprecationWarnings for updater module when we are on purpose
testing deprecated methods from updater
* SubjectAltNameWarning for connections to our test server
These warnings are visible with e.g.
python3 test_updater.py
The large change in test_download.py is just indentation into with-block.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
It's convenient to be able to run unit test scripts directly, rather than
having to pass them as arguments to Python. This is already possible for
several of our unit tests, make it possible for all by setting the execute
bit.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Modify test cases which use unsigned metadata.
Update test_sign_metadata to check for empty key list.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Fixes an issue where rmtree tries to access and consequently remove
a temp folder where the server has opened a file already.
This results in error:
"PermissionError: [WinError 32] The process cannot access the file
because it is being used by another process"
For reference read:
https://github.com/theupdateframework/tuf/issues/1119
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
I don't see a need to leave a comment about what setupClass,
tearDownClass, setup and tearDown functions do.
There is documentation that describes that.
Additionally, the links referenced in the comments are from
Python 2 is deprecated.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Logging the stdout and stderr from the test subprocesses into
temporary files clean the console from unnecessary messages from
the server-side such as "code 404, message File not found" or
"GET" queries.
I have decided to create TestServerProcess class that will handle
the server subprocess creation and redirection to a temporary file
object. That way that code can be reused in more than 10 files.
Also, I have cleaned some parts of the unit test to make them more
readable and efficient with the new abstraction.
The unit tests are executed in sequential order and that's why
we can reuse one temporary file object for multiple tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Now clients can leave out targets_path or metadata_path if the
client knows the mirror does not have that type of targets.
This is backwards compatible: old mirror configs continue to work.
Fixes#1079
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
* move code to only create objects after potential raises
* Use 'with' when possible
* close manually if those did not help
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
We want the tests own log output visible as well, and they are not
under the "tuf" logger. Set root level to the same value as "tuf".
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
all test_*.py files now accept zero or more '-v' to increase tuf
logging level. The default is now ERROR.
default: ERROR
"-v": ERROR, but unittest prints test names
"-vv": WARNING
"-vvv": INFO
"-vvvv": DEBUG
Example to run a single test with DEBUG level:
python3 test_updater.py -vvvv TestUpdater.test_4_refresh
Also make test_log.py restore the log level it modifies during test.
Fixes#1093
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
All of these tests create an exception and log it. They do not
actually test anything.
The alternative would be to change the log level to something that is
not error to prevent verbose error output on successful test runs --
but that still wouldn't make them actual tests.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Change Metadata.verify(key) behavior to raise an exception if
none or multiple signatures for the passed key are found on the
Metadata object.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add convenience wrapper that takes a json string and passes it
to from_dict to create a Metadata object.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This commit better separates the Metadata class model from the
Metadata wireline format, by tailoring the constructors
towards class-based parameters and adding an additional
factory classmethod that creates Metadata objects based on the
wireline json/dictionary metadata representation. (pythonic
way of constructor overloading).
This 'from_dict' factory method recurses into the 'from_dict'
methods of each contained complex field/attribute that is also
represented by a class. Currently 'signed' is the only such
attribute.
This commit further:
- Changes optional constructor keyword arguments to mandatory
positional arguments: Reduces code and simplifies usage by
restricting it. For now, users are unlikely to call
constructor directly anyway, but the 'from_dict' factory (or
its 'from_json_file' wrapper) instead.
- Removes Signed.__expiration (datetime) vs. Signed.expires
(datestring) dichotomy: Keeping only one representation of the
same attribute in memory makes the interface simpler and less
ambiguous. We choose the datetime object, because it is more
convenient to modify. Transformation from and to the string
format required by the tuf wireline format is performed in the
corresponding metadata de/serialization methods, i.e.
('to_dict' and 'from_dict').
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Consistenly rename de/serialization interface methods, using
a 'from_' and 'to_' prefix.
read_from_json -> from_json_file
write_to_json -> to_json_file
as_json -> to_json
as_dict -> to_dict
signed_bytes -> to_canonical_bytes
The latter is also changed from a property to a method for
consistency with the other serialization methods.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove metadata factory on Signed class, for the sake of API
simplicity/non-ambiguity, i.e. it's enough to have one
way of loading any Metadata, that is:
Metadata.read_from_json
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add simple methods to create or verify signatures of the
canonical_signed property of a Metadata object.
See corresponding docstrings for behavior and design
considerations.
The commit also adds tests and updates the test setup to load
some test keys into memory.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add simple as_json Metadata method and use it instead of repository
lib's internal _get_written_metadata function in write_to_json.
This commit further adds code documentation and the possibility to
write compact json by excluding whitespace to write_to_json, and
also removes a call to the sign method from write_to_json.
The commit also adds tests.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add generic read from json class method that returns a Metadata
object with a signed field that contains the appropriate Signed
subclass, based on the signed._type field of the read metadata.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The new metadata module uses constructs that are only available
on Python >= 3.6 (typing, f-format strings, etc.).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add a new test case in test_generate_targets_metadata
to check if targets metadata is up-to-date with its delegated
roles.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Tests logic is modified to accommodate for the update of the
delegations during generate_targets_metadata().
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Add metadata module with container classes for TUF role metadata, including
methods to read/serialize/write from and to JSON, perform TUF-compliant
metadata updates, and create and verify signatures.
The 'Metadata' class provides a container for inner TUF metadata objects (Root,
Timestamp, Snapshot, Targets) (i.e. OOP composition)
The 'Signed' class provides a base class to aggregate common attributes (i.e.
version, expires, spec_version) of the inner metadata classes. (i.e. OOP
inheritance). The name of the class also aligns with the 'signed' field of
the outer metadata container.
Based on prior observations in TUF's sister project in-toto, this architecture
seems to well represent the metadata model as it is defined in the
specification (see in-toto/in-toto#98 and in-toto/in-toto#142 for related
discussions).
This commits also adds tests.
**TODO: See doc header TODO list**
**Additional design considerations**
(also in regards to prior sketches of this module)
- Aims at simplicity, brevity and recognizability of the wireline metadata
format.
- All attributes that correspond to fields in TUF JSON metadata are public.
There doesn't seem to be a good reason to protect them with leading
underscores and use setters/getters instead, it just adds more code, and
impedes recognizability of the wireline metadata format.
- Although, it might be convenient to have short-cuts on the Metadata class
that point to methods and attributes that are common to all subclasses of
the contained Signed class (e.g. Metadata.version instead of
Metadata.signed.version, etc.), this also conflicts with goal of
recognizability of the wireline metadata. Thus we won't add such short-cuts
for now. See:
https://github.com/theupdateframework/tuf/pull/1060#discussion_r452906629
- Signing keys and a 'consistent_snapshot' boolean are not on the targets
metadata class. They are a better fit for management code. See:
https://github.com/theupdateframework/tuf/pull/1060#issuecomment-660056376,
and #660.
- Does not use sslib schema checks (see TODO notes about validation in
doc header)
- Does not use existing tuf utils, such as make_metadata_fileinfo,
build_dict_conforming_to_schema, if it is easy and more explicit to
just re-implement the desired behavior on the metadata classes.
- All datetime's are treated as UTC. Since timezone info is not captured in
the wireline metadata format it should not be captured in the internal
representation either.
- Does not use 3rd-party dateutil package, in order to minimize dependency
footprint, which is especially important for update clients which often have
to vendor their dependencies.
However, compatibility between the more advanced dateutil.relativedelta (e.g
handles leap years automatically) and timedelta is tested.
- Uses PEP8 indentation (4 space) and Google-style doc string instead of
sslab-style. See
https://github.com/secure-systems-lab/code-style-guidelines/issues/20
- Does not support Python =< 3.5
Co-authored-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Teodora Sechkova <tsechkova@vmware.com>
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This method is duplicating verification steps which have already been
completed before the file was written to disk.
Signed-off-by: Joshua Lock <jlock@vmware.com>
