Add generic read from json class method that returns a Metadata
object with a signed field that contains the appropriate Signed
subclass, based on the signed._type field of the read metadata.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The new metadata module uses constructs that are only available
on Python >= 3.6 (typing, f-format strings, etc.).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add metadata module with container classes for TUF role metadata, including
methods to read/serialize/write from and to JSON, perform TUF-compliant
metadata updates, and create and verify signatures.
The 'Metadata' class provides a container for inner TUF metadata objects (Root,
Timestamp, Snapshot, Targets) (i.e. OOP composition)
The 'Signed' class provides a base class to aggregate common attributes (i.e.
version, expires, spec_version) of the inner metadata classes. (i.e. OOP
inheritance). The name of the class also aligns with the 'signed' field of
the outer metadata container.
Based on prior observations in TUF's sister project in-toto, this architecture
seems to well represent the metadata model as it is defined in the
specification (see in-toto/in-toto#98 and in-toto/in-toto#142 for related
discussions).
This commits also adds tests.
**TODO: See doc header TODO list**
**Additional design considerations**
(also in regards to prior sketches of this module)
- Aims at simplicity, brevity and recognizability of the wireline metadata
format.
- All attributes that correspond to fields in TUF JSON metadata are public.
There doesn't seem to be a good reason to protect them with leading
underscores and use setters/getters instead, it just adds more code, and
impedes recognizability of the wireline metadata format.
- Although, it might be convenient to have short-cuts on the Metadata class
that point to methods and attributes that are common to all subclasses of
the contained Signed class (e.g. Metadata.version instead of
Metadata.signed.version, etc.), this also conflicts with goal of
recognizability of the wireline metadata. Thus we won't add such short-cuts
for now. See:
https://github.com/theupdateframework/tuf/pull/1060#discussion_r452906629
- Signing keys and a 'consistent_snapshot' boolean are not on the targets
metadata class. They are a better fit for management code. See:
https://github.com/theupdateframework/tuf/pull/1060#issuecomment-660056376,
and #660.
- Does not use sslib schema checks (see TODO notes about validation in
doc header)
- Does not use existing tuf utils, such as make_metadata_fileinfo,
build_dict_conforming_to_schema, if it is easy and more explicit to
just re-implement the desired behavior on the metadata classes.
- All datetime's are treated as UTC. Since timezone info is not captured in
the wireline metadata format it should not be captured in the internal
representation either.
- Does not use 3rd-party dateutil package, in order to minimize dependency
footprint, which is especially important for update clients which often have
to vendor their dependencies.
However, compatibility between the more advanced dateutil.relativedelta (e.g
handles leap years automatically) and timedelta is tested.
- Uses PEP8 indentation (4 space) and Google-style doc string instead of
sslab-style. See
https://github.com/secure-systems-lab/code-style-guidelines/issues/20
- Does not support Python =< 3.5
Co-authored-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Teodora Sechkova <tsechkova@vmware.com>
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
* Remove reference to deprecated settings
* Mention that the tutorial expects the dependencies and link to
instructions
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Lot of changes in 7 lines:
* PyCrypto is no longer an option: remove mention of it
* RSA-PSS wiki page now redirects to a fairly useless stub: replace it
with the RFC (it's not light reading but better than nothing)
* Mention ECDSA
* Remove mention of json for RSA keys: that does not seem to be true
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Fix load_repository to actually load the full targets file info from
file system for delegated targets.
Update _load_top_level_metadata to load targets and delegated targets
metadata in a consistent way.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
This was erroneously absent in PR 1024, which added support for abstract
files and directories. Resolve by adding a storage_backend argument to
generate_timestamp_metadata() and using it so that the fileinfo (hashes
and length) for the snapshot file can be generated for a snapshot
metadata file on any supported storage.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add a class implementing StorageBackendInterface for testhing which
mutates filenames on put()/get(), such that trying to read the expected
file paths for TUF metadata from the local filesystem doesn't find the
files.
Use this class when creating a repository and writing metadata to test
abstract files and directories support for metadata writing.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Clarify, through the docstrings and code comments, the expected behaviour
of generate_targets_metadata() and the interactions of the
use_existing_fileinfo and write_consistent_targets parameters.
Signed-off-by: Joshua Lock <jlock@vmware.com>
This makes it easier for consumers of repository_tool to mark the
appropriate delegated bin as dirty when using delegated targets.
Signed-off-by: William Woodruff <william@trailofbits.com>
Utilise the abstract files and directories support to enable generating
targets metadata for files which aren't necessarily locally accessible,
rather than requiring that metadata for non-local files be provided via
existing fileinfo structures.
Signed-off-by: Joshua Lock <jlock@vmware.com>
The specification lists four fundamental roles: root, targets, snapshot
and timestamp. Loading a repository where those roles are not present
should not be supported, therefore convert debug messages on the absence
of metadata files for these fundamental roles into a RepositoryError
exception.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Rather than check for the existence of metadata files before trying to
load them in _load_top_level_metadata, we should just try and load them.
This is more idiomatic Python through employing EAFP (Easier to Ask
Forgiveness than Permission) principles.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Switch to using the new abstract files and directories support in
securesystemslib by taking an object which implements
securesystemslib.storage.StorageBackendInterface in the Repository
constructor, passed in by tuf.repository_tool.create_new_repository() and
tuf.repository_tool.load_repository()
The Updater class in tuf.client.updater does not specify a storage backend
and instead allows the functions in securesystemslib to perform the
default action of instantiating a LocalFilesystemBackend, that is the
updater does not currently support abstract filesystem backends and always
defaults to using local storage.
Finally we drop support for tuf.settings.CONSISTENT_METHOD as it's not as
clear how different copying modes should work when the details of the
underlying storage are abstracted away.
Signed-off-by: Joshua Lock <jlock@vmware.com>
tuf removed support for compressed metadata in v0.10.x, therefore it is
confusing to carry comments referring to compressed versions of metadata.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Support for compressed files was removed in tuf v0.10.x leaving behind
some vestiges like the test logic in test_repository_lib, which is
duplicated below and carries a redundant comment, and setting compression
on in generate_project_data.py
Signed-off-by: Joshua Lock <jlock@vmware.com>
After a discussion with Joshua Lock, we agreed that for
Windows users it would be good to provide the option to use
SimpleHTTPRequestHandler, but still leave a warning about it,
knowing that this caused an error before.
See: 7dbb30ae10
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Passing a pipe to the subprocess, but not reading from it
conceals helpful error messages.
As the code redirects all of the stderr from the subprocess
to nowhere, the error output of the process is never read.
If we remove the PIPEs from the tests we should see some
error messages on the console/logger that can
help us understand what went wrong.
On another hand, when we stop passing stderr=subprocess.PIPE arg
to the subprocess.Popen function call there are a lot of
HTTP messages together with the helpful error messages.
One decision is to make QuietHTTPRequestHandler
the default. That way we receive the helpful error messages
without the HTTP messages.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Fixes issue: https://github.com/theupdateframework/tuf/issues/1010
When running the tests this error appears
"[Errno 111] Connection refused". After some digging Lukas
found another error "No module named tests.simple_server"
which is the root case for error 111.
With the help of Joshua Lock, we found out that the simple_server.py
was not being found because subprocess.Popen was being passed
a cwd kwarg which moved the current working directory
away from tuf/tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Update the docstring to make it clear that 'custom' and 'fileinfo' are
mutually exclusive, and further that the 'custom' parameter will be
dropped in a future release.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Change CUSTOM_SCHEMA to a dictionary mapping AnyString to Any value. This
makes error reporting more indicative of what the expected value is, by
reporting something like:
FormatError: Expected a dict but got ‘meta’
instead of:
FormatError: Expected a object but got 'meta'
Signed-off-by: Joshua Lock <jlock@vmware.com>
