We have tests which make sure we can use `Timestamp.update()` and
`Snapshot.update()` with MetaFile instance storing only version
(because length and hashes are optional).
Those tests were created to make sure that we are actually supporting
optional hashes and length when we call `update` for those classes, but
after we changed the `update()` signature to accept `MetaFile` instance
the tests are obsolete.
The reason is that length and hashes can be optional because of the
MetaFile implementation, no the update function itself and we have
other tests validating creating a MetaFie instance without hashes and
length.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Currently, when we call Targets/Snapshot/Timestamp.update() we are
passing all of the necessary values to create MetaFile/Targets File
respectively.
This is not needed, given that one of the reasons we have created
MetaFile and TargetFile is to make the API easier to use.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In the top-level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.
As written in the spec "targets" in "targets.json" has defined the
"custom" field serving the same purpose as "unrecognized_fields" in the
implementation.
That's why to conform against the spec and support "custom" and allow
"unrecognized_fields" everywhere where it's not sensitive we can define
custom as property which actually access data stored in
unrecognized_fields.
For context read ADR 8 in tuf/docs/adr.
Additionally, after adding the TargetFile class, when we create a
Targets an object we are now calling from dict twice - one for the main
Targets class and one for each of the complex attributes
TargetFile.from_dict() and Delegations.from_dict().
Given that the "from_dict" methods have the side effect of destroying
the given dictionary, we would need to start using deepcopy()
for our tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In the top-level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.
Additionally, after adding the MetaFile class, when we create an object
we are now calling from dict twice - one for the main class (Timestamp,
Snapshot) and one for the pacticular complex attribute -
MetaFile.from_dict(). Given that the "from_dict" methods have the
side effect of destroying the given dictionary, we would need to
start using deepcopy() for our tests.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
NOTE: making consistent_snapshot optional requires using a default value
for the argument in __init__ in Root and thus consistent_snapshot should
be rearranged in the end.
Read more: https://github.com/theupdateframework/tuf/pull/1394#issuecomment-842134961
From chapter 7 in the spec (version 1.0.17)
"Finally, the root metadata should write the Boolean
"consistent_snapshot" attribute at the root level of its keys of
attributes.
If consistent snapshots are not written by the repository,
then the attribute may either be left unspecified or be set to the
False value. Otherwise, it must be set to the True value."
We want to make sure we support repositories
without consistent_snapshot set.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Use either "if X is not None:" or a try-except instead of a "if X:".
I believe Targets.from_dict() was not really broken with previous code
but it looks suspicious and did fail the added test with a strange
exception: I expect the from_dict() methods to mainly fail with
KeyErrors, ValueErrors or AttributeErrors if file format structure
is incorrect.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
A DelegatedRole with paths=[] fails to serialize correctly (paths is not
included in the output json).
Fix the issue, modify tests to notice a regression.
Fixes#1389
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
After the implementation of a Key class representing
the public portion of a key, the method add_key() should
take an argument of type Key, instead of a dictionary.
Test cases are updated accordingly.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
In the top level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.
DelegatedRole shares a couple of fields with the Role class and that's
why it inherits it.
I decided to use a separate Delegations class because I thought it will
make it easier to read, verify and add additional helper functions.
Also, I tried to make sure that I test each level of the delegations
representation for support of storing unrecognized fields.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We should not do multiple lookups through data structures if one is
enough (here we have extra lookups on both roles and keyids).
Also in this case raising on missing key seems like the preferable
alternative so even a try-except is not needed.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
As per the specification (v1.0.1) length and hashes fields
in timestamp and snapshot metadata are optional.
We have implement this in the older API
(see https://github.com/theupdateframework/tuf/pull/1031) and we should
implement it in the new API.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add a use case for the root class to be tested in test_generic_read
and test_read_write_read_compare tests in test_apy.py
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Verify that adding an already existing key to keyid for a particular
role in Root won't create duplicate key.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In the top level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In the top level metadata classes, there are complex attributes such as
"meta" in Targets and Snapshot, "key" and "roles" in Root etc.
We want to represent those complex attributes with a class to allow
easier verification and support for metadata with unrecognized fields.
For more context read ADR 0004 and ADR 0008 in the docs/adr folder.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
In order to support ADR 0008 we would want to accept unrecognized
fields in all metadata classes.
Input that contains unknown fields in the 'signed' dictionary should
successfully deserialize into a Metadata object, and that object should
successfully serialize with the unknown fields intact.
Also, we should test that we support unrecognized fields when adding
new classes or modifying existing ones to make sure we support
ADR 0008.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
This is essentially short-hand for
JSONDeserializer().deserialize(data)
but seems much easier for the API user so may be worth it.
Metadata.from_file() now uses Metadata.from_bytes() internally.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Checks metadata expiration against a reference time (a naive datetime in UTC).
If not provided, checks against the current UTC date and time.
Returns True if expiration time is less than the reference time.
Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
In the securesystemslib pr https://github.com/secure-systems-lab/securesystemslib/pull/319
I added a new Signer interface with the purpose of supporting multiple
signing implementations.
Additionally, I added the SSlibSigner implementation of that interface
which implements the signing operation for rsa, ed25519 and ecdsa
schemes.
With this commit, I integrate the SSlibSigner into the new API in tuf.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Revert an earlier commit that moved to/from_dict metadata class
model methods to a util module of the serialization sub-package.
We keep to/from_dict methods on the metadata classes because:
- It seems **idiomatic** (see e.g. 3rd-party libaries such as attrs,
pydantic, marshmallow, or built-ins that provide default or
customizable dict representation for higher-level objects).
The idiomatic choice should make usage more intuitive.
- It feels better **structured** when each method is encapsulated
within the corresponding class, which in turn should make
maintaining/modifying/extending the class model easier.
- It allows us to remove function-scope imports (see subsequent
commit).
Caveat:
Now that "the meat" of the sub-packaged JSON serializer is
implemented on the class, it might make it harder to create a
non-dict based serializer by copy-paste-amending the JSON
serializer.
However, the benefits from above seem to outweigh the disadvantage.
See option 5 of ADR0006 for further details (#1270).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add tuf.api.serialization.util module with functions to
convert between TUF metadata class model and the corresponding
dictionary representation. These functions replace the
corresponding to/from_dict classmethods.
Configure api/pylintrc to exempt '_type' from protected member
access warning, because the underscore prefix here is only used to
avoid name shadowing.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Re-raise all errors that happen during de/serialization as custom
De/SerializationError.
Whilelist 'e', which is idiomatic for error, in api/pylintrc, and
inline exempt broad-except, which are okay if re-raised.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Currently, we are importing the "utils" module in tests/utils
with "import utils".
This could become a problem when there is another module with
the same general name "utils" and could lead to import mistakes.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Use deepcopy to ensure that the dictionaries with expected data
are not referencing the same memory as the tested ones.
Add a check asserting that metadata is not equal prior to its
update.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
It's convenient to be able to run unit test scripts directly, rather than
having to pass them as arguments to Python. This is already possible for
several of our unit tests, make it possible for all by setting the execute
bit.
Signed-off-by: Joshua Lock <jlock@vmware.com>
all test_*.py files now accept zero or more '-v' to increase tuf
logging level. The default is now ERROR.
default: ERROR
"-v": ERROR, but unittest prints test names
"-vv": WARNING
"-vvv": INFO
"-vvvv": DEBUG
Example to run a single test with DEBUG level:
python3 test_updater.py -vvvv TestUpdater.test_4_refresh
Also make test_log.py restore the log level it modifies during test.
Fixes#1093
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Change Metadata.verify(key) behavior to raise an exception if
none or multiple signatures for the passed key are found on the
Metadata object.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add convenience wrapper that takes a json string and passes it
to from_dict to create a Metadata object.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This commit better separates the Metadata class model from the
Metadata wireline format, by tailoring the constructors
towards class-based parameters and adding an additional
factory classmethod that creates Metadata objects based on the
wireline json/dictionary metadata representation. (pythonic
way of constructor overloading).
This 'from_dict' factory method recurses into the 'from_dict'
methods of each contained complex field/attribute that is also
represented by a class. Currently 'signed' is the only such
attribute.
This commit further:
- Changes optional constructor keyword arguments to mandatory
positional arguments: Reduces code and simplifies usage by
restricting it. For now, users are unlikely to call
constructor directly anyway, but the 'from_dict' factory (or
its 'from_json_file' wrapper) instead.
- Removes Signed.__expiration (datetime) vs. Signed.expires
(datestring) dichotomy: Keeping only one representation of the
same attribute in memory makes the interface simpler and less
ambiguous. We choose the datetime object, because it is more
convenient to modify. Transformation from and to the string
format required by the tuf wireline format is performed in the
corresponding metadata de/serialization methods, i.e.
('to_dict' and 'from_dict').
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Consistenly rename de/serialization interface methods, using
a 'from_' and 'to_' prefix.
read_from_json -> from_json_file
write_to_json -> to_json_file
as_json -> to_json
as_dict -> to_dict
signed_bytes -> to_canonical_bytes
The latter is also changed from a property to a method for
consistency with the other serialization methods.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove metadata factory on Signed class, for the sake of API
simplicity/non-ambiguity, i.e. it's enough to have one
way of loading any Metadata, that is:
Metadata.read_from_json
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add simple methods to create or verify signatures of the
canonical_signed property of a Metadata object.
See corresponding docstrings for behavior and design
considerations.
The commit also adds tests and updates the test setup to load
some test keys into memory.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add simple as_json Metadata method and use it instead of repository
lib's internal _get_written_metadata function in write_to_json.
This commit further adds code documentation and the possibility to
write compact json by excluding whitespace to write_to_json, and
also removes a call to the sign method from write_to_json.
The commit also adds tests.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add generic read from json class method that returns a Metadata
object with a signed field that contains the appropriate Signed
subclass, based on the signed._type field of the read metadata.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The new metadata module uses constructs that are only available
on Python >= 3.6 (typing, f-format strings, etc.).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add metadata module with container classes for TUF role metadata, including
methods to read/serialize/write from and to JSON, perform TUF-compliant
metadata updates, and create and verify signatures.
The 'Metadata' class provides a container for inner TUF metadata objects (Root,
Timestamp, Snapshot, Targets) (i.e. OOP composition)
The 'Signed' class provides a base class to aggregate common attributes (i.e.
version, expires, spec_version) of the inner metadata classes. (i.e. OOP
inheritance). The name of the class also aligns with the 'signed' field of
the outer metadata container.
Based on prior observations in TUF's sister project in-toto, this architecture
seems to well represent the metadata model as it is defined in the
specification (see in-toto/in-toto#98 and in-toto/in-toto#142 for related
discussions).
This commits also adds tests.
**TODO: See doc header TODO list**
**Additional design considerations**
(also in regards to prior sketches of this module)
- Aims at simplicity, brevity and recognizability of the wireline metadata
format.
- All attributes that correspond to fields in TUF JSON metadata are public.
There doesn't seem to be a good reason to protect them with leading
underscores and use setters/getters instead, it just adds more code, and
impedes recognizability of the wireline metadata format.
- Although, it might be convenient to have short-cuts on the Metadata class
that point to methods and attributes that are common to all subclasses of
the contained Signed class (e.g. Metadata.version instead of
Metadata.signed.version, etc.), this also conflicts with goal of
recognizability of the wireline metadata. Thus we won't add such short-cuts
for now. See:
https://github.com/theupdateframework/tuf/pull/1060#discussion_r452906629
- Signing keys and a 'consistent_snapshot' boolean are not on the targets
metadata class. They are a better fit for management code. See:
https://github.com/theupdateframework/tuf/pull/1060#issuecomment-660056376,
and #660.
- Does not use sslib schema checks (see TODO notes about validation in
doc header)
- Does not use existing tuf utils, such as make_metadata_fileinfo,
build_dict_conforming_to_schema, if it is easy and more explicit to
just re-implement the desired behavior on the metadata classes.
- All datetime's are treated as UTC. Since timezone info is not captured in
the wireline metadata format it should not be captured in the internal
representation either.
- Does not use 3rd-party dateutil package, in order to minimize dependency
footprint, which is especially important for update clients which often have
to vendor their dependencies.
However, compatibility between the more advanced dateutil.relativedelta (e.g
handles leap years automatically) and timedelta is tested.
- Uses PEP8 indentation (4 space) and Google-style doc string instead of
sslab-style. See
https://github.com/secure-systems-lab/code-style-guidelines/issues/20
- Does not support Python =< 3.5
Co-authored-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Teodora Sechkova <tsechkova@vmware.com>
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
