Use "from tuf import <module>" instead of "import tuf.<module>": this
makes it possible for vendoring tool to vendor tuf. Fix all references
to <module> in the code.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Use "from tuf import <module>" instead of "import tuf.<module>": this
makes it possible for vendoring tool to vendor tuf. Fix all references
to <module> in the code.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Black standardizes single to double quotes where feasible.
However, it doesn't seem to change double to single quotes nor adds
escape characters, as a consequence it skips standardization on
strings with mixed quotes.
Unfortunately, pylint's quote consistency check also doesn't detect
this, so the onus will remain on the reviewer in these cases.
**Unrelated changes**:
The commit still enables pylint's "check-quote-consistency" just in
case it can detect something the black doesn't.
The commit also fixes a syntax inconsistency in pylintrc.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The updated pylintrc is based on the Google Python Style Guide
pylint configuration at
https://google.github.io/styleguide/pylintrc with the following
differences:
- We don't list defaults which are applied anyway.
- We don't configure checks that seem unrelated to the code style
guide.
- We don't disable any checks that are not in conflict with the
current code or code style guide.
This has the advantage of a minimal configuration file which should
be easy to maintain and extend as required, e.g. if conflicting
code is added, or linting time becomes too long, due to unnecessary
checks.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
In the securesystemslib pr https://github.com/secure-systems-lab/securesystemslib/pull/319
I added a new Signer interface with the purpose of supporting multiple
signing implementations.
Additionally, I added the SSlibSigner implementation of that interface
which implements the signing operation for rsa, ed25519 and ecdsa
schemes.
With this commit, I integrate the SSlibSigner into the new API in tuf.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Call an instance method and a static method that are only defined
in a parent class from child instances using self (instance) and
cls (static) instead of super().
While this doesn't make a practical difference, the new syntax is
probably less confusing to the reader.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Make class docstrings wording consistent.
- Emphasize that we use the OLPC Canonical JSON specification.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
tuf.api is not designed for Python 2 compatibility. This commit
removes the following stray compatibility constructs in its
serialization subpackage:
- '__metaclass__ = abc.ABCMeta'
- six.raise_from
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Prior to this commit the (abstract) 'Signed' base class implemented
from/to_dict methods, to be used by any subclass in addition to
or instead of a custom from/to_dict method. The design led to some
confusion, especially in 'Signed.from_dict' factories, which
instantiated subclass objects when called on a subclass, which
didn't implement its own 'from_dict' method.
This commit demystifies the design, by implementing from/to_dict
on all 'Signed' subclasses, and moving common from/to_dict tasks
to helper functions in the 'Signed' class.
The newly gained clarity and explicitness comes at the cost of
slightly more lines of code.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Clarify that the TUF metadata class model is not bound to a JSON
wireline format by:
- re-wording module, class and method docstrings and code comments
to add details about custom and default serialization and the
purpose of from/to_dict methods, and
- removing the 'JsonDict' type annotation -- instead we use
generic Mapping[str, Any] for method arguments and strict
Dict[str, Any] as return value as suggested in
https://docs.python.org/3/library/typing.html#typing.Dict
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Rename _dict to json_dict to avoid wrong semantics of leading
underscore. (leading underscore was initially chosen to avoid name
shadowing)
- Rename 'serializer' argument of type 'SignedSerializer' to
'signed_serializer', to distinguish from 'serializer' argument of
type 'MetadataSerializer'.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Revert an earlier commit that moved to/from_dict metadata class
model methods to a util module of the serialization sub-package.
We keep to/from_dict methods on the metadata classes because:
- It seems **idiomatic** (see e.g. 3rd-party libaries such as attrs,
pydantic, marshmallow, or built-ins that provide default or
customizable dict representation for higher-level objects).
The idiomatic choice should make usage more intuitive.
- It feels better **structured** when each method is encapsulated
within the corresponding class, which in turn should make
maintaining/modifying/extending the class model easier.
- It allows us to remove function-scope imports (see subsequent
commit).
Caveat:
Now that "the meat" of the sub-packaged JSON serializer is
implemented on the class, it might make it harder to create a
non-dict based serializer by copy-paste-amending the JSON
serializer.
However, the benefits from above seem to outweigh the disadvantage.
See option 5 of ADR0006 for further details (#1270).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add tuf.api.serialization.util module with functions to
convert between TUF metadata class model and the corresponding
dictionary representation. These functions replace the
corresponding to/from_dict classmethods.
Configure api/pylintrc to exempt '_type' from protected member
access warning, because the underscore prefix here is only used to
avoid name shadowing.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Re-raise all errors that happen during de/serialization as custom
De/SerializationError.
Whilelist 'e', which is idiomatic for error, in api/pylintrc, and
inline exempt broad-except, which are okay if re-raised.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add serializer.json module with implementations to serialize and
deserialize TUF role metadata to and from the JSON wireline format
for transportation, and to serialize the 'signed' part of TUF role
metadata to the OLPC Canonical JSON format for signature generation
and verification.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add sub-package with 3 abstract base classes to:
- serialize Metadata objects to bytes (transport)
- deserialize Metadata objects from bytes (transport)
- serialize Signed objects to bytes (signatures)
pylint notes:
- configure tox to use api/pylintrc
- configure api/pylintrc to allow classes without public methods
(default was 2)
Design considerations
---------------------
- Why not implement de/serialization on metadata classes?
-> See ADR0006.
- Why use separate classes for serialization and deserialization?
-> Some users might only need either one, e.g. client only needs
Deserializer. Maybe there are use cases where different
implementations are used to serialize and deserialize.
- Why use separate classes for Metadata- and Signed-Serialization?
-> They require different concrete types, i.e. Metadata and
Signed as parameters, and using these specific types seems to
make the interface stronger.
- Why are de/serialize methods not class/staticmethods?
-> In reality we only use classes to namespace and define a
type annotated interface, thus it would be enough to make the
methods classmethods. However, to keep the de/serialize
interface minimal, we move any custom format configuration to
the constructor. (See e.g. "compact" for JSONSerializer in
subsequent commit).
Naming considerations
---------------------
- Why de/serialize?
-> Implies byte stream as input or output to the function, which
is what our interface needs.
- Why not marshaling?
-> Synonym for serialize but implies transport, would be okay.
- Why not encoding?
-> Too abstract and too many connotations (character, a/v).
- Why not parse?
-> Too abstract and no good opposite terms (unparse, write,
dump?)
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add root metadata class to tuf.api.metadata module and implement
(de)serialisation and modification methods.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
The Targets constructor takes seven arguments, which violates pylints
default value of five for max-arguments:
R0913: Too many arguments (7/5) (too-many-arguments)
As this feels like a coding style decision that should be made and
documented disable that test for only the Targets constructor until
a coding style decision has been made and documented as a decision
record.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Using an else after a raise results in a refactor message from pylint:
R1720: Unnecessary "elif" after "raise" (no-else-raise)
This is because the raise will exit the block, and pylint suggests that
explicit if's, rather than an if-elif-else, are clearer style. Update the
style of Metadata.verify() to match pylint expectations.
Signed-off-by: Joshua Lock <jlock@vmware.com>
A single letter variable name of 'f' causes pylint to throw a coding style
convention warning:
C0103: Variable name "f" doesn't conform to snake_case naming style
(invalid-name)
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add a minimal pylintrc to lint for new code being developed in tuf/api and
update the tox configuration to ignore tuf/api with the default pylintrc
and run an extra invocation of pylint for just the modules in tuf/api.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Our 'expires' strings are constrained by the ISO8601_DATETIME_SCHEMA
which matches regex '\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z'. This can be
parsed with just a datetime.strptime(): iso8601 module is not needed.
* Add formats.expiry_string_to_datetime() helper function
* Modify the 3 locations that used iso8601 and the api/metadata.py usage
of datetime.strptime()
* Remove related unnecessary logger setup
* Add the missing exception documentation to relevant functions (in many
cases the exception is rather unlikely as the schema has been verified
many times before this though...)
Fixes#1065
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
See:
Add root metadata class to new TUF metadata model #1137
Add classes for complex metadata fields #1139
Add input validation to simple metadata api #1140
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Change Metadata.verify(key) behavior to raise an exception if
none or multiple signatures for the passed key are found on the
Metadata object.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add convenience wrapper that takes a json string and passes it
to from_dict to create a Metadata object.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This commit better separates the Metadata class model from the
Metadata wireline format, by tailoring the constructors
towards class-based parameters and adding an additional
factory classmethod that creates Metadata objects based on the
wireline json/dictionary metadata representation. (pythonic
way of constructor overloading).
This 'from_dict' factory method recurses into the 'from_dict'
methods of each contained complex field/attribute that is also
represented by a class. Currently 'signed' is the only such
attribute.
This commit further:
- Changes optional constructor keyword arguments to mandatory
positional arguments: Reduces code and simplifies usage by
restricting it. For now, users are unlikely to call
constructor directly anyway, but the 'from_dict' factory (or
its 'from_json_file' wrapper) instead.
- Removes Signed.__expiration (datetime) vs. Signed.expires
(datestring) dichotomy: Keeping only one representation of the
same attribute in memory makes the interface simpler and less
ambiguous. We choose the datetime object, because it is more
convenient to modify. Transformation from and to the string
format required by the tuf wireline format is performed in the
corresponding metadata de/serialization methods, i.e.
('to_dict' and 'from_dict').
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Consistenly rename de/serialization interface methods, using
a 'from_' and 'to_' prefix.
read_from_json -> from_json_file
write_to_json -> to_json_file
as_json -> to_json
as_dict -> to_dict
signed_bytes -> to_canonical_bytes
The latter is also changed from a property to a method for
consistency with the other serialization methods.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove metadata factory on Signed class, for the sake of API
simplicity/non-ambiguity, i.e. it's enough to have one
way of loading any Metadata, that is:
Metadata.read_from_json
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add simple methods to create or verify signatures of the
canonical_signed property of a Metadata object.
See corresponding docstrings for behavior and design
considerations.
The commit also adds tests and updates the test setup to load
some test keys into memory.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add simple as_json Metadata method and use it instead of repository
lib's internal _get_written_metadata function in write_to_json.
This commit further adds code documentation and the possibility to
write compact json by excluding whitespace to write_to_json, and
also removes a call to the sign method from write_to_json.
The commit also adds tests.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add generic read from json class method that returns a Metadata
object with a signed field that contains the appropriate Signed
subclass, based on the signed._type field of the read metadata.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add metadata module with container classes for TUF role metadata, including
methods to read/serialize/write from and to JSON, perform TUF-compliant
metadata updates, and create and verify signatures.
The 'Metadata' class provides a container for inner TUF metadata objects (Root,
Timestamp, Snapshot, Targets) (i.e. OOP composition)
The 'Signed' class provides a base class to aggregate common attributes (i.e.
version, expires, spec_version) of the inner metadata classes. (i.e. OOP
inheritance). The name of the class also aligns with the 'signed' field of
the outer metadata container.
Based on prior observations in TUF's sister project in-toto, this architecture
seems to well represent the metadata model as it is defined in the
specification (see in-toto/in-toto#98 and in-toto/in-toto#142 for related
discussions).
This commits also adds tests.
**TODO: See doc header TODO list**
**Additional design considerations**
(also in regards to prior sketches of this module)
- Aims at simplicity, brevity and recognizability of the wireline metadata
format.
- All attributes that correspond to fields in TUF JSON metadata are public.
There doesn't seem to be a good reason to protect them with leading
underscores and use setters/getters instead, it just adds more code, and
impedes recognizability of the wireline metadata format.
- Although, it might be convenient to have short-cuts on the Metadata class
that point to methods and attributes that are common to all subclasses of
the contained Signed class (e.g. Metadata.version instead of
Metadata.signed.version, etc.), this also conflicts with goal of
recognizability of the wireline metadata. Thus we won't add such short-cuts
for now. See:
https://github.com/theupdateframework/tuf/pull/1060#discussion_r452906629
- Signing keys and a 'consistent_snapshot' boolean are not on the targets
metadata class. They are a better fit for management code. See:
https://github.com/theupdateframework/tuf/pull/1060#issuecomment-660056376,
and #660.
- Does not use sslib schema checks (see TODO notes about validation in
doc header)
- Does not use existing tuf utils, such as make_metadata_fileinfo,
build_dict_conforming_to_schema, if it is easy and more explicit to
just re-implement the desired behavior on the metadata classes.
- All datetime's are treated as UTC. Since timezone info is not captured in
the wireline metadata format it should not be captured in the internal
representation either.
- Does not use 3rd-party dateutil package, in order to minimize dependency
footprint, which is especially important for update clients which often have
to vendor their dependencies.
However, compatibility between the more advanced dateutil.relativedelta (e.g
handles leap years automatically) and timedelta is tested.
- Uses PEP8 indentation (4 space) and Google-style doc string instead of
sslab-style. See
https://github.com/secure-systems-lab/code-style-guidelines/issues/20
- Does not support Python =< 3.5
Co-authored-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Teodora Sechkova <tsechkova@vmware.com>
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
