Clarify that the TUF metadata class model is not bound to a JSON
wireline format by:
- re-wording module, class and method docstrings and code comments
to add details about custom and default serialization and the
purpose of from/to_dict methods, and
- removing the 'JsonDict' type annotation -- instead we use
generic Mapping[str, Any] for method arguments and strict
Dict[str, Any] as return value as suggested in
https://docs.python.org/3/library/typing.html#typing.Dict
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Rename _dict to json_dict to avoid wrong semantics of leading
underscore. (leading underscore was initially chosen to avoid name
shadowing)
- Rename 'serializer' argument of type 'SignedSerializer' to
'signed_serializer', to distinguish from 'serializer' argument of
type 'MetadataSerializer'.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Revert an earlier commit that moved to/from_dict metadata class
model methods to a util module of the serialization sub-package.
We keep to/from_dict methods on the metadata classes because:
- It seems **idiomatic** (see e.g. 3rd-party libaries such as attrs,
pydantic, marshmallow, or built-ins that provide default or
customizable dict representation for higher-level objects).
The idiomatic choice should make usage more intuitive.
- It feels better **structured** when each method is encapsulated
within the corresponding class, which in turn should make
maintaining/modifying/extending the class model easier.
- It allows us to remove function-scope imports (see subsequent
commit).
Caveat:
Now that "the meat" of the sub-packaged JSON serializer is
implemented on the class, it might make it harder to create a
non-dict based serializer by copy-paste-amending the JSON
serializer.
However, the benefits from above seem to outweigh the disadvantage.
See option 5 of ADR0006 for further details (#1270).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add tuf.api.serialization.util module with functions to
convert between TUF metadata class model and the corresponding
dictionary representation. These functions replace the
corresponding to/from_dict classmethods.
Configure api/pylintrc to exempt '_type' from protected member
access warning, because the underscore prefix here is only used to
avoid name shadowing.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Re-raise all errors that happen during de/serialization as custom
De/SerializationError.
Whilelist 'e', which is idiomatic for error, in api/pylintrc, and
inline exempt broad-except, which are okay if re-raised.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add serializer.json module with implementations to serialize and
deserialize TUF role metadata to and from the JSON wireline format
for transportation, and to serialize the 'signed' part of TUF role
metadata to the OLPC Canonical JSON format for signature generation
and verification.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add sub-package with 3 abstract base classes to:
- serialize Metadata objects to bytes (transport)
- deserialize Metadata objects from bytes (transport)
- serialize Signed objects to bytes (signatures)
pylint notes:
- configure tox to use api/pylintrc
- configure api/pylintrc to allow classes without public methods
(default was 2)
Design considerations
---------------------
- Why not implement de/serialization on metadata classes?
-> See ADR0006.
- Why use separate classes for serialization and deserialization?
-> Some users might only need either one, e.g. client only needs
Deserializer. Maybe there are use cases where different
implementations are used to serialize and deserialize.
- Why use separate classes for Metadata- and Signed-Serialization?
-> They require different concrete types, i.e. Metadata and
Signed as parameters, and using these specific types seems to
make the interface stronger.
- Why are de/serialize methods not class/staticmethods?
-> In reality we only use classes to namespace and define a
type annotated interface, thus it would be enough to make the
methods classmethods. However, to keep the de/serialize
interface minimal, we move any custom format configuration to
the constructor. (See e.g. "compact" for JSONSerializer in
subsequent commit).
Naming considerations
---------------------
- Why de/serialize?
-> Implies byte stream as input or output to the function, which
is what our interface needs.
- Why not marshaling?
-> Synonym for serialize but implies transport, would be okay.
- Why not encoding?
-> Too abstract and too many connotations (character, a/v).
- Why not parse?
-> Too abstract and no good opposite terms (unparse, write,
dump?)
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove the magic number, a whence value of 2 for file.seek(), and instead
use the io.SEEK_END constant from the io module.
Signed-off-by: Joshua Lock <jlock@vmware.com>
- Drop Python 2.7 from GitHub Actions workflows. Note: There is likely
additional cleanup that can be done to the workflow now we no longer
care about supporting Python 2.7.
- No longer tell dependabot to ignore idna updates.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Remove references to, and handling of, Python 2.7 in our project scaffolding:
- updated python_requires in setup.py to state our intent to support
Python 3.6 and above (but not Python 4, yet)
- Drop no longer required dependencies in setup.py, and requirements-*.txt
(further refinement of requirements files will be handled in #1161)
- Remove Python 2.7 from our tox environments
Signed-off-by: Joshua Lock <jlock@vmware.com>
Ensure that the newly added files' docstrings adhere to the
recently adopted code style guideline (#1232).
Small code style improvements in comments and imports.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Use a common test level constant for defining
the host address forming the download URL on
the client side.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
* Move FetcherInterface to tuf/client/ directory: This way everything
inside that directory is clearly part of client API, and everything
outside _may_ be more of an implementation detail (settings is still
an unfortunate exception)
* Keep RequestsFetcher in tuf/ for same reasons: it's just the default
implementation, not explicitly part of client API
An even clearer division would be if we moved all the client specific
implementation details (download.py, mirrors.py, requests_fetcher.py)
to tuf/client/_internal/ but that's a larger change...
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
A custom error is required so that updater is able to special case
403 & 404 status codes.
Rewrite the test case a bit to be more readable.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Add test cases to test_fetcher and test_download that
decrease default chunk size and download data in more
than one chunk.
Small code-style improvements.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Use '>=' as the defensive version of the equality check.
Add a comment describing the need of a chunks() generator.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
On Windows (Github Actions) the lookup for 'localhost' takes 1 second.
This is because:
- Windows retries connect() with a timeout
- the machine has IPv6 and IPv4 but Testserver only binds the port on IPv4
- the test clients connect to 'localhost'
Since socketserver.TCPServer does not seem to support IPv6 before 3.8,
just replace 'localhost' with '127.0.0.1' in client-side URLs.
See #1257
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
- Update RequestsFetcher.fetch to return a generator object.
- Update _download_file to skip connection time when calculating
average download speed.
- Write chunk to temp file before exiting the fetcher loop
on error.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Initialize Updater with an external implementation of
FetcherInterface. If not provided, tuf.fetcher.RequestsFetcher
is used.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Abstract the network IO. Move the network operations from
tuf.download to the RequestsFercher class which is TUF's
implementation of the abstract FetcherInterface.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
The new class FetcherInterface defines an interface for
abstract network download which can be implemented for a
variety of network libraries and configurations.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Dependabot pushes to main repository and ends up triggering two builds
every time (one for PR, one for push): limit the rule for build-on-push
to apply to develop branch only.
If release branches are used later on they should be added to list here.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
On Python3 bump cryptography from 3.3.1 to 3.4.5.
On python2 bump from 3.3.1 to 3.3.2 (3.3-branch is the last branch
with python2 support).
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
