Commit graph

4478 commits

Author SHA1 Message Date
Jussi Kukkonen
ed520ee55d Metadata API: Improve serialization docs
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-23 17:52:55 +03:00
Jussi Kukkonen
a77c0831e7 docs: rename "helpers" to "supporting classes"
Also add a summary to the page -- unfortunately getting a standard
TOC would require creating a rst page for each class.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-23 17:52:55 +03:00
Jussi Kukkonen
892aa04cb3 Metadata API: Rewrite module doc
Rewrite the module docstring to better fit the current sphinx documentation.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-23 17:52:55 +03:00
Jussi Kukkonen
424cc3282b docs: Include the Serialization interfaces
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-23 17:52:55 +03:00
Jussi Kukkonen
d5743c2312 metadata API docs: remove duplication
Annotations already include type info and e.g. Optional:
remove those from text where useful.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-23 17:52:55 +03:00
Jussi Kukkonen
21ce5e2915 Hide to_dict()/from_dict()
These are helpers for serialization implementers, not useful for
Metadata API users.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-23 17:52:55 +03:00
Jussi Kukkonen
217bd9dbc7 Separate API documentation pages
This makes the individual pages easier to read.

Use some autodoc configuration so we can have less config
in the automodule/autoclass declarations.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-23 17:52:52 +03:00
Jussi Kukkonen
868afda420 docs: Change object attribute doc style
Situation before
* constructor args are not documented
* object attributes are documented
* sphinx cannot show object attribute type annotations
* attribute docs take a lot of vertical space

Now:
* constructor args are documented
* sphinx can show annotated types of constructor args
* class docstring now explains the attributes are the same as
  constructor args (and attributes are not explicitly documented)

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-23 17:51:46 +03:00
Jussi Kukkonen
cc1f95e789
Merge pull request #1537 from MVrachev/role-name-uniqueness
Enforce role name uniqueness and add Targets key helpers
2021-09-22 11:12:52 +03:00
Martin Vrachev
f00295f147 API CHANGE: ValueError in add/remove key in Root
This is an API change to the exceptions thrown in Root.add_key()
and Root.remove_key().
The reason for that change is that in my opinion the correct exceptions
in these cases should be "ValueError" instead of "KeyError" as
the problems are in the given values - role doesn't exist or
key is not used by a particular role.

Additionally, document the thrown exceptions in "Root.add_key" and
add a test which invokes that exception.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-21 12:20:09 +03:00
Martin Vrachev
e27070305f Metadata API: Add key helpers in Targets
Root class has the functionality to add and remove keys for delegated
metadata (add_key()/remove_key()) but the other delegator Targets does
not.
It should provide the same/similar functionality.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-21 12:20:09 +03:00
Martin Vrachev
1a5912aa7c Remove some unused imports
We can remove the conditional imports from tests as now we support
python versions 3.6+.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-21 12:20:09 +03:00
Martin Vrachev
510078b542 Move tests to test_metadata_serialization
Move the Delegation class serialization tests from "test_api.py"
to test_metadata_serialization.py module focused on serialization
testing.

Additionally, a test for empty keys and roles will be added in my
upcomming pr #1511.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-21 12:20:09 +03:00
Martin Vrachev
f8620c1992 API CHANGE: enforce role name uniqueness
The spec does not say anything about role name uniqueness in a
delegations object, but I believe we cannot safely allow multiple roles
with the same role name in the roles array of a delegations object.
If we did then the roles could have different keyids, and then we would
end up in a situation where metadata may be both a valid delegation
and an invalid delegation at the same time, depending on how the role
gets chosen and that does not seem like the intention of the design.
There is an issue open in the specification with number 167 about
that issue.

Regardless of the Metadata API, I think we should enforce role name
uniqueness.
I chose to change the data structure containing roles to
OrderedDict, where keys are role names and values are DelegatedRole
instances.
This made sense to me as role names are the unique identifier of a role
and their order is important to the way they are traversed afterward.

Note: we can't use OrderedDict as type annotation until we drop support
for Python 3.6:
https://docs.python.org/3/library/typing.html#typing.OrderedDict
That's why I used quotes around "OrderedDict" annotation, because I
can't import it.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-21 12:15:45 +03:00
Jussi Kukkonen
afc67d967b
Merge pull request #1586 from theupdateframework/dependabot/pip/charset-normalizer-2.0.6
build(deps): bump charset-normalizer from 2.0.5 to 2.0.6
2021-09-21 11:59:44 +03:00
Jussi Kukkonen
67c52987a6
Merge pull request #1446 from MVrachev/snapshot-property
Metadata API: change meta type in Timestamp
2021-09-20 14:39:55 +03:00
Martin Vrachev
bf12e7565f Metadata API: change meta type in Timestamp
In Timestamp, the only valid "meta" value is the dictionary representing
meta information for the snapshot file. This makes the API unnecessarily
complicated and requires validation that only information about snapshot
is available inside "meta".
Together with the python-tuf maintainers, we decided that snapshot meta
information will not be represented by a "meta" dictionary but instead
by a MetaFile instance and with this it will diverge from the
specification.
Additionally, to prevent confusion, I will rename the "meta" attribute
to "snapshot_meta" as this attribute will be related only to meta
information about snapshot.

This decision is coherent with ADR9 and the rationale
behind it is to provide easier, safer, and direct access to the
snapshot meta information.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-20 14:09:38 +03:00
dependabot[bot]
ba28b5a560
build(deps): bump charset-normalizer from 2.0.5 to 2.0.6
Bumps [charset-normalizer](https://github.com/ousret/charset_normalizer) from 2.0.5 to 2.0.6.
- [Release notes](https://github.com/ousret/charset_normalizer/releases)
- [Commits](https://github.com/ousret/charset_normalizer/compare/2.0.5...2.0.6)

---
updated-dependencies:
- dependency-name: charset-normalizer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 10:26:34 +00:00
Jussi Kukkonen
234cf64f0f
Merge pull request #1585 from jku/coveralls-fin-not-required
CI: Do not require coveralls-fin to succeed
2021-09-20 13:22:45 +03:00
Jussi Kukkonen
65fc968b7f CI: Do not require coveralls-fin to succeed
We already do not require individual build uploads to succeed: let's
also not require the final step to succeed.

The immediate context for this is that coveralls has been down for
three days now.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-20 12:51:32 +03:00
Joshua Lock
48eaeabef6
Merge pull request #1583 from jku/improve-ci-github-ui
GitHub CI workflow: tweak names
2021-09-20 10:48:32 +01:00
Jussi Kukkonen
553a4d3284
Merge pull request #1581 from theupdateframework/dependabot/pip/charset-normalizer-2.0.5
build(deps): bump charset-normalizer from 2.0.4 to 2.0.5
2021-09-17 19:43:40 +03:00
Jussi Kukkonen
4110a1cf9c GitHub workflow: Use Python 3.x for lint
pylint 2.7 supports Python 3.9. This issue might reappear with next
Python release but let's deal with that if it happens.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-17 19:34:08 +03:00
Jussi Kukkonen
b59679c374 GitHub CI workflow: tweak names
Currently the github UI dropdown for checks looks useless since
checks are named "Run TUF tests and...".

Tweak the workflow and job names to hopefully fit the actual
step name in the UI.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-17 19:34:08 +03:00
Jussi Kukkonen
c5e0bddcd9
Merge pull request #1584 from MVrachev/fix-warning
Fix pylint warnings
2021-09-17 19:25:38 +03:00
Martin Vrachev
f9442a507d Remove some uses of "global"
Simplify the code by removing some of the uses of "global" by using
alternativies to the assignements.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-17 19:15:06 +03:00
Martin Vrachev
1a96bdd2b9 Fix pylint warnings
New pylint warnings appeared with code "W0602: Using global for X but
no assignment is done (global-variable-not-assigned)" where X is a
global variable.
In the lines pointed by the warnings, "global" was used inside a
function scope in order to enable assignment of the global variable to
a new value in the function.
The reason for these warnings was that pylint noticed we are using
"global" for the variable X inside a function scope without actually
assigning it to a new value.
Overall that was true for the reported cases and I removed the use of
"global", but there were only two cases where pylint reported a
false-positive. Then we were using "global", but pylint didn't
understand there were assignments.
In those cases, I disabled the warnings.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-17 18:51:51 +03:00
Jussi Kukkonen
279ac7daf8
Merge pull request #1533 from MVrachev/update-metadata-api-documentation
Metadata API: improve metadata module documentation
2021-09-17 13:48:24 +03:00
Martin Vrachev
59c486bf57 Metadata API: improve module documentation
Clarify the purpose of metadata API and that it's a low-level API
and as such it doesn't use concepts like "repository" or
"trusted collection of metadata" and don't implement the repository
logic or client updater workflow.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-09-16 12:46:58 +03:00
dependabot[bot]
42eedcae56
build(deps): bump charset-normalizer from 2.0.4 to 2.0.5
Bumps [charset-normalizer](https://github.com/ousret/charset_normalizer) from 2.0.4 to 2.0.5.
- [Release notes](https://github.com/ousret/charset_normalizer/releases)
- [Commits](https://github.com/ousret/charset_normalizer/compare/2.0.4...2.0.5)

---
updated-dependencies:
- dependency-name: charset-normalizer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-15 10:07:20 +00:00
Jussi Kukkonen
6a5b64295c
Merge pull request #1521 from avelichka/targetfile-from-data
Add an option to create TargetFile from data/file
2021-09-13 09:50:35 +03:00
Jussi Kukkonen
57985a0e30
Merge pull request #1574 from jku/ngclient-persist-metadata-safely
Ngclient: persist metadata safely
2021-09-11 11:57:18 +03:00
Velichka Atanasova
65fd1aaf8a Add an option to create TargetFile from data/file
This is a repository tooling use case but also helpful when testing.
It could be useful when we need to update the targets object.

Signed-off-by: Velichka Atanasova <avelichka@vmware.com>
2021-09-10 13:59:33 +03:00
Jussi Kukkonen
09534816bd
Merge pull request #1569 from jku/updater-tests-with-simulated-repo
test ngclient with simulated repo
2021-09-10 11:28:32 +03:00
Jussi Kukkonen
ad813a5d0d tests: Add type checks suggested by mypy
also black fixes.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-10 09:59:03 +03:00
Jussi Kukkonen
87c200d014 tests: Add state dumping into RepositorySimulator
if state dumping is enabled with e.g.
    python3 test_updater_with_simulator.py --dump
The repository state can be dumped at will.

Modify the test so it dumps the state on every updater refresh if
--dump is set.

Add a root modifying case to test_refresh()

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-10 09:59:03 +03:00
Jussi Kukkonen
d64730b1e6 tests: RepositorySimulator: Add special handling for roots
We need to store past versions of root: that means an explicit
publish step (publish_root()) is required. It stores a serialization of
current root as a new version: fetch() then serves only these
serialized root versions.

Add a few tests demonstrating how to create root versions and change
signatures.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-10 09:59:03 +03:00
Jussi Kukkonen
c7f106cf89 ngclient: Improve logging
Use info level to log which versions we currently have as trusted
metadata versions.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-10 09:59:03 +03:00
Jussi Kukkonen
4e5980e89d tests: Start testing ngclient with repo simulator
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-10 09:59:03 +03:00
Jussi Kukkonen
5d10735fe2 TrustedMetadataSet: Improve module docstring
Explain what "intermediate" metadata is and how it affects the loading
process.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-10 09:59:03 +03:00
Jussi Kukkonen
d018279e21 ngclient: Fix rollback checks
The rollback checks themselves work, but they create a situation
where Updater does not realize that it needs to download e.g. a new
snapshot because the local snapshot is valid as _intermediate_ snapshot
(that can be used for rollback protection but nothing else), but is not
valid as final snapshot.

Raise in the end of update_snapshot and update_timestamp if the files
are not valid final metadata: this way the intermediate metadata does
get loaded but Updater also knows it is not the final metadata.

This modifies the existing tests but does not yet test the situation
described in the first paragraph.

Fixes #1563

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-10 09:59:03 +03:00
Jussi Kukkonen
a102f95a86
Merge pull request #1572 from joshuagl/joshuagl/dependabot
Have dependabot also monitor actions
2021-09-09 17:22:17 +03:00
Jussi Kukkonen
bc05a1071e tests: Make sure ngclient stores the metadata we expect
Do not add similar assertions for test_refresh_on_consistent_targets():
The test is broken and can't actually update metadata (#1573).

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-08 16:03:03 +03:00
Suvaditya Mukherjee
ab81cfba7f Fixes #1526 to make the persist_metadata function an atomic operation
Co-authored-by: Jussi Kukkonen <jkukkonen@vmware.com>
Signed-off-by: Suvaditya Mukherjee <suvadityamuk@gmail.com>
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-09-08 15:33:11 +03:00
Jussi Kukkonen
ea9acf2bfd
Merge pull request #1564 from sechkova/improve-coverage
Add ngclient to coverage report
2021-09-08 13:07:01 +03:00
Jussi Kukkonen
5714885df9
Merge pull request #1486 from MVrachev/change-adr8
ADR 8: change "Decision outcome"
2021-09-08 13:05:14 +03:00
Joshua Lock
4f30d0ce96 Have dependabot also monitor actions
Have dependabot monitor our GitHub Actions for updates too

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-09-08 10:56:16 +01:00
Teodora Sechkova
6a178f4c96
Apply black to test_updater_ng.py
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-09-07 14:46:24 +03:00
Teodora Sechkova
867c2b39f0
Improve ngclient/updater.py coverage
Add tests covering missing branches of the Updater
code. Inlcude ngclient in the total coverage report.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-09-07 14:42:34 +03:00
Teodora Sechkova
61e8f40346
Update tests/test_trusted_metadata_set.py
Modify root tests to cover both loading inital
root metadata and updating it.

Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
2021-09-07 14:41:08 +03:00