Commit graph

4560 commits

Author SHA1 Message Date
Jussi Kukkonen
15e84dfb2e GH actions: limit GitHub token visibility
Token should be visible to only the code that actually needs it.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-11-01 09:47:50 +02:00
Jussi Kukkonen
589ed9e0d4
Merge pull request #1635 from jku/key-rotation-tests
tests: Add ngclient root key rotation tests
2021-10-27 19:07:06 +03:00
Jussi Kukkonen
e817473e3c tests: Add root key rotation tests
Add one test with 1 subtests for various root key rotation situations.

The test data definition format is a bit tricky but I tried to document
that in the test function docstring.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-27 19:01:23 +03:00
Jussi Kukkonen
ad80bd96c6 tests: Mark RepositorySimulator. create_key() static
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-27 18:57:12 +03:00
Jussi Kukkonen
fd40dfc094 tests: Refactor simulator signer handling
Store signers with their keyids so they are easier to remove.
The signers structure now looks like:
{
  "role1": {
    "keyidA": SSlibSigner,
    "keyidB": SSlibSigner,
  }
}

Add convenience method for adding a signer.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-27 18:57:08 +03:00
Jussi Kukkonen
7b8ff220b1
Merge pull request #1604 from jku/ngclient-api-polish
Ngclient api polish
2021-10-27 18:40:49 +03:00
Jussi Kukkonen
1d115b57b6
Merge pull request #1630 from MVrachev/validate-role
Metadata API: validate root role names
2021-10-27 18:39:58 +03:00
Jussi Kukkonen
6aaa1ead59 ngclient: Refactor target path generation
Also tweak the docstrings: the "caching" target_dir usage is
presented in the module doc example: there should be no need for
additional comments in the methods themselves as long as the argument
docs are readable.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-27 10:19:00 +03:00
Jussi Kukkonen
d519a413b0 ngclient: Rename get_one_valid_targetinfo()
This is slightly cosmetic but rename get_one_valid_targetinfo to
get_targetinfo:
* The function name is long without any reason: "one" and "valid" are
  always implicit
* shortening makes code (incl. our examples and tests) easier to read
* We're also already changing updater API (compared to legacy) so this
  alone does not break things -- it's also not a difficult "port".

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-27 09:55:57 +03:00
Jussi Kukkonen
9b761b8620 ngclient: Simplify caching
Remove updated_targets() as it doesn't fit the rest of the API.

In its stead add find_cached_target() which has a similar signature
as download_target(): both accept an optional local filepath as
argument and return full local filepath. In the
find_cached_target() case None is returned if the local file is not the
correct target file.

Updater constructor gets a new optional target_dir argument: This means
client can avoid giving a local filepath as an argument to
find_cached_target()/download_target() -- Updater will then generate a
filename within targets_dir.

A reasonable use pattern (when targets_dir is set in constructor):

    info = updater.get_one_valid_targetinfo("targetname")
    path = updater.find_cached_target(info)
    if path is None:
        path = updater.download_target(info)

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-27 09:55:57 +03:00
Jussi Kukkonen
cc9f3876c4 tests: Shorten variable names to reasonable length
Otherwise absolutely everything is split on multiple lines.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-27 09:55:57 +03:00
Martin Vrachev
4158272a7a Use TOP_LEVEL_ROLE_NAMES across TUF
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-10-25 15:58:56 +03:00
Martin Vrachev
9bc55ee568 Metadata API: validate root role names
Validate that root role names are 4 and that they are exactly
"root", "snapshot", "targets" and "timestamp" as described in
the spec:
https://theupdateframework.github.io/specification/latest/#root-role

Additionally, fix the valid_roots dataset, so each of the cases contains
the top metadata role names inside the roles dictionary.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-10-25 15:58:55 +03:00
Martin Vrachev
59b5498918 Add TOP_LEVEL_ROLE_NAMES constant
This constant can be used across tuf without defining it each time.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-10-25 15:32:43 +03:00
Jussi Kukkonen
2206fc917e
Merge pull request #1631 from MVrachev/fix-test-error
Test metadata files: bump expiration date and resign
2021-10-25 11:02:34 +03:00
Martin Vrachev
d3d2f57f3a Test files: bump expiration date and resign
Our newly added metadata files in the
tests/repository_data/fishy_rolenames/metadata directory have an expiry
date until "2021-10-22T11:21:56Z" and today while running the tests on
develop branch I recived this error:
ExpiredMetadataError("Metadata X expired on Fri Oct 22 11:21:56 2021")
when running the tests in tests/test_updater.py file and more precisly
the TestUpdaterRolenames.test_unusual_rolenames() test.

That's why I decided to bump the expiration date to a random time in
the future (October 22-nd 2050) and I had to resign all of the metadata
files.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-10-23 18:39:22 +03:00
Joshua Lock
6d18d53ec0
Merge pull request #1626 from joshuagl/joshuagl/build
Modernise packaging configuration
2021-10-21 20:01:55 +01:00
Joshua Lock
de1a3af019 build: more intentional about what's included in sdist
Our sdist has typically included everything from git apart from the CI
related files (.github/*, .fossa.yml, .readthedocs.yaml). Update our
MANIFEST.in and the check-manifest section of setup.cfg to be explicit
about this.

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:30:46 +01:00
Joshua Lock
352f6c2c56 Add vscode project directory to gitignore
Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:30:46 +01:00
Joshua Lock
ada35c9e8f build: add pyproject.toml to list build tools
Per PEP 517 the pyproject.toml file lists tools required when building the
project with build: https://pypa-build.readthedocs.io/en/stable/index.html

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:30:46 +01:00
Joshua Lock
52ad17a710 build: update check-manifest options
Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:30:46 +01:00
Joshua Lock
ac0ea24ca9 Remove references to setup.py
We now use a static setup.cfg instead, update __init__.py and
docs/RELEASE.md to point there.

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:30:46 +01:00
Joshua Lock
38ea974674 build: convert to static setuptools metadata
This is preferred and removes the crutch of invoking setup.py as a CLI
tool for building dists.

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:30:46 +01:00
Joshua Lock
d3e34acd98 build: add docs to project_urls
Add a 'Documentation' entry to project_urls pointing to our stable docs
on readthedocs.io. This will result in a 'Documentation' entry under the
'Project links' section on PyPI.

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:28:08 +01:00
Joshua Lock
36242adc74 build: recommend using build, not setup.py
Invoking setup.py directly is deprecated, see:
https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

Therefore:
* remove the executable bit from setup.py's permissions
* remove the shebang entry from setup.py
* update the comments in setup.py to recommend using build to create dists
  and pip to install them

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:28:08 +01:00
Joshua Lock
4fd35434bd build: update MANIFEST.in to match sdist
Update the MANIFEST.in to be explicit about what we choose to ship in our
sdist. This _does not_ result in any additional files being included in
our sdist, but does remove warnings from build.

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:28:08 +01:00
Joshua Lock
08decea2d0 Remove unused .gitmodules directory
Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:28:07 +01:00
Joshua Lock
2ab518b3f0 build: cleanup setup.cfg
* List license files in a new metadata section
* Remove .travis.yml from check-manifest section's ignore entry

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:28:07 +01:00
Joshua Lock
2ed51167f6 build: add build and release deps to requirements-dev
build, twine and wheel packages should all be installed in order to be
able to build and release python-tuf -- add those dependencies to
requirements-dev.txt

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:28:07 +01:00
Joshua Lock
9e113d6135 docs/RELEASE: use build for building dists
This is the PyPA recommend tool for building Python packages:
https://packaging.python.org/guides/distributing-packages-using-setuptools/

Furthermore, direct invocation of setup.py is long ago deprecated:
https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

Signed-off-by: Joshua Lock <jlock@vmware.com>
2021-10-21 13:28:07 +01:00
Jussi Kukkonen
69eb29fc80
Merge pull request #1628 from MVrachev/add-python3.10
Add python3.10 and fix test failures on python3.10
2021-10-21 14:40:43 +03:00
Martin Vrachev
2e94e39275 Use quotes for python version for github workflows
Fix GitHub workflow failures by using quotes for python versions.
It seems that adding `3.10` as a number is transformed then to `3.1`
which as a result is translated to Python version 3.1 instead of Python
version 3.10.
This seems to work for other projects as well:
https://github.com/MasoniteFramework/masonite4/blob/master/.github/workflows/pythontest.yml
https://github.com/python-pillow/Pillow/blob/main/.github/workflows/test-windows.yml
https://github.com/PyGithub/PyGithub/blob/master/.github/workflows/ci.yml

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-10-21 14:32:05 +03:00
Martin Vrachev
6ff852ad0f Add support for python 3.10
Python 3.10 is released on October 4-th 2021 and it seems
logical to add support for it as it doesn't require any major effort
from the project.

For reference read:
https://www.python.org/downloads/release/python-3100/

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-10-21 14:32:04 +03:00
Martin Vrachev
4c81340610 Replace depricated ssl function and fix CI errors
When I tried adding support for Python3.10 we had CI errors due to
test failures: https://github.com/theupdateframework/python-tuf/pull/1610/checks?check_run_id=3861875325
The problem comes from the fact that we start a subprocess
executing simple_https_server.py, but then we fail to communicate the
message we expect from the server process to the main process actually
running the test. We expect our custom message to be the first line
printed from the server process, but instead, a deprecation warning is
printed first about the usage of ssl.wrap_socket(). Our custom message
is printed second.
As of Python 3.7 this function has been deprecated:
https://docs.python.org/3/library/ssl.html#ssl.wrap_socket and for
whatever the reason we didn't get a warning when using it before.

My fix does what is suggested in the warning and replaces the usage of
ssl.wrap_socket() by instantiating a ssl.SSLContext object and then
calling SSLContext.wrap_socket().
This removes the warning.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-10-21 14:15:29 +03:00
lukpueh
0cf6ba2258
Merge pull request #1620 from lukpueh/misc-metadata-api-docs
Add missing method args docs in metadata API
2021-10-20 15:51:07 +02:00
Jussi Kukkonen
4aef2b2ae6
Merge pull request #1611 from theupdateframework/dependabot/pip/charset-normalizer-2.0.7
build(deps): bump charset-normalizer from 2.0.6 to 2.0.7
2021-10-20 12:04:39 +03:00
dependabot[bot]
9864e8ef5e
build(deps): bump charset-normalizer from 2.0.6 to 2.0.7
Bumps [charset-normalizer](https://github.com/ousret/charset_normalizer) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/ousret/charset_normalizer/releases)
- [Commits](https://github.com/ousret/charset_normalizer/compare/2.0.6...2.0.7)

---
updated-dependencies:
- dependency-name: charset-normalizer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-20 08:40:11 +00:00
Jussi Kukkonen
6839e81edc
Merge pull request #1609 from theupdateframework/dependabot/pip/certifi-2021.10.8
build(deps): bump certifi from 2021.5.30 to 2021.10.8
2021-10-20 11:39:43 +03:00
dependabot[bot]
b642a44ce1
build(deps): bump certifi from 2021.5.30 to 2021.10.8
Bumps [certifi](https://github.com/certifi/python-certifi) from 2021.5.30 to 2021.10.8.
- [Release notes](https://github.com/certifi/python-certifi/releases)
- [Commits](https://github.com/certifi/python-certifi/compare/2021.05.30...2021.10.08)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-20 08:25:45 +00:00
Jussi Kukkonen
761349919b
Merge pull request #1621 from theupdateframework/dependabot/pip/cffi-1.15.0
build(deps): bump cffi from 1.14.6 to 1.15.0
2021-10-20 11:23:45 +03:00
Jussi Kukkonen
d19d40353d
Merge pull request #1616 from theupdateframework/dependabot/pip/idna-3.3
build(deps): bump idna from 3.2 to 3.3
2021-10-20 11:23:33 +03:00
Jussi Kukkonen
e86a61571c
Merge pull request #1625 from jku/release-0.19
Release 0.19.0
2021-10-19 17:21:41 +03:00
Jussi Kukkonen
f2f7f9df53 Release 0.19.0
For users of legacy client (tuf/client/) this is purely a security fix
release with no API or functionality changes. For ngclient and Metadata
API, some API changes are included.

All users are advised to upgrade.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-19 17:17:49 +03:00
Joshua Lock
4ad7ae48fd
Merge pull request from GHSA-wjw6-2cqr-j4qr
Fix client issue with rolenames as filenames
2021-10-19 14:21:29 +01:00
dependabot[bot]
0b9317a8a9
build(deps): bump cffi from 1.14.6 to 1.15.0
Bumps [cffi](http://cffi.readthedocs.org) from 1.14.6 to 1.15.0.

---
updated-dependencies:
- dependency-name: cffi
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-14 10:06:30 +00:00
Lukas Puehringer
86d4b35a98 Add missing method args docs in metadata API
Affected methods are:

- Root.add_key()
- Root.remove_key()
- Targets.add_key()
- Targets.remove_key()
- DelegatedRole.is_delegated_path()
- Key.from_securesystemslib_key()

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-10-14 11:12:05 +02:00
Jussi Kukkonen
677377899e tests: Use spec version from Metadata API
When constructing new metadata from scratch, use the spec version
the API supports.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-13 15:59:56 +03:00
Jussi Kukkonen
7da1f1e41b legacy client: Remove dead code
_fileinfo_has_changed() and _update_fileinfo() have been unused internal
methods since 2016. Remove them.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-10-13 15:59:56 +03:00
Jussi Kukkonen
98e97e31d9 legacy client: Do local filename encoding in all places
The original commit 051b8229 handled the loading and saving metadata
cases but the legacy client actually checks for the files existence
in various other places:
 * _update_versioninfo() never reads the file but operates differently
   depending on whether the file exists or not
 * _move_current_to_previous() that copies files around
 * MultiRepoUpdater initialization: this only handle root.json so
   is still correct
 * _update_fileinfo() which is dead code

Fix the first two of these cases.
2021-10-13 15:59:56 +03:00
Jussi Kukkonen
f569754f5e tests: Fix a bug in RepoSimulator signer lookup 2021-10-13 15:59:56 +03:00