Commit graph

5008 commits

Author SHA1 Message Date
Lukas Puehringer
a1a71c11a1 build: update CI/CD workflow to run in series
- Change CI workflow to also run on push to (release) tag
- Change CD workflow to run on successful CI run, and only if a
  (release) tag push triggered the CI

NOTE: Unfortunately the setup is not very robust
      (see code comment in cd.yml)

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2022-04-07 12:15:39 +02:00
Lukas Puehringer
5bfe897335 build: update CD workflow to create GH release
- Create preliminary GitHub release (X.Y.Z-rc) in 'build' job,
  using popular 3rd-party 'softprops/action-gh-release'.
- Finalize GH release in 'release' job using custom GH script.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2022-04-06 17:30:56 +02:00
Lukas Puehringer
faef040407 build: add GH workflow to build + release on PyPI
Add workflow with two jobs to build and publish on PyPI.  The
release job waits for the build job and uses a custom release
environment, which can be configured to require review.

To share the build artifacts between the jobs and to make them
available for intermediate review, they are stored using
'actions/upload-artifact' and 'actions/download-artifact'.
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts

To upload the build artifacts to PyPI, the PyPA recommended
'pypa/gh-action-pypi-publish' is used.
https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

**Caveat**
The URL to grab the artifacts, e.g. for review, requires knowledge
of action ID and artifact ID, and a login token (no special
permissions). This makes it a bit cumbersome to fetch the artifacts
with a script and compare them to a local build.
https://docs.github.com/en/actions/managing-workflow-runs/downloading-workflow-artifacts

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2022-04-06 17:30:13 +02:00
Jussi Kukkonen
d36b701bca
Merge pull request #1930 from theupdateframework/dependabot/pip/black-22.3.0
build(deps): bump black from 22.1.0 to 22.3.0
2022-03-29 13:36:48 +03:00
dependabot[bot]
811000f272
build(deps): bump black from 22.1.0 to 22.3.0
Bumps [black](https://github.com/psf/black) from 22.1.0 to 22.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.1.0...22.3.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-29 10:08:45 +00:00
Jussi Kukkonen
71259a3e75
Merge pull request #1925 from theupdateframework/dependabot/pip/mypy-0.942
build(deps): bump mypy from 0.941 to 0.942
2022-03-29 09:14:00 +03:00
dependabot[bot]
e1e8645bac
build(deps): bump mypy from 0.941 to 0.942
Bumps [mypy](https://github.com/python/mypy) from 0.941 to 0.942.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.941...v0.942)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-28 13:11:23 +00:00
Jussi Kukkonen
3ec455ca8a
Merge pull request #1928 from theupdateframework/dependabot/pip/types-requests-2.27.15
build(deps): bump types-requests from 2.27.14 to 2.27.15
2022-03-28 16:10:29 +03:00
dependabot[bot]
10f7375101
build(deps): bump types-requests from 2.27.14 to 2.27.15
Bumps [types-requests](https://github.com/python/typeshed) from 2.27.14 to 2.27.15.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-28 11:49:14 +00:00
Jussi Kukkonen
bde78bcb9f
Merge pull request #1927 from theupdateframework/dependabot/pip/pylint-2.13.2
build(deps): bump pylint from 2.12.2 to 2.13.2
2022-03-28 14:48:41 +03:00
dependabot[bot]
b482886b92
build(deps): bump pylint from 2.12.2 to 2.13.2
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.12.2 to 2.13.2.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.12.2...v2.13.2)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-28 10:09:17 +00:00
Lukas Pühringer
57c610df15
Merge pull request #1926 from jku/verify-release-imports
verify_release: Warn about missing requirements
2022-03-28 09:52:19 +02:00
Jussi Kukkonen
bf878ceaa6 verify_release: Warn about missing requirements
This is mostly useful for build module as it's not imported otherwise:
we explicitly call "python -m build" so everything works like in a
real release build.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2022-03-25 11:50:15 +02:00
Jussi Kukkonen
81637596ec
Merge pull request #1923 from theupdateframework/dependabot/pip/urllib3-1.26.9
build(deps): bump urllib3 from 1.26.8 to 1.26.9
2022-03-25 09:03:28 +02:00
Jussi Kukkonen
d1c52b5bb5
Merge pull request #1919 from theupdateframework/dependabot/pip/cryptography-36.0.2
build(deps): bump cryptography from 36.0.1 to 36.0.2
2022-03-25 09:03:11 +02:00
Lukas Pühringer
7da03ee40d
Merge pull request #1913 from jku/verify-release
build: Add verify-release script
2022-03-24 13:50:48 +01:00
Jussi Kukkonen
6819d4174a verify_release: Be specific about expected artifacts
Use a hard-coded list of artifacts that we expect to find in a
release. Specifically check that each of those files matches
the corresponding file in locally built release.

Also add two missing annotations.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2022-03-24 14:39:59 +02:00
dependabot[bot]
845441468e
build(deps): bump urllib3 from 1.26.8 to 1.26.9
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.8 to 1.26.9.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.9/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.8...1.26.9)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-24 10:09:26 +00:00
Jussi Kukkonen
65d6503e63 verify_release: Be explicit about PyPI version
We are interested in what pip thinks is the current tuf version: make
that explicit in method naming and comments.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2022-03-23 15:22:27 +02:00
Jussi Kukkonen
b7b035aea1
Merge pull request #1758 from ivanayov/updater_api_input_validation
Add tests for Updater input validation
2022-03-23 15:17:48 +02:00
Jussi Kukkonen
05c295987a
Merge pull request #1915 from MVrachev/test-statics-data-generation
Tests: provide a way to generate a simple metadata set
2022-03-23 15:14:46 +02:00
dependabot[bot]
02890d122e
build(deps): bump cryptography from 36.0.1 to 36.0.2
Bumps [cryptography](https://github.com/pyca/cryptography) from 36.0.1 to 36.0.2.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/36.0.1...36.0.2)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-23 12:55:30 +00:00
Lukas Pühringer
b272ac7734
Merge pull request #1918 from lukpueh/pin-direct-test-deps-only
build: pin direct test dependencies only
2022-03-23 13:54:47 +01:00
Lukas Puehringer
ec8a767c10 build: pin direct test dependencies
Fixes #1899
Reverts #1867

In #1867 we started pinning direct and transitive test
dependencies for stable test results, i.e. to not have an unnoticed
update of a used test tool (or their dependencies) break our tests.

This resulted in a dependabot updates inundating our PR tracker,
potentially obfuscating updates, which we care to address with
higher priority.

As a compromise we now only pin direct test dependencies, which
should still give us relatively stable test runs, while reducing
the spam.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2022-03-23 11:54:01 +01:00
Lukas Puehringer
1e9967b69a Revert "build: pin test requirements for deterministic CI"
This reverts commit 5643cecf68.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2022-03-23 11:51:37 +01:00
Martin Vrachev
384772efc3 Provide a way to generate a simple repository
I created a new script called "generate_md.py" which can be used
to easily generate a repository. Additionally, I created a new
test file making sure that the locally stored metadata files and
the newly generated metadata roles are the same.
This will allow us to test that we are not changing the metadata
file structure when making changes.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2022-03-22 18:13:54 +02:00
Martin Vrachev
69cc684230 gitattributes: make all JSON files end with LF
A really specific bug occurred on CI runs on all Windows machines
https://github.com/theupdateframework/python-tuf/runs/5467473050?check_suite_focus=true
where we weren't able to verify that what was generated is the same
as the stored on Git.

After research with Jussi, we found out that the problem comes not
from the content of the file that was generated, but because on Windows
Git proactively replaced all line endings for text files with CRLF symbol
("\r") this made the locally stored JSON files different from the one
generated.

We want to make sure such bugs doesn't occur again and that's why we
disable this behavior for all JSON files.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2022-03-22 16:42:00 +02:00
Jussi Kukkonen
53bacdf7e3 build: Add verify-release script
verify-release
* Builds a release from current commit
* Notifies if git describe does not match built version
* Notifies if built version is not the latest GitHub or PyPI version
* Asserts that the GitHub and PyPI release artifacts match the built
  release artifacts

This should be useful after release as any developer (or a CI job) can
easily verify that the release matches the sources in git.

Note that the last checks currently fail as the 1.0 build was not
reproducible. They should succeed after next release.

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2022-03-22 14:47:17 +02:00
Lukas Pühringer
ff770eacd9
Merge pull request #1896 from ofek/modernize-metadata
Update package metadata
2022-03-22 13:15:47 +01:00
Jussi Kukkonen
1d166f0b4e
Merge pull request #1876 from jku/more-details-on-verify-failure
Logging and error message improvements
2022-03-21 14:21:44 +02:00
Jussi Kukkonen
d9f2d9d24a
Merge pull request #1707 from ivanayov/test_expired_metadata
Test expired metadata from cache
2022-03-21 14:19:01 +02:00
Lukas Pühringer
f2e80a82cb
Merge pull request #1843 from ivanayov/metadata_docstrings_imprv_follow_up
Improve docstrings in Metadata API to be more descriptive
2022-03-21 09:31:01 +01:00
Ivana Atanasova
8d4d9af70b Update expired metadata tests logic
This change improves the logic of expired metadata tests, so that
it is explicitly visible what the expiry time and the versions are
and when update/refresh is called in that period

Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
2022-03-18 22:01:33 +02:00
Ivana Atanasova
d8d0486514 Fix expired metadata tests
This change fixes the expired metadata tests to mock `datetime`
as previously they mocked `time` incorrectly, which did not affect
update methods, as they use `datetime.datetime.utcnow()` to
calculate now

Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
2022-03-18 19:53:50 +02:00
Ivana Atanasova
cab99f58b6 Verify validation is performed from local metadata
This change verifies that when local metadata has expired, it is
still used to verify new metadata that's pulled from remote

Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
2022-03-18 19:53:50 +02:00
Ivana Atanasova
15c8d80b8a Test expired metadata from cache
This tests that an expired timestamp/snapshot/targets when loaded
from cache is not stored as final but is used to verify the new
timestamp

Fixes #1681

Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
2022-03-18 19:53:50 +02:00
Ivana Atanasova
e26363cf6a Add tests for Updater input validation
This test covers `targetinfo`, `target_path`, `target_base_url`,
`metadata_dir` and `filepath` input validation of the `Updater`
methods

Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
2022-03-18 18:59:05 +02:00
Ivana Atanasova
e71aa4a7d7 Improve Signer docstrings in Metadata API
Change to @lukpueh proposal with more clarification on why and how
the `securesystemslib.signer.Signer` interface is used

Co-authored-by: Lukas Pühringer <luk.puehringer@gmail.com>

Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
2022-03-18 18:32:25 +02:00
Ivana Atanasova
db7fbb21fa Improve docstrings in Metadata API to be more descritpive
This change updates some parts of the Metadata API docstrings
that did not give enough details and context

Fixes #1600

Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
2022-03-18 18:32:25 +02:00
Ofek Lev
98db711cca Update package metadata
Signed-off-by: Ofek Lev <ofekmeister@gmail.com>
2022-03-18 11:30:07 -04:00
Lukas Pühringer
9c8622d125
Merge pull request #1908 from MVrachev/update-spec-ver
Use spec version from tuf/api/metadata in examples
2022-03-17 16:37:59 +01:00
Martin Vrachev
06118843ca Use spec version from tuf/api/metadata in examples
Replace the hardcoded specification version with the one defined inside
tuf/api/metadata.py

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2022-03-17 15:41:05 +02:00
Jussi Kukkonen
8ac7167f8d
Merge pull request #1900 from joshuagl/joshuagl/build-nits
Minor, mostly packaging, clean-ups
2022-03-11 14:19:59 +02:00
Joshua Lock
150bfd0eec gitignore: fix directory patterns
Fix the directory ignore patterns to ignore the entire directories,
including child directories.
https://git-scm.com/docs/gitignore#_pattern_format

Co-authored-by: Ofek Lev <ofekmeister@gmail.com>
Signed-off-by: Joshua Lock <jlock@vmware.com>
2022-03-09 11:54:15 +00:00
Joshua Lock
22fee97dc3 setup: remove upper bound limit on python_requires
Setting upper bound version constraints in libraries is a source of
problems for users of those libraries, see:
https://iscinumpy.dev/post/bound-version-constraints/

The intent of the python-tuf version constraint is to ensure we're
using a version of Python which supports all the features we rely
on, this is a better fit for a lower limit.

Suggested-by: Ofek Lev <ofekmeister@gmail.com>
Signed-off-by: Joshua Lock <jlock@vmware.com>
2022-03-09 11:53:55 +00:00
Joshua Lock
430bdf5750 test: use tox isolated environments
Enable tox isolated environments to perform build operations in a virtual
environment.
See https://tox.wiki/en/latest/config.html#conf-isolated_build

Co-Authored-By: Ofek Lev <ofekmeister@gmail.com>
Signed-off-by: Joshua Lock <jlock@vmware.com>
2022-03-09 11:53:49 +00:00
Joshua Lock
0d2f6951a7 Remove redundant comment about version
The version is no longer duplicated in setup.cfg (since 5155ba74), so remove
redundant TODO suggesting folks update in two places.

Co-authored-by: Ofek Lev <ofekmeister@gmail.com>
Signed-off-by: Joshua Lock <jlock@vmware.com>
2022-03-09 11:53:13 +00:00
Jussi Kukkonen
e7037cf8c4
Merge pull request #1860 from MVrachev/serialization-bytes-array
Metadata test full serialization cycle
2022-03-07 11:14:31 +02:00
Jussi Kukkonen
29e4e63d46
Merge pull request #1895 from jku/single-source-version
Single source version number
2022-03-04 13:32:26 +02:00
Jussi Kukkonen
bf511ec0c6 docs: Update release docs
* version number is single sourced now
* Mention that using pip against test.pypi.org is unsafe
* Fix some filenames in the examples

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2022-03-03 16:04:08 +02:00