Python 3.10 is released on October 4-th 2021 and it seems
logical to add support for it as it doesn't require any major effort
from the project.
For reference read:
https://www.python.org/downloads/release/python-3100/
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add tests covering missing branches of the Updater
code. Inlcude ngclient in the total coverage report.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
pip nowadays recognizes that we are asking for two different versions of
securesystemslib in the "with-sslib-master" env, and errors out.
Instead install normal dependencies first, then install the new
securesystemslib separately (this ends up upgrading securesystemslib).
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This allows using existing documentation in the published documentation
without
* moving the existing docs (which would break external links)
* tricks like symlinks that create issues with relative links
Put the api reference files into a subdirectory to avoid polluting the
main docs/ directory.
Include "Installation" and "Instructions for Contributors" in the
published documentation.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
- New 'docs' environment in tox enables
building the sphinx documentation in isolation.
- New requirements-docs.txt.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Updated/removed documented commands and comments which were referencing Python2. Also updated links to documentation referencing Python2 docs (unchanged where needed)
Signed-off-by: Samuel Gregorovic <samuelgregorovic@gmail.com>
Signed-off-by: samuelgregorovic <samuelgregorovic@gmail.com>
pylint on the legacy code is by far the slowest part of linting (to
the extent that parallelizing the tox env itself doesn't really help):
pylint can fortunately parallelize itself.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This is an initial setup: By default check only tuf/api/,
and ignore securesystemslib imports.
Change lint working directory to source root: This saves repeating a lot
of {toxinidir} in the command lines.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Keep using newest versions of build tools, but don't use pre-releases:
* less downloading (with chances of failures) happens in the CI because
versions change less often
* tools are less likely to break the build (or at least it happens less
often)
This change should not affect the software under test as we install
pinned versions of those.
Note that this also doesn't affect black: tox still ends up with a
pre-release version of black because that project has never made an
actual release.
Fixes#1350
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Runing module from python will help
portability between systems.
Those scripts may be renamed by distros,
for instance Debian provides python3-converage.
Signed-off-by: Philippe Coval <rzr@users.sf.net>
Configure lint build in tox.ini to check if code in tuf/api/* is
formatted according to black and isort style rules:
https://black.readthedocs.io/en/stable/the_black_code_style.htmlhttps://pycqa.github.io/isort/
In addition to our new style guide (#1128) and corresponding linter
configuration, requiring auto-formatting should help to further
reduce reviewing effort. The auto-formatter black was chosen for
the following reasons:
- It seems to be the most popular formatter in the Python ecosystem
- It is well documented including integration instructions with
most of the tools we use (git, GitHub Actions, pylint, a range of
editors, pyproject.toml #1161)
- It checks that the reformatted code produces a valid AST that is
equivalent to the original
- It has almost no ways of customization, which means no
customization effort required, and more (cross-project) style
uniformity, lowering contribution barriers
- It converts single to double quotes, where reasonable, which is
exactly what we recommend
- The style choices it makes seem generally reasonable and don't
conflict with our style guide, except for favoring hanging over
aligned indentation, which is the opposite of what we recommend.
But we are willing to update the adapt our style guide.
Auto-format pre-commit configuration will be added in a subsequent
commit.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add sub-package with 3 abstract base classes to:
- serialize Metadata objects to bytes (transport)
- deserialize Metadata objects from bytes (transport)
- serialize Signed objects to bytes (signatures)
pylint notes:
- configure tox to use api/pylintrc
- configure api/pylintrc to allow classes without public methods
(default was 2)
Design considerations
---------------------
- Why not implement de/serialization on metadata classes?
-> See ADR0006.
- Why use separate classes for serialization and deserialization?
-> Some users might only need either one, e.g. client only needs
Deserializer. Maybe there are use cases where different
implementations are used to serialize and deserialize.
- Why use separate classes for Metadata- and Signed-Serialization?
-> They require different concrete types, i.e. Metadata and
Signed as parameters, and using these specific types seems to
make the interface stronger.
- Why are de/serialize methods not class/staticmethods?
-> In reality we only use classes to namespace and define a
type annotated interface, thus it would be enough to make the
methods classmethods. However, to keep the de/serialize
interface minimal, we move any custom format configuration to
the constructor. (See e.g. "compact" for JSONSerializer in
subsequent commit).
Naming considerations
---------------------
- Why de/serialize?
-> Implies byte stream as input or output to the function, which
is what our interface needs.
- Why not marshaling?
-> Synonym for serialize but implies transport, would be okay.
- Why not encoding?
-> Too abstract and too many connotations (character, a/v).
- Why not parse?
-> Too abstract and no good opposite terms (unparse, write,
dump?)
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Without this new pip will notice the conflict between
securesystemslib (git master)
securesystemslib[crypto,pynacl] >= 0.18
The former does not have the extras as it should.
Add the extras and also do not accept plain http for anything.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Remove references to, and handling of, Python 2.7 in our project scaffolding:
- updated python_requires in setup.py to state our intent to support
Python 3.6 and above (but not Python 4, yet)
- Drop no longer required dependencies in setup.py, and requirements-*.txt
(further refinement of requirements files will be handled in #1161)
- Remove Python 2.7 from our tox environments
Signed-off-by: Joshua Lock <jlock@vmware.com>
Prior to this commit our GitHub workflow would set up one Python
version only for each build, which means that the commands to run
the tests and publish coverage (tox and coveralls) were run with
the same Python version as tox runs the tests in.
Given that the coveralls CLI tool dropped py2 a couple of releases
ago, this commit sets up an additional service py3 to run coveralls
(and tox) on when building for py2.
To prevent tox from using the wrong Python version to run the tests
on, this commit changes the toxenv value from the generic 'py'
(uses default python on path) to 'py27'.
For convenience and readability we use the environment variable
TOXENV instead of the tox -e option.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Configure tox to use legacy resolver as a temporary workaround
for pypa/pip#9215, which results in huge unnecessary downloads.
Co-authored-by: Jussi Kukkonen <jkukkonen@vmware.com>
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Installing securesystemslib in editable mode leads to a problem
in sys.path where we have two "tests" packages.
By not installing securesystemslib in an editable mode we are not
adding the securesystemslib tests to sys.path.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Python 3.9 is released on October 5-th 2020 and it seems
logical to add support for it.
For reference read:
https://docs.python.org/3/whatsnew/3.9.html
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Commit eb00d14 modified requirements-pinned.txt so that sslib specifiers
are now "[crypto,pynacl]". This happens to match the exact specifiers
used for the sslib git master dependency in tox.ini. This triggers pip
to say:
ERROR: Double requirement given: securesystemslib[crypto,pynacl]==0.16.0
(from -r /home/jku/src/tuf/requirements-pinned.txt (line 12)) (already
in securesystemslib[crypto,pynacl] from
git+http://github.com/secure-systems-lab/securesystemslib.git@master#egg=securesystemslib[crypto,pynacl],
name='securesystemslib')
Avoid this by not setting any specifiers for the sslib git master
dependency in tox.ini: This makes pip happy and we get the git master
version installed. pynacl and crypto are still installed because they
are in requirements-pinned.txt.
Fixes#1184.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
We don't need to lint the code with every version of Python, instead add
an extra tox env which lints once with the latest supported Python version
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add a minimal pylintrc to lint for new code being developed in tuf/api and
update the tox configuration to ignore tuf/api with the default pylintrc
and run an extra invocation of pylint for just the modules in tuf/api.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Follows up on #978, which had the following problems:
- too many requirements files (cc @trishankatdatadog ;)
- used extra tooling around pip-compile that
- didn't take into account requirement markers (see comments
in requirements.txt in this commit), and
- confused Dependabot, which expects the hashed requirements
file in a certain format, as pip-compile would generate it
without custom tooling (see #979).
This commit restructures the requirements files as follows:
- Merges requirements-tox.txt and requirements-test.txt. The
separation was semantically correct but operationally irrelevant.
- Removes the hashed requirements file, which doesn't add much
security, especially with PEP 458 on the way (see python/peps#1306),
but extra maintenance (see notes about requirements.txt in #978
and about Dependabot above)
- Manually adds environment markers to requirements-pinned.txt (see
comments in requirements.txt in this commit).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
* Use suffixed instead of prefixed sub-requirements files to group
them alphabetically in the file tree.
* Layer requirements files akin to the in-toto project
(see in-toto/in-toto#294). The hierarchy is:
- *requirements.in*
tuf runtime requirements, including optional requirements
(pynacl and cyrptography)
- *requirements-pinned.txt*
pinned tuf runtime requirements, including optional
and transitive (1 level deep) requirements and their hashes.
The file is generated semi-automatically using pip-compile
and a bash script (see document header), based off of
requirements.in, combining requirements from all supported
Python versions.
This file should be auto-updated, by e.g. dependabot, and be used
for ci/cd tests, to catch issues with new dependencies.
- *requirements-test.txt*
additional test runtime requirements
- *requirements-tox.txt*
combines requirements.txt, requirements-test.txt and additional
test tools (for linting and coverage), i.e. everything that is
needed in each tox environment to run the tests.
- *requirements-dev.txt*
lists tox for local development and testing, and also
requirements-tox.txt and tuf in editable mode to run
the test suite or individual tests directly.
- *requirements.txt*
requirements-pinned.txt with the hashes of the dependencies
as reported by pip at the time of creating the file.
NOTE: this is not used for testing or dev-install because pip
doesn't allow mixed (with and without hashes) installations.
This file should also be auto-updated, by e.g. dependabot.
* Removes an obsolete version constraint on coverage
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Drop 3.4 (due to end-of-life) and add 3.7 and 3.8 to tox, travis
and appveyor configuration for automated testing.
- Adapt classifiers in setup.py accordingly.
- Add python_requires field in setup.py to reflect supported
versions. This will prevent pip from trying to install tuf on
a non-supported version.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add a tox build that runs tests against securesystemslib's tip of
development, i.e. master branch, to ease preparation of tuf for a
new securesystmeslib release.
The tox build is run on travis but is allowed to fail.
This commit also fleshes out the testing section of the
contribution documentation.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This commit partially reverts the workaround introduced by
157167e0cc. Instead of patching the
path, we configure tox to install TUF in editable mode, which makes
the tests run against the same files as if the path were patched.
This makes it so that coverage records paths that it can then
normalize when sending them to coveralls.io (see .travis.yml).
See af22701140
for detailed background information.
As a consequence we can now skip building of sdist and installing it
into a virtual env directory in tox.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
This replicates behavior of unittest's `discover` method, and
allows `coverage` and the tool that posts coverage reports to
coveralls.io, i.e. `coveralls`, to record the correct paths and
left-strip the parts leading to the project directory.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Add a build matrix to run each tox env in a corresponding
travis env as per travis/tox best practices.
https://docs.travis-ci.com/user/languages/python/#using-tox-as-the-build-script
- Add Python 3.5 tests
- Remove only build on certain branch restrictions
- Use "install" instead of "before_script" to install dependencies.
Explicitly listing "install" prevents Travis from automatically
running `pip install -r requirements.txt`, which is not necessary
because most of those requirements are installed again in each
tox environment.
- Move pylint and bandit calls to tox (pylint requires
dependencies) to be installed.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
