* The conformance test suite is likely to still change quite a bit so
the workflow is not enabled on PRs yet
* The actual conformance client is copied from the tuf-conformance project
* This is mostly a test to see how things should work out, and a
demonstration of how the tuf-conformance project should be used
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
* Remove bandit
* Add ruff ruleset "flake8-bandit"
* verify_release is now checked by bandit
* Avoid some asserts as suggested
* ignore a subprocess.run lint: it seems dumb
* ignore all bandit rules for tests and examples (just like before)
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
- adpot changes in dependabot.yml and remove --diff from ruff check.
- select pydocstyle, isort, pyflakes, pep8-naming, pycodestyle for ruff and ignore some small issues / add inline comments.
- adjust docstring length to 80 in various files
Signed-off-by: E3E <ntanzill@purdue.edu>
We already have 6 files and I'm planning to add another one: maybe it's
time to move these out of the top level directory.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
If the sslib release version matches, pip does not install the version from git
because the same version is already installed. Force the install.
Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Enable tox to lint 'verify_release' script and fix:
- whitespace
- unused import (we only import here to see if the module is
available for use in a subprocess)
- unfound import (same as unused import)
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Enable tox isolated environments to perform build operations in a virtual
environment.
See https://tox.wiki/en/latest/config.html#conf-isolated_build
Co-Authored-By: Ofek Lev <ofekmeister@gmail.com>
Signed-off-by: Joshua Lock <jlock@vmware.com>
Configures tox to use a pinned requirements file for deterministic
CI builds, i.e. our CI shouldn't start failing because of an
incompatible upstream release of any of our testing tools:
NOTE: pinned tuf runtime requirements were already were already
used for test builds before (included via `-r
requirements-pinned.txt` in 'requirements-test.txt'). Now they are
explicitly listed in 'requirements-test-pinnned.txt'.
'requirements-test-pinnned.txt' was generated semi-automatically by
running pip-compile over 'requirements-test.txt' for each
supported/tested Python version (see snippet below) and manually
merging the resulting per-Python version requirements files into
one, adding environment markers as needed.
```
for ver in 3.7.12 3.8.12 3.9.9 3.10.0; do
pyenv virtualenv ${ver} tuf-env-${ver}
pyenv activate tuf-env-${ver}
python3 -m pip install -U pip pip-tools
pip-compile --no-header --annotation-style line \
-o requirements-test-pinned-${ver}.txt \
requirements-test.txt
pyenv deactivate
pyenv uninstall -f tuf-env-${ver}
done
```
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Update linter config to no longer distinguish between legacy
and new implementation. This requires addressing a linter warning
in an until now not linted module (tuf/__init__.py).
- Remove obsolete rules in MANIFEST.in (source distribution) and
tests/.coveragerc (test coverage).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Stop linting tuf/exceptions.py with mypy as we are going to use
tuf/api/exceptions.py for exceptions in the new code.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Simplified ``tox`` environ list not to expect that developer has
all multiple Python version, but instead run with the python version
available.
Also, it adds docs build to environ list.
Running ``tox`` will run the lint, docs, and py.
The CI covers the multiple supported Python versions, and the
developers still can use tox -e py{version}
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
This commit adds to the docs tox session the flag ``-W``, which
turns the warnings into errors.
The CI will fail once it gets errors.
Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Exclude regexs/globs are needed to exclude the test files testing
the old code.
After we remove those files we will be able to remove the exclude
regex/globs.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Instead of providing a target directory for linting by each of the
tools use one variable which will be the source of truth about which
directories do we lint.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
The examples folder currently contains a repository example and it's
good if we start linting its content and as a result add type
annotations.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We are using 4 linters: black, isort, pylint and mypy.
It's good if we use one file as a source for truth for all linter
configurations.
I tried multiple ways to use the src_path option,
so we can just call isort without pointing out the target folders, but I was not
successful.
I tried running isort with "isort --settings-path=pyproject.toml"
I got the error:
"Error: arguments passed in without any paths or content."
Additionally, I saw one project with source configuration https://github.com/Pylons/pyramid/blob/master/pyproject.toml,
but they had to give explicit folders too 8061fce297/tox.ini (L26)
and 8061fce297/tox.ini (L66)
It was a similar situation with "check" and "diff".
In the documentation it's said that for both check and diff are not
supported in configuration files.
See:
- https://pycqa.github.io/isort/docs/configuration/options.html#check
- https://pycqa.github.io/isort/docs/configuration/options.html#show-diff
Additionally, in two issues it was confirmed that in integration tests
we should use --check and --diff the way we did until now.
As a result, I moved part of the configuration options for isort inside
pyproject.toml without the actual directories that need to be linted
and "check" and "diff" options.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
We are using 4 linters: black, isort, pylint and mypy.
It's good if we use one file as a source for truth for all linter
configurations.
As a first step move black options in pyproject.toml.
I tried multiple ways to use the include option,
so we can just call black --config=pyproject.toml, but I was not
successful. Then I found this comment https://github.com/psf/black/issues/861#issuecomment-680411125
explaining that the path argument is mandatory.
As a result, I will move all configuration options for black inside
pyproject.toml without the actual directories that need to be linted.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Python 3.10 is released on October 4-th 2021 and it seems
logical to add support for it as it doesn't require any major effort
from the project.
For reference read:
https://www.python.org/downloads/release/python-3100/
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Add tests covering missing branches of the Updater
code. Inlcude ngclient in the total coverage report.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
pip nowadays recognizes that we are asking for two different versions of
securesystemslib in the "with-sslib-master" env, and errors out.
Instead install normal dependencies first, then install the new
securesystemslib separately (this ends up upgrading securesystemslib).
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This allows using existing documentation in the published documentation
without
* moving the existing docs (which would break external links)
* tricks like symlinks that create issues with relative links
Put the api reference files into a subdirectory to avoid polluting the
main docs/ directory.
Include "Installation" and "Instructions for Contributors" in the
published documentation.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
- New 'docs' environment in tox enables
building the sphinx documentation in isolation.
- New requirements-docs.txt.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Updated/removed documented commands and comments which were referencing Python2. Also updated links to documentation referencing Python2 docs (unchanged where needed)
Signed-off-by: Samuel Gregorovic <samuelgregorovic@gmail.com>
Signed-off-by: samuelgregorovic <samuelgregorovic@gmail.com>
pylint on the legacy code is by far the slowest part of linting (to
the extent that parallelizing the tox env itself doesn't really help):
pylint can fortunately parallelize itself.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
This is an initial setup: By default check only tuf/api/,
and ignore securesystemslib imports.
Change lint working directory to source root: This saves repeating a lot
of {toxinidir} in the command lines.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Keep using newest versions of build tools, but don't use pre-releases:
* less downloading (with chances of failures) happens in the CI because
versions change less often
* tools are less likely to break the build (or at least it happens less
often)
This change should not affect the software under test as we install
pinned versions of those.
Note that this also doesn't affect black: tox still ends up with a
pre-release version of black because that project has never made an
actual release.
Fixes#1350
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Runing module from python will help
portability between systems.
Those scripts may be renamed by distros,
for instance Debian provides python3-converage.
Signed-off-by: Philippe Coval <rzr@users.sf.net>
Configure lint build in tox.ini to check if code in tuf/api/* is
formatted according to black and isort style rules:
https://black.readthedocs.io/en/stable/the_black_code_style.htmlhttps://pycqa.github.io/isort/
In addition to our new style guide (#1128) and corresponding linter
configuration, requiring auto-formatting should help to further
reduce reviewing effort. The auto-formatter black was chosen for
the following reasons:
- It seems to be the most popular formatter in the Python ecosystem
- It is well documented including integration instructions with
most of the tools we use (git, GitHub Actions, pylint, a range of
editors, pyproject.toml #1161)
- It checks that the reformatted code produces a valid AST that is
equivalent to the original
- It has almost no ways of customization, which means no
customization effort required, and more (cross-project) style
uniformity, lowering contribution barriers
- It converts single to double quotes, where reasonable, which is
exactly what we recommend
- The style choices it makes seem generally reasonable and don't
conflict with our style guide, except for favoring hanging over
aligned indentation, which is the opposite of what we recommend.
But we are willing to update the adapt our style guide.
Auto-format pre-commit configuration will be added in a subsequent
commit.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add sub-package with 3 abstract base classes to:
- serialize Metadata objects to bytes (transport)
- deserialize Metadata objects from bytes (transport)
- serialize Signed objects to bytes (signatures)
pylint notes:
- configure tox to use api/pylintrc
- configure api/pylintrc to allow classes without public methods
(default was 2)
Design considerations
---------------------
- Why not implement de/serialization on metadata classes?
-> See ADR0006.
- Why use separate classes for serialization and deserialization?
-> Some users might only need either one, e.g. client only needs
Deserializer. Maybe there are use cases where different
implementations are used to serialize and deserialize.
- Why use separate classes for Metadata- and Signed-Serialization?
-> They require different concrete types, i.e. Metadata and
Signed as parameters, and using these specific types seems to
make the interface stronger.
- Why are de/serialize methods not class/staticmethods?
-> In reality we only use classes to namespace and define a
type annotated interface, thus it would be enough to make the
methods classmethods. However, to keep the de/serialize
interface minimal, we move any custom format configuration to
the constructor. (See e.g. "compact" for JSONSerializer in
subsequent commit).
Naming considerations
---------------------
- Why de/serialize?
-> Implies byte stream as input or output to the function, which
is what our interface needs.
- Why not marshaling?
-> Synonym for serialize but implies transport, would be okay.
- Why not encoding?
-> Too abstract and too many connotations (character, a/v).
- Why not parse?
-> Too abstract and no good opposite terms (unparse, write,
dump?)
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Without this new pip will notice the conflict between
securesystemslib (git master)
securesystemslib[crypto,pynacl] >= 0.18
The former does not have the extras as it should.
Add the extras and also do not accept plain http for anything.
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Remove references to, and handling of, Python 2.7 in our project scaffolding:
- updated python_requires in setup.py to state our intent to support
Python 3.6 and above (but not Python 4, yet)
- Drop no longer required dependencies in setup.py, and requirements-*.txt
(further refinement of requirements files will be handled in #1161)
- Remove Python 2.7 from our tox environments
Signed-off-by: Joshua Lock <jlock@vmware.com>
