Add optional pre-commit configuration to install and run
auto-formatters when committing new code to tuf/api/*.
Auto-formatters include:
- trailing-whitespace
- end-of-file-fixer
- black
- isort
This commit also adds pre-commit to the dev dependencies
and updates the contributor instructions accordingly.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Configure lint build in tox.ini to check if code in tuf/api/* is
formatted according to black and isort style rules:
https://black.readthedocs.io/en/stable/the_black_code_style.htmlhttps://pycqa.github.io/isort/
In addition to our new style guide (#1128) and corresponding linter
configuration, requiring auto-formatting should help to further
reduce reviewing effort. The auto-formatter black was chosen for
the following reasons:
- It seems to be the most popular formatter in the Python ecosystem
- It is well documented including integration instructions with
most of the tools we use (git, GitHub Actions, pylint, a range of
editors, pyproject.toml #1161)
- It checks that the reformatted code produces a valid AST that is
equivalent to the original
- It has almost no ways of customization, which means no
customization effort required, and more (cross-project) style
uniformity, lowering contribution barriers
- It converts single to double quotes, where reasonable, which is
exactly what we recommend
- The style choices it makes seem generally reasonable and don't
conflict with our style guide, except for favoring hanging over
aligned indentation, which is the opposite of what we recommend.
But we are willing to update the adapt our style guide.
Auto-format pre-commit configuration will be added in a subsequent
commit.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
If you follow the instructions we provide for our contributors in
docs/CONTRIBUTORS.rst your sys.path (used to search for imports)
will put securesystemlib project directory first and tuf directory
second.
This creates a problem with imports from tuf modules because we can
import the wrong file or on relative imports (as currently we
use in the tests when we import utils), the imports cannot be resolved.
If we change the installation order, then tuf directory will be the
first in the import resolution path and those problems will be fixed.
PS: I want to express my gratitude towards Jussi who helped me find
this problem.
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
Replace mentions of travis/appveyor with GitHub Actions in
governance (contribution) and readme (badges) documents.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Use Google style guide with refinements, because the Google style
guide is a comprehensive, well-established style guide that is
mostly based on PEP-8 and was accepted by everyone on the TUF team.
There is no need to replicate these recommendations. However, we do
provide a very slim document with additional refinements, in order
to emphasize on items the we consider especially important, want to
be handled differently, or in one specific way, where the Google
guide would allow multiple.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Add MADR that justifies why we want to add custom classes for
complex tuf metadata attributes.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Describe pros of developing TUF 1.0.0 in a subdirectory
of the current implementation against the rest of the options.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Document the outcome of #1126 to develop TUF 1.0.0
in a subdirectory of the current TUF implementation.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Update badge URL in readme after migrating from travis-ci.org to
travis-ci.com, due to brownout on the former.
Migration was performed via Travis Web UI:
https://docs.travis-ci.com/user/migrate/open-source-repository-migration
NOTE: This is a quick fix to speed up Travis builds until we switch
to GitHub Actions (#1195)
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
secure-systems-lab/securesystemslib#288 changes the key generation
interface functions in such a way that it is clear if a call opens
a blocking prompt, or writes the key unencrypted. To do this two
functions are added per key type:
- `generate_and_write_*_keypair_with_prompt`
- `generate_and_write_unencrypted_*_keypair`
The default `generate_and_write_*_keypair` function now only allows
encrypted keys and only using a passed password. This respects the
principle of secure defaults and least surprise.
sslib#288 furthermore adds a protected
`_generate_and_write_*_keypair`, which is not exposed publicly
because it does not encrypt by default, but is more flexible and
thus convenient e.g. to consume all arguments from a key generation
command line tool such as 'repo.py'.
This commit adds the new public functions to the tuf namespace and
adopts their usage accordingly.
NOTE regarding repo.py:
This commit does not fix any problematic password behavior of
'repo.py' like default passwords, etc. (see #881). It only adopts
the sslib#288 changes to maintain the current behvior, plus
removing one glaringly obsolete password prompt.
NOTE regarding key import:
The securesystemslib private key import functions were also changed
to no longer auto-prompt for decryption passwords , TUF, however,
only exposes custom wrappers (see repository_lib) that do
auto-prompt. sslib#288 changes to the prompt texts are nevertheless
propagated to tuf and reflected in this commit.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Provide additional context to clarify where we expect Python 3.6+ to be used
exclusively (new modules) and link to other discussions around the future of
Python 2.7 supporting code.
Signed-off-by: Joshua Lock <jlock@vmware.com>
In order to make decisions about the code and the design explicit and easier
to reference in future we want to record significant architectural decisions.
This commit introduces docs/adr with a template Architectural Decision Record
and index using the [MADR](https://adr.github.io/madr/) format.
It also adds ADR 0000 to document the decisions to use MADR.
Fixes#1141
Signed-off-by: Joshua Lock <jlock@vmware.com>
* Strongly encourage the reader to use Python 3 for creating release
artefacts
* Recommend cleaning the tuf directory before creating the release
Signed-off-by: Joshua Lock <jlock@vmware.com>
Consistently enclose filenames referred to througout the release process in
backticks to ensure they are rendered in the code style.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Ensure someone following the release process pushes the tag they create
and include a tag message matching the common form, rather than requiring
somebody following the steps to figure out what to enter into their editor
Signed-off-by: Joshua Lock <jlock@vmware.com>
For some reason the first level 3 heading 'Added' was not rendering
correctly. The level 2 heading for 'Fixed' should be level 3.
Signed-off-by: Joshua Lock <jlock@vmware.com>
* Remove reference to deprecated settings
* Mention that the tutorial expects the dependencies and link to
instructions
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Lot of changes in 7 lines:
* PyCrypto is no longer an option: remove mention of it
* RSA-PSS wiki page now redirects to a fairly useless stub: replace it
with the RFC (it's not light reading but better than nothing)
* Mention ECDSA
* Remove mention of json for RSA keys: that does not seem to be true
Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
Improve the coding style in TUTORIAL in the case
where absolute path to a file is needed to perform file system
access and at the same time is rejected by Targets methods.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Replace the absolute paths returned by get_filepaths_in_directory()
in the tutorial with a hard-coded list of relaive filepaths since
add_target(s) and delegate() methods raise excception on absolute
paths.
Remove an obsolete warning about path pattern's location.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Remove link to incomplete and severely outdated
client_setup_and_repository_example.md in client section of
TUTORIAL.md.
Instead we should link (or move the entire client tutorial part) to
tuf/client/README.md, which is more comprehensive and less outdated
than above document (see #808).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
In tutorial code snippet fix a call to repository.mark_dirty that
passes a non-existing bin role name raising:
"tuf.exceptions.UnknownRoleError: Role name does not exist: 90-7"
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
colorama is now an optional dependency for securesystemslib, and so isn't
installed by default. However, the repo script uses colorama and doesn't
handle its absence - add securesystemslib[colors] to ensure colorama is
installed.
Signed-off-by: Joshua Lock <jlock@vmware.com>
* Use suffixed instead of prefixed sub-requirements files to group
them alphabetically in the file tree.
* Layer requirements files akin to the in-toto project
(see in-toto/in-toto#294). The hierarchy is:
- *requirements.in*
tuf runtime requirements, including optional requirements
(pynacl and cyrptography)
- *requirements-pinned.txt*
pinned tuf runtime requirements, including optional
and transitive (1 level deep) requirements and their hashes.
The file is generated semi-automatically using pip-compile
and a bash script (see document header), based off of
requirements.in, combining requirements from all supported
Python versions.
This file should be auto-updated, by e.g. dependabot, and be used
for ci/cd tests, to catch issues with new dependencies.
- *requirements-test.txt*
additional test runtime requirements
- *requirements-tox.txt*
combines requirements.txt, requirements-test.txt and additional
test tools (for linting and coverage), i.e. everything that is
needed in each tox environment to run the tests.
- *requirements-dev.txt*
lists tox for local development and testing, and also
requirements-tox.txt and tuf in editable mode to run
the test suite or individual tests directly.
- *requirements.txt*
requirements-pinned.txt with the hashes of the dependencies
as reported by pip at the time of creating the file.
NOTE: this is not used for testing or dev-install because pip
doesn't allow mixed (with and without hashes) installations.
This file should also be auto-updated, by e.g. dependabot.
* Removes an obsolete version constraint on coverage
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
TUF does not reliably mark roles as dirty whose metadata needs
to be re-generated.
Only roles that have changed are marked as dirty, but sometimes
roles metadata needs to be updated, although the role wasn't
changed directly (see #958).
Furthermore, the tutorial assumes at one point that the reader
leaves and re-enter the interpreter session, being forced to reload the
signing keys, roles that later need to be re-written, are marked as
dirty. If the reader does not leave the interpreter, the roles are
not marked as dirty (see #964).
To not confuse the reader with flawed state-keeping, and to never
write an inconsistent repository to disk, the tutorial lets the
reader explicitly mark all roles that need to be re-written as
"dirty".
This can be changed once above issues are fixed.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Explain and show output of delegate_hashed_bins() function call in
tutorial snippet.
Also update the subsequent comment for better continuity.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Ask the reader to ignore a misleading warning about the location of
a delegation path pattern.
The comment may be removed when fixing the warning in
theupdateframework/tuf#963.
- Comment out text that has become obsolete when commenting out
the "Revoke Delegated Role" section (in an earlier commit).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The text above the snippet explains the basic idea of "consistent
snapshots" and how to generate them with `write` and `writeall`.
The commands in the snippet just leave the repo in an inconsistent
state (see comment).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Fix expected output
- Update comments
- Add a few additional calls, to help the reader understand the
repo state
- Also see theupdateframework/tuf#958
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
In "Targets" section:
- Remove `repository.targets('<delegated rolename>').add_target(...)`
command, because it is not copy-pastable and delegations
have not yet been covered at that point.
- Update the "remove targets" snippet to remove the previously added
"myproject/file4.txt" instead of "file3.txt", because we will add
"myproject/file4.txt" to the delegated "unclaimed" role in the
Delegation section.
In "Delegation" section:
- Change "unclaimed" delegation pattern from 'foo*.tgz', for which
no file exists in the tutorial, to 'myproject/*.txt'.
- Add "myproject/file4.txt" to the delegated unclaimed targets role
- Remove the command that updates the version of the "unclaimed"
role, because this should not be done manually, and the add_target
call shows just as well how to access delegated roles.
- Comment out the revoke delgated role section, leaving a TODO note
for required updates (should be ticketized).
In "Delegate to Hashed Bins":
- Add call to remove target "myproject/file4.txt" from "unclaimed",
because it is further delgated to hashed bins
- Add dirty_roles() call to show all the newly created bins
- Add mark_dirty() and writeall() calls to create a consistent
state of the repo
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
The section does not fit in the continuity of the tutorial and
misses commands to make it fully copy-pastable.
This commit marks the section as "Excursion" and updates the
snippets to make the commands fully copy-pastable, using files
created in prior tutorial sections.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
- Correctly show that repo.get_filepaths_in_directory() returns
absolute and not relative paths
- Pass absolute path to repo.targets.add_target() to fix exception
- Also see theupdateframework/tuf#957
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Reasons are:
- The prompt says 2.7.3 Sep 26 2013, which makes the tutorial look
outdated
- There is another section that explains how snippets should be
executed in a Python interactive interpreter
- The only activity in the snippet is importing tuf functions and
creating a repo, both of which is done in another snippet below.
And the here created repo is not re-used
- The tutorial is long enough
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Because leading '/' is no longer allowed in target filenames in
target addition or delegation.
See https://github.com/theupdateframework/tuf/issues/639
While we're at it, remove some other unnecessary '/' characters
in repository creation and loading in the tutorial and tutorial
test.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
add_restricted_paths was renamed to add_path; however, this
function represents a problematic element of TUF that assumes
that roles are have a single delegator and delegatee, and that
one can refer to a role's expected keys without being concerned
about any delegation metadata....
So this is being removed from the tutorial. In time, add_paths
will either be removed or changed (to expect a delegator role
and a delegatee role, not just a delegatee role).
This comment does not do justice to the issue: please see TUF
GitHub Issue #660:
https://github.com/theupdateframework/tuf/issues/660
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
While it may be nice, the use of a keystore/ directory for the
temporary keys created in the tutorial complicates the code a
good bit when it's done in a portable way (tons of os.path.join()
calls), and that's not worth it. It also is a slight complication
in a tutorial that profits from being as simple as possible.
Tests will be run in multiple environments (including non-Linux
environments) and to leave so many extra subdirectory uses in the
tutorial means that the tutorial test will deviate over a large
number of lines from the TUTORIAL.md content it is intended to
test, which would be bad.
This commit adjusts both the tutorial doc and the regression test
for the tutorial.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
- line-wraps are integrated with
190a736d297ef5d8d3c1e8761ac196666954f3a2 in #775
- sorting the value returned by `get_dirty_roles()` in
`dirty_roles()` is not necessary as `get_dirty_roles()` already
returns a sorted list per ac010337f0e154f9c183f8abe759ce26ed16731c
in #775.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
An attempt to make part of the tutorial less jarring, and reformatting
of the surrounding text to keep the lines <80 characters.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Since https://github.com/theupdateframework/tuf/pull/781 we
only provide limited protection against slow retrieval attacks.
So far this has only been discussed in above issue and hinted at
by a disabled test and a code comment in that test.
This change adds a corresponding disclaimer to a more prominent
place, i.e. the list of attacks in SECURITY.md.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Co-Authored-By: Trishank K Kuppusamy <33133073+trishankatdatadog@users.noreply.github.com>
Add hint to install `coverage` before using it.
This should be installed via dev-requirements.txt, however it
does not seem to fit in there, because dev-requirements.txt pins
all its dependencies which does not seem to make sense for a
development tool.
Maybe a hierarchy of requirements.txt similar to
06a28987dc
could be established.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Add a tox build that runs tests against securesystemslib's tip of
development, i.e. master branch, to ease preparation of tuf for a
new securesystmeslib release.
The tox build is run on travis but is allowed to fail.
This commit also fleshes out the testing section of the
contribution documentation.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
and note that if errors occur while installing the optional
crypto dependencies, that the non-Python dependencies section
should be consulted.
Also adds a git clone step to the development installation
instructions in the contributors' instructions, and other
minor installation notes for contributors.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
- correctly frame the CLI's current state as a tutorial toy.
- provide a friendlier quickstart that puts what it's doing into
perspective and guides you to next steps.
- provide a better sense of what each tutorial/quickstart doc is
for.
- make the getting started page slightly more friendly.
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
Add note to make clear what was implicit: that once a delegated role is created,
it can be accessed (for the same purposes as you'd access the Targets role
using `repository.targets`) as: `repository.targets("<delegated rolename>")`.
Also add that specific note to the add_targets section of the tutorial.
Note that the `repo.py --init` call also sets up a client directory.
A student recently ran into some confusion on this point, and it's not properly documented here.
- refer to the authoritative sources for the latest releases
- include new packager information
- add tasks for upcoming releases
- generalize release schedule
Signed-off-by: Sebastien Awwad <sebastien.awwad@gmail.com>
Hi, I did the quick-start today and missed some context. This is my suggestion.
Thanks for a cool framework!
Regards Håvard
Signed-off-by: Håvard B Line <havard.line@gmail.com>
Some users will actually attempt to execute the example listed in the overview section. Use a working example that doesn't raise an error.
Signed-off-by: Vladimir Diaz <vladimir.v.diaz@gmail.com>
* The client script is now named 'client.py' and the user is no longer required to manually copy the basic_client.py to the CWD
* A log file is not saved to the CWD, by default
* The client.py script doesn't download all target files available on the repo.
Signed-off-by: Vladimir Diaz <vladimir.v.diaz@gmail.com>
Correct links to modules and documentation.
Broken links appear to be mostly fallout from moving
documentation into the docs/ directory, as relative paths
to python source modules were broken.
