Commit graph

4658 commits

Author SHA1 Message Date
Jussi Kukkonen
942e6d25ab
Merge pull request #1691 from MVrachev/key-rotations
Tests: add tests for non-root key rotations
2021-12-08 11:07:59 +02:00
Jussi Kukkonen
9ff5bfcff8
Merge pull request #1695 from ivanayov/repository_simulator_inconsistent_metafile_hashes
Fix inconsistent behaviour of compute_metafile_hashes_length
2021-12-07 14:08:22 +02:00
lukpueh
aebd8b0faf
Merge pull request #1708 from MVrachev/prepare-for-lint
Lint additional four files in the tests folder
2021-12-07 11:03:04 +01:00
Martin Vrachev
d748bd3ed9 Apply black on tests/aggregate_tests.py
All changes are automatically generated by black.
The other linters didn't find any errors in the file.
It's expected that most if not all code in aggregate_tests.py will be
useful even when we remove the tests file on the old code.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-07 11:51:23 +02:00
Martin Vrachev
028b7592a0 Apply linters on tests/simple_server.py
Apply all 4 linters on tests/simple_server.py, so we can lint it in the
future and not rename and exclude it.
As we are going to use part or most of tests/utils.py after we remove
the test files testing the old code, then we would need to keep the
simple_server.py file. It's currently used in tests/updater_ng.py
testing the new updater implementation.

Black and isort changes where automatically made.
The only manual changes are:
- pylint disable once in can_connect
- add type annotations
- simplifications around setting up the handler variable
- function docstrings

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-07 11:51:23 +02:00
Jussi Kukkonen
2db0cd6303
Merge pull request #1675 from kairoaraujo/issue#1518/python-client-example-tutorial
TUF Python Client Example/Tutorial
2021-12-07 11:07:53 +02:00
Martin Vrachev
580fd4d2a8 Apply linters on tests/test_utils.py
Apply all 4 linters on tests/test_utils.py, so we can lint it in the
future and not rename and exclude it.
As we are going to use part or most of tests/utils.py after we remove
the test files testing the old code, then we would need to keep the
test file testing utils.py - test_utils.py

Black and isort changes where automatically made.
The only manual changes are:
- pylint disable once in can_connect
- remove TestServerProcess.tearDown and instead clean it manually in the
tests, so we don't have to define server_process_handler as a class
variable and assert it's type in every test
- move can_connect outside TestServerProcess as it doesn't use self
anymore
- add type annotations
- function docstrings

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-03 23:04:33 +02:00
Kairo de Araujo
134441019c This commit fixes the verbosity (-v) levels.
without -v is ERROR logging, -v  WARNINGS logging, -vv INFO logging
and -vvv (+) DEBUG logging.

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:59:04 +01:00
Kairo de Araujo
183223e59c Simplify README, fix some flows and output
- Simplify README and better keywords
- Fix the verbosity
- Better docstrings
- Client flow for init and main are clear

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:59:01 +01:00
Kairo de Araujo
ca13d059b6 Single error handling for updater
To simple exemplify, the updater calls are added to a single block
that handles the exceptions

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:58:42 +01:00
Kairo de Araujo
4c37f1bc35 Removes the Python Client Example as WIP
This commit adds many comments from the WIP stage

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:58:42 +01:00
Kairo de Araujo
4c5581a800 Add a simple example handling TUF exceptions.
To highlight TUF ngclient exceptions.

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:58:42 +01:00
Kairo de Araujo
46629c4547 Implement a combination of print and logging
Using ``print`` for high-level client output
Option to see ``tuf.ngclient`` logging output

Update the README.

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:58:42 +01:00
Kairo de Araujo
9b4fcdb564 Added lint, rst to md, output
- Added the lint to examples
- README format moved from Restructuredtext to Markdown
- Removed the [INFO] and [ERROR] from output, to avoid confundint with
  logging structure

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:58:33 +01:00
Kairo de Araujo
ffe2c11fcf WIP: TUF Python Client Example/Tutorial
It is a simple example of TUF ngclient implementation.

This example contains a README.rst that is a tutorial/how-to-use
this simple client using static test data from TUF repository.

The code aims to be straightforward implementation, using basic
concepts from Python and Command Line Interface.

This is part of theupdateframework#1518

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-12-03 19:57:53 +01:00
Ivana Atanasova
71a7f11bd4 Fix inconsistent behaviour of compute_metafile_hashes_length
Previously when `compute_metafile_hashes_length` was set to `False`
`update_timestamp` did not set the hash and length values to `None`
as expected. This change fixes that, so they are not `None` when
`compute_metafile_hashes_length=True` and `None` when
`compute_metafile_hashes_length=False`

Signed-off-by: Ivana Atanasova <iyovcheva@vmware.com>
2021-12-03 18:18:58 +02:00
Martin Vrachev
d203834e5c Apply linters on tests/utils.py
Apply all 4 linters on tests/utils.py, so we can lint it in future and
not rename and exclude it.
It's expected that most if not all code in tests/utils.py will be
useful even when we remove the tests file on the old code.
Keep in mind when reviewing that type annotations where already added
in e2deff3148

Black and isort changes where automatically made.
The only manual changes are:
- pylint disables
- function docstrings
- initializations of attributes of "TestServerProcess" __init__()
- additional asserts of the attributes types

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-03 18:17:33 +02:00
lukpueh
a93f618839
Merge pull request #1701 from MVrachev/lint-examples
Start linting the examples folder
2021-12-03 10:31:38 +01:00
Martin Vrachev
3def667844 Tests decorator: save case name
Save the case name in the "unittest.TestCase" object when executing
test cases through the "run_sub_tests_with_dataset" decorator.
This allows dumping directories with names specific to the test case
that can be used for debugging as showed in
test_updater_key_rotations.py

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 15:40:30 +02:00
Martin Vrachev
8daf60d011 Tests: add tests for non-root key rotations
The amount of non-root key rotations is lower than the one for root as
the repository stores all root versions in order to be able to do
rollback check compared to timestamp, snapshot, and targets where it
stores only one version.

All of the root roles are needed in those tests again as root
delegates the keys for each of the other three top-level metadata roles.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 15:40:29 +02:00
Martin Vrachev
0235337119 test key-rotations: setup & teardown to class func
Make setup and teardown classmethods. There is no reason why we would
want to create the signers and keys for each test case and this only
slows the test module execution.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 15:39:19 +02:00
Martin Vrachev
2c03f6d1fe Rename expected_result to expected_error
Rename expected_result to expected_error as the value expected is never
a true result in the current test cases, but it's an expected error.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 15:39:19 +02:00
Martin Vrachev
68db62d827 Rename RootVersions to MdVersions
RootVersions can be used for key rotation tests beside root and as such
it makes sense to rename it to something more generic.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 15:39:19 +02:00
Jussi Kukkonen
dd5deeecf0
Merge pull request #1672 from ivanayov/ivanayov/rolenames
Create constants for top-level rolenames
2021-12-02 14:37:22 +02:00
Martin Vrachev
d697f73da2 Linting in tox.ini: use variable for target dirs
Instead of providing a target directory for linting by each of the
tools use one variable which will be the source of truth about which
directories do we lint.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 14:12:05 +02:00
Martin Vrachev
1ae6a22b8e Start linting the examples folder
The examples folder currently contains a repository example and it's
good if we start linting its content and as a result add type
annotations.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-02 13:40:14 +02:00
Ivana Atanasova
00589f08e3 Apply top-level rolenames constants in tests
This applies the use of constants of top-level rolenames in the
tests instead of the previously hardcoded strings.
Fixes #1648

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-12-02 12:25:48 +02:00
Ivana Atanasova
d7c653470a Create constants for top-level rolenames
This is a change in the metadata API to remove hardcoded rolenames
and use constants instead.
Fixes #1648

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-12-02 12:16:25 +02:00
Jussi Kukkonen
d991362ff0
Merge pull request #1699 from MVrachev/move-configs
Move linters configurations in pyproject.toml
2021-12-02 12:11:13 +02:00
Martin Vrachev
5c8a86665f Fix small pylint error
Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-01 16:28:15 +02:00
Martin Vrachev
ed8a06bcb3 Move part of isort options in pyproject.toml
We are using 4 linters: black, isort, pylint and mypy.
It's good if we use one file as a source for truth for all linter
configurations.

I tried multiple ways to use the src_path option,
so we can just call isort without pointing out the target folders, but I was not
successful.
I tried running isort with "isort --settings-path=pyproject.toml"
I got the error:
"Error: arguments passed in without any paths or content."

Additionally, I saw one project with source configuration https://github.com/Pylons/pyramid/blob/master/pyproject.toml,
but they had to give explicit folders too 8061fce297/tox.ini (L26)
and 8061fce297/tox.ini (L66)

It was a similar situation with "check" and "diff".
In the documentation it's said that for both check and diff are not
supported in configuration files.
See:
- https://pycqa.github.io/isort/docs/configuration/options.html#check
- https://pycqa.github.io/isort/docs/configuration/options.html#show-diff

Additionally, in two issues it was confirmed that in integration tests
we should use --check and --diff the way we did until now.

As a result, I moved part of the configuration options for isort inside
pyproject.toml without the actual directories that need to be linted
and "check" and "diff" options.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-01 16:28:15 +02:00
Martin Vrachev
4597761adb Move black configuration in pyproject.toml
We are using 4 linters: black, isort, pylint and mypy.
It's good if we use one file as a source for truth for all linter
configurations.

As a first step move black options in pyproject.toml.
I tried multiple ways to use the include option,
so we can just call black --config=pyproject.toml, but I was not
successful. Then I found this comment https://github.com/psf/black/issues/861#issuecomment-680411125
explaining that the path argument is mandatory.
As a result, I will move all configuration options for black inside
pyproject.toml without the actual directories that need to be linted.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-01 16:28:13 +02:00
Martin Vrachev
8ba3cc4394 Move mypy and pylint configs in pyproject.toml
This aims to add a single source of truth for pylint and mypy
configurations.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-12-01 15:49:41 +02:00
lukpueh
171f9ee51d
Merge pull request #1698 from jku/remove-roadmap
docs: Remove ROADMAP
2021-11-30 10:16:01 +01:00
Jussi Kukkonen
0a453e2fcc docs: Remove ROADMAP
This file is out of date to the point of being obsolete. An updated
ROADMAP document would be warmly welcome but an out of date roadmap
is worse than nothing.

Fixes #1525

Signed-off-by: Jussi Kukkonen <jkukkonen@vmware.com>
2021-11-30 10:22:17 +02:00
Jussi Kukkonen
bca1e67adc
Merge pull request #1686 from ivanayov/ivanayov/remove_url_normalisation
Remove URL normalisation
2021-11-30 10:02:45 +02:00
Jussi Kukkonen
2de883aedc
Merge pull request #1690 from kairoaraujo/issue#1558-delegate-roles-name-not-top-level-roles
Prevents delegate role name as top-level role name
2021-11-29 15:52:58 +02:00
Kairo de Araujo
72d68f34fb Remove the unnecessary set()
The set() is not required in the OrderedDict.

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-11-29 14:37:29 +01:00
lukpueh
51248db173
Merge pull request #1685 from lukpueh/repo-examples
Docs: Add repository tutorial based on metadata API
2021-11-29 13:31:16 +01:00
Lukas Puehringer
a1531d8c50 test: add test module for basic repo example
Adds new test module that executes the basic repo example
Python script and checks that it created certain (metadata)
files.

The test module is tailored for testing similar example scripts.

Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Jussi Kukkonen <jku@goto.fi>

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-11-29 13:25:26 +01:00
Lukas Puehringer
190c21b65b doc: add repository example based on metadata API
As 'repository_tool' and 'repository_lib' are being deprecated,
repository metadata must to be created and maintained manually
using the low-level Metadata API. The added example code shall
serve as temporary replacement until a new repository tool is
available.

The sample code contains the following repo workflows:
 - creation of top-level metadata
 - target file handling
 - consistent snapshots
 - key management
 - top-level delegation and signing thresholds
 - target delegation
 - in-band and out-of-band metadata signing
 - writing and reading metadata files
 - root key rotation

Co-authored-by: Teodora Sechkova <tsechkova@vmware.com>
Co-authored-by: Joshua Lock <jlock@vmware.com>
Co-authored-by: Jussi Kukkonen <jku@goto.fi>

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
2021-11-29 13:25:10 +01:00
Ivana Atanasova
aa591920a2 Remove URL normalisation
As a target path is a URL path it's not correct to consider it as
interchangeable with a filepath within every operation system. The
unquote is also removed as the ngclient cannot assume correctly
which encoding is intended and which not

Fixes #1483

Signed-off-by: Ivana Atanasova <iyovcheva@iyovcheva-a02.vmware.com>
2021-11-25 15:42:28 +02:00
Jussi Kukkonen
600eb8636a
Merge pull request #1687 from MVrachev/address-mypy-warnings
Tests on the new implementation: address mypy warnings
2021-11-25 11:48:08 +02:00
Martin Vrachev
e2deff3148 Address mypy warnings
This commit includes manual fixes for a lot of mypy warnings.
When there were warnings that we are calling non-annotated function
in annotated context I decided to add annotations instead of ignoring
those warnings.
That's how I end up adding annotations in the whole tests/utils.py
module.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 20:50:10 +02:00
Martin Vrachev
0d4d7f820c Add two more invalid serialization tests
Those tests are about missing "keys" and "roles" attributes in
Targets.Delegations.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Martin Vrachev
1f77b107df Fix annotations in test_metadata_serialization
In test_metadata_serialization.py "test_case_data" is actually a string
when the decorator calls the actual test functions.

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Martin Vrachev
e18b6ba506 Configure mypy to show error codes
By configuring mypy to show error codes when we get a warning by mypy
we will receive an error code as well.
Those error codes are useful when you want to disable specific mypy
warning for a line with:

Signed-off-by: Martin Vrachev <mvrachev@vmware.com>
2021-11-24 18:29:32 +02:00
Kairo de Araujo
b6c482e1ee Ccheck empty strings, remove comprehensive lists
- Add the check for empty strings in the Delegate Role name
- Remove the comprehensive lists to make the code more readable
- Remove the test for empty file name from
``test_updater_with_simulator``

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
2021-11-24 17:12:45 +01:00
Jussi Kukkonen
a24c4e95be
Merge pull request #1593 from sechkova/avoid_reloading_targets
ngclient: Avoid loading targets metadata twice
2021-11-24 14:15:19 +02:00
Jussi Kukkonen
8e6485b066
Merge pull request #1688 from MVrachev/delegationrole-attributes-validation
Delegations: paths & path_hash_prefixes validation
2021-11-24 14:12:02 +02:00