Update docstrings in repository lib's write and writeall
docstrings, to remove inaccurate information and add missing
information about consistent_snapshots.
In particular this commit adds details about how
consistent_snapshots and the recently added use_existing_fileinfo
options play together.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
'repository_lib._generate_and_write_metadata' used to pass on the
'consistent_snapshot' option as 'write_consistent_targets' to the
'generate_targets_metadata' function.
This equation prevented using `consistent_snapshot` (to create
version-prefixed metadata) together with the recently added
`use_existing_fileinfo` option (to handle target files out of
band), because `generate_targets_metadata` rightfully does not
accept `write_consistent_targets` and `use_existing_fileinfo`.
This PR fixes the issue by assuming write_consistent_targets == \
consistent_snapshot and not use_existing_fileinfo
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove unnecessary copying of paths to another list in
add_targets() and add_paths() methods.
Fix incorrect docstring text.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Use a hard-coded unix separator ('/') so that an
exception is also raised for paths starting with '/'
when executing on Windows systems.
Update test_check_path to explicitly test invalid paths
starting with Windows style separator.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Improve the coding style in TUTORIAL in the case
where absolute path to a file is needed to perform file system
access and at the same time is rejected by Targets methods.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
- add a test for _check_path() method of Targets class.
- update all tests calling _check_path() respectively
- update test_tutorial
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Replace the absolute paths returned by get_filepaths_in_directory()
in the tutorial with a hard-coded list of relaive filepaths since
add_target(s) and delegate() methods raise excception on absolute
paths.
Remove an obsolete warning about path pattern's location.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Normalization of the target path is no longer necessary
during metadata generation, since an exception is raised
earlier, on the addition of an incorrect target path or pattern.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Adopt a stict behavor allowing only paths that match
the definition of a PATHPATTERN or aTARGETPATH (use the forward
slash (/) as directory separator and do not start with a
directory separator). Raise an exception otherwise.
Rename the method to a more general _check_path().
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Adopt the same path verification strategy in all methods
of Targets class that add new target files by utilizing
the _check_relpath() method.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Adopt the same behavior in all methods of Targets class
(add_paths(), delegate()) that add new paths to a role.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
This method checks if the passed path is relative and returns
the normalized path. Checks are performed without accessing the
file system.
Signed-off-by: Teodora Sechkova <tsechkova@vmware.com>
Due to the performance overhead of deepcopy(), as used extensively in
roledb, the delegate function is rather slow. This is especially
noticeable when we have a large number_of_bins when calling
delegate_hashed_bins.
In order to be able to easily reduce the number of deepcopy() operations
we remove direct calls to delegate() and instead use the newly added
helper functions to replicate the behaviour, only with a single call
update to the roledb.
This improves the performance of a 16k bins delegation from a 1hr 24min
operation on my laptop to 33s.
Ideally once Issue #1005 has been properly fixed this commit can be
reverted and we can once again just call delegate() here.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Due to the performance overhead of deepcopy(), as used extensively in
roledb, the delegate function is rather slow. This is especially
noticeable when we have a large number_of_bins when calling
delegate_hashed_bins.
In order to be able to easily reduce the number of deepcopy() operations
we will need to to be able to replicate the logic of delegate() in
delegate_hashed_bins(), only with batched updates to the roledb.
To support that work we refactor delegate() to split out logic that will
be required in delegate_hashed_bins() into helper functions.
Signed-off-by: Joshua Lock <jlock@vmware.com>
delegate_hashed_bins() has a number_of_bins parameter which defaults to
1024. add_target_to_bin() and remove_target_from_bin() both have a
number_of_bins parameter with no default. This means that in the
(somewhat) unlikely case that someone is using the default
number_of_bins when creating hashed bins they will need know what that
default value is and pass it to add_target_to_bin() and
remove_target_from_bin().
In order to be consistent and simpler to use define the default number
of bins as a module level constant and use it as the default value for
the number_of_bins argument for each of:
* delegate_hashed_bins()
* add_target_to_bin()
* remove_target_from_bin()
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add some additional checks to test_add_target_to_bin to ensure the code
to add a target passing a fileinfo is tested.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Factor out code to calculate the prefix length, total number of prefixes
and number of prefixes to be allocated to each bin into a helper
function _get_bin_numbers() so that we a) do not duplicate logic,
b) can document the logic in one place and c) ensure the logic is
consistent.
Signed-off-by: Joshua Lock <jlock@vmware.com>
We intend to deprecate the custom parameter of add_target() in favour
of using the fileinfo parameter with the custom value populated,
therefore it does not make sense to _add_ the custom parameter to
add_target_to_bin()
Signed-off-by: Joshua Lock <jlock@vmware.com>
When testing delegate_hashed_bins to ensure that hash_path_prefixes
map to the generated name of the bin, also check to ensure that at least
one of the delegations contains one or more path_hash_prefixes.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Test the newly added functionality to:
* add a target to the repository without access to the target file on disk
* write targets metadata without access to target files on disk, by using
the existing fileinfo data from the roledb
Signed-off-by: Joshua Lock <jlock@vmware.com>
Merge the logger calls reporting information about the hashed bin
delegations into a single logger.info() call to ensure the messages
will be grouped together even when integrated into a logging system
with multiple parallel sources.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Previously, at the time of writing targets metadata, we accessed all
targets files and ensured they exist on disk before generating a
filinfo dict containing hashes of the target file.
This change enables targets metadata to be generated using the
fileinfo data that is already stored in the roledb.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add an additional optional parameter to add_target() and
add_target_to_bin() which is a fileinfo object matching
tuf.formats.FILEINFO_OBJECT
This parameter and the custom parameter are mutually exclusive and
thus cannot be passed at the same time.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Match the pattern in add_target() where if the filepath already exists
in roleinfo['paths'] it is updated to replace the existing entry with
the new fileinfo.
Signed-off-by: Joshua Lock <jlock@vmware.com>
The file isn't strictly needed on-disk at the time add_target() and
add_targets() are called and this duplicates the check for the file's
presence in write[_all]()
By removing this check we allow extra versatility in adding targets.
Signed-off-by: Joshua Lock <jlock@vmware.com>
In delegate_to_hashed_bins() duplicate the code which determines the
target's file path for writing to the metadata in repository_lib's
generate_targets_metadata to ensure that the target path hashed by
delegate_hashed_bins() matches the target path in the metadata file.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Vastly simplify the implementation, using the _get_hash() and
_find_bin_for_hash() helpers added in earlier commits.
Furthermore, enable passing of the custom parameter to
add_target_to_bin() to better match add_target()
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add a helper function to determine the name of a bin that a hashed
targetfile will be delegated to.
Based sketches by Lukas Puehringer in issues #994 & #995
Signed-off-by: Joshua Lock <jlock@vmware.com>
This helper will generate a hash of the passed target_filepath using the
algorithm defined by the repository_tool's HASH_FUNCTION variable.
Signed-off-by: Joshua Lock <jlock@vmware.com>
As we are adding and removing items from the hashed bins and checking
for their presence/absence it's simplest if we being with the hashed
bins initially empty.
If we pass a list of targets when we call delgate_hashed_bins() the
delegated roles have an initial set of targets delegated to them,
which complicates testing of adding then removing a target to a
delegated bin.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Add test to ensure delegated bin names are consistent with the hash
prefixes that are delegated to the role.
This is an implicit assumption of the current implementation, the
testing of which will enable us to modify the code with greater
confidence.
Signed-off-by: Joshua Lock <jlock@vmware.com>
An up-to-date version of the minimal client usage instructions
from client_setup_and_repository_example.md can be found in
the main TUF tutorial.
More comprehensive documentation of the tuf client can be found in
tuf/client/README.md, which also needs a revision.
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
Remove link to incomplete and severely outdated
client_setup_and_repository_example.md in client section of
TUTORIAL.md.
Instead we should link (or move the entire client tutorial part) to
tuf/client/README.md, which is more comprehensive and less outdated
than above document (see #808).
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
