From 6ae6a0cda060e2c018ad851b45ed8f1321029948 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Benno=20F=C3=BCnfst=C3=BCck?= Date: Sat, 3 Oct 2015 15:40:24 +0200 Subject: [PATCH 01/20] tuf-spec.txt: fix duplicate 'the' --- docs/tuf-spec.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tuf-spec.txt b/docs/tuf-spec.txt index 7abce4e8..d78a3785 100644 --- a/docs/tuf-spec.txt +++ b/docs/tuf-spec.txt @@ -277,7 +277,7 @@ Version 0.9 To prevent an adversary from replaying an out-of-date signed metadata file whose signature has not yet expired, an automated process periodically signs - a timestamped statement containing the the hash of the snapshot file. Even + a timestamped statement containing the hash of the snapshot file. Even though this timestamp key must be kept online, the risk posed to clients by compromise of this key is minimal. From 7aef036f3ad3c906b909e95846cc8e50e9e249e4 Mon Sep 17 00:00:00 2001 From: awwad Date: Mon, 5 Oct 2015 17:16:30 -0400 Subject: [PATCH 02/20] Link correction Correcting the Virtualenv links (introduction and installation). --- README.rst | 4 +- README.rst~ | 259 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 261 insertions(+), 2 deletions(-) create mode 100644 README.rst~ diff --git a/README.rst b/README.rst index 94606630..9728cd82 100644 --- a/README.rst +++ b/README.rst @@ -189,12 +189,12 @@ Instructions for Contributors Development: `https://github.com/theupdateframework/tuf `_ -`Virtualenv `_ +`Virtualenv `_ is a tool to create isolated Python environments. It also includes ``pip`` and ``setuptools``, Python packages used to install TUF and its dependencies. All installation methods of virtualenv are outlined in the `installation -section `_ +section `_ and instructions for installing locally from source here: :: diff --git a/README.rst~ b/README.rst~ new file mode 100644 index 00000000..d768d16c --- /dev/null +++ b/README.rst~ @@ -0,0 +1,259 @@ +A Framework for Securing Software Update Systems +------------------------------------------------ + +.. image:: https://travis-ci.org/theupdateframework/tuf.svg?branch=develop + :target: https://travis-ci.org/theupdateframework/tuf + +.. image:: https://coveralls.io/repos/theupdateframework/tuf/badge.png?branch=develop + :target: https://coveralls.io/r/theupdateframework/tuf?branch=develop + + +TUF (The Update Framework) helps developers secure their new or existing +software update systems. Software update systems are vulnerable to many +known attacks, including those that can result in clients being +compromised or crashed. TUF helps solve this problem by providing a +flexible security framework that can be added to software updaters. + +What Is a Software Update System? +--------------------------------- + +Generally, a software update system is an application (or part of an +application) running on a client system that obtains and installs +software. This can include updates to software that is already installed +or even completely new software. + +Three major classes of software update systems are: + +- **Application updaters** which are used by applications to update + themselves. For example, Firefox updates itself through its own + application updater. + +- **Library package managers** such as those offered by many + programming languages for installing additional libraries. These are + systems such as Python's pip/easy_install + PyPI, Perl's CPAN, + Ruby's Gems, and PHP's PEAR. + +- **System package managers** used by operating systems to update and + install all of the software on a client system. Debian's APT, Red + Hat's YUM, and openSUSE's YaST are examples of these. + +Our Approach +------------ + +There are literally thousands of different software update systems in +common use today. (In fact the average Windows user has about `two +dozen `_ +different software updaters on their machine!) + +We are building a library that can be universally (and in most cases +transparently) used to secure software update systems. + +Overview +-------- + +At the highest level, TUF simply provides applications with a secure +method of obtaining files and knowing when new versions of files are +available. We call these files, the ones that are supposed to be +downloaded, "target files". The most common need for these abilities is +in software update systems and that's what we had in mind when creating +TUF. + +On the surface, this all sounds simple. Securely obtaining updates just +means: + +- Knowing when an update exists. +- Downloading the updated file. + +The problem is that this is only simple when there are no malicious +parties involved. If an attacker is trying to interfere with these +seemingly simple steps, there is plenty they can do. + +Background +---------- + +Let's assume you take the approach that most systems do (at least, the +ones that even try to be secure). You download both the file you want +and a cryptographic signature of the file. You already know which key +you trust to make the signature. You check that the signature is correct +and was made by this trusted key. All seems well, right? Wrong. You are +still at risk in many ways, including: + +- An attacker keeps giving you the same file, so you never realize + there is an update. +- An attacker gives you an older, insecure version of a file that you + already have, so you download that one and blindly use it thinking + it's newer. +- An attacker gives you a newer version of a file you have but it's not + the newest one. It's newer to you, but it may be insecure and + exploitable by the attacker. +- An attacker compromises the key used to sign these files and now you + download a malicious file that is properly signed. + +These are just some of the attacks software update systems are +vulnerable to when only using signed files. See +`Security `_ for a full list of attacks and updater +weaknesses TUF is designed to prevent. + +The following papers provide detailed information on securing software +updater systems, TUF's design and implementation details, attacks on +package managers, and package management security: + +- `Survivable Key Compromise in Software Update + Systems `_ + +- `A Look In the Mirror: Attacks on Package + Managers `_ + +- `Package Management + Security `_ + +What TUF Does +------------- + +In order to securely download and verify target files, TUF requires a +few extra files to exist on a repository. These are called metadata +files. TUF metadata files contain additional information, including +information about which keys are trusted, the cryptographic hashes of +files, signatures on the metadata, metadata version numbers, and the +date after which the metadata should be considered expired. + +When a software update system using TUF wants to check for updates, it +asks TUF to do the work. That is, your software update system never has +to deal with this additional metadata or understand what's going on +underneath. If TUF reports back that there are updates available, your +software update system can then ask TUF to download these files. TUF +downloads them and checks them against the TUF metadata that it also +downloads from the repository. If the downloaded target files are +trustworthy, TUF hands them over to your software update system. See +`Metadata `_ for more information and examples. + +TUF specification document is also available: + +- `The Update Framework Specification `_ + +TUF Home Page +------------- + +The home page for the TUF project can be found at: +https://updateframework.com + +Mailing List +------------ +Please visit `https://groups.google.com/forum/?fromgroups#!forum/theupdateframework `_ if you would like to contact the TUF team. Questions, feedback, and suggestions are welcomed in this low-volume mailing list. + +A group feed is available at: https://groups.google.com/forum/feed/theupdateframework/msgs/atom.xml?num=50 + + +Installation +------------ + +:: + + pip - installing and managing Python packages (recommended) + + Installing from Python Package Index (https://pypi.python.org/pypi). + Note: Please use "pip install --no-use-wheel tuf" if your version + of pip <= 1.5.6 + $ pip install tuf + + Installing from local source archive. + $ pip install + + Or from the root directory of the unpacked archive. + $ pip install . + +Installation of Optional Requirements (after minimal install) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The optional ``tuf[tools]`` can be installed by repository maintainers +that need to generate TUF repository files, such as metadata, +cryptographic keys, and signatures. Whereas the minimal install can only +verify ed25519 signatures and is intended for sofware updater clients, +``tuf[tools]`` provides repository maintainers secure ed25519 key and +signature generation with PyNaCl / libsodium. + +TUF tools also enable general-purpose cryptography with PyCrypto. +Software updaters that want to support verification of RSASSA-PSS +signatures should require their clients to install ``tuf[tools]``. + +Installing extras does not work if minimal install was a wheel (pip <= 1.5.6.) +`https://github.com/pypa/pip/issues/1885 `_ + +:: + + $ pip install --no-use-wheel tuf + $ pip install tuf[tools] + +Instructions for Contributors +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Development: `https://github.com/theupdateframework/tuf `_ + +`Virtualenv `_ +is a tool to create isolated Python environments. It also includes +``pip`` and ``setuptools``, Python packages used to install TUF and its +dependencies. All installation methods of virtualenv are outlined in the +`installation +section `_ +and instructions for installing locally from source here: +:: + + $ curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.11.6.tar.gz + $ tar xvfz virtualenv-1.11.6.tar.gz + $ cd virtualenv-1.11.6 + $ python virtualenv.py myVE + + +PyCrypto and PyNaCl (third-party dependencies needed by the repository +tools) require Python and FFI (Foreign Function Interface) development +header files. Debian-based distributions can install these header +libraries with apt (Advanced Package Tool.) +:: + + $ apt-get install python-dev + $ apt-get install libffi-dev + +Installation of minimal, optional, development, and testing requirements +can then be accomplished with one command: +:: + + $ pip install -r dev-requirements.txt + +The Update Framework's unit tests can be executed by invoking +`tox `_. All supported Python versions are +tested, but must already be installed locally. +:: + + $ tox + +Using TUF +--------- + +TUF has four major classes of users: clients, for whom TUF is largely +transparent; mirrors, who will (in most cases) have nothing at all to do +with TUF; upstream servers, who will largely be responsible for care and +feeding of repositories; and integrators, who do the work of putting TUF +into existing projects. + +A low-level integration requires importing a single module and calling +particular methods to perform updates. A high-level integration, on the +other hand, can handle TUF-related updates transparently. The client +populates a configuration file and the library interposes on urllib calls. +Generating metadata files stored on upstream servers can be handled by the +repository tool, covered in ``Creating a Repository``. + + +- `Creating a Repository `_ + +- `Low-level Integration `_ + +- `High-level Integration `_ + +Acknowledgements +---------------- + +This material is based upon work supported by the National Science +Foundation under Grant No. CNS-1345049 and CNS-0959138. Any opinions, +findings, and conclusions or recommendations expressed in this material +are those of the author(s) and do not necessarily reflect the views of +the National Science Foundation. From da1486134254906a802e98b66eaa4ba7fc788b52 Mon Sep 17 00:00:00 2001 From: awwad Date: Mon, 5 Oct 2015 17:20:13 -0400 Subject: [PATCH 03/20] Removing temp file cruft Removing temp file cruft --- README.rst~ | 259 ---------------------------------------------------- 1 file changed, 259 deletions(-) delete mode 100644 README.rst~ diff --git a/README.rst~ b/README.rst~ deleted file mode 100644 index d768d16c..00000000 --- a/README.rst~ +++ /dev/null @@ -1,259 +0,0 @@ -A Framework for Securing Software Update Systems ------------------------------------------------- - -.. image:: https://travis-ci.org/theupdateframework/tuf.svg?branch=develop - :target: https://travis-ci.org/theupdateframework/tuf - -.. image:: https://coveralls.io/repos/theupdateframework/tuf/badge.png?branch=develop - :target: https://coveralls.io/r/theupdateframework/tuf?branch=develop - - -TUF (The Update Framework) helps developers secure their new or existing -software update systems. Software update systems are vulnerable to many -known attacks, including those that can result in clients being -compromised or crashed. TUF helps solve this problem by providing a -flexible security framework that can be added to software updaters. - -What Is a Software Update System? ---------------------------------- - -Generally, a software update system is an application (or part of an -application) running on a client system that obtains and installs -software. This can include updates to software that is already installed -or even completely new software. - -Three major classes of software update systems are: - -- **Application updaters** which are used by applications to update - themselves. For example, Firefox updates itself through its own - application updater. - -- **Library package managers** such as those offered by many - programming languages for installing additional libraries. These are - systems such as Python's pip/easy_install + PyPI, Perl's CPAN, - Ruby's Gems, and PHP's PEAR. - -- **System package managers** used by operating systems to update and - install all of the software on a client system. Debian's APT, Red - Hat's YUM, and openSUSE's YaST are examples of these. - -Our Approach ------------- - -There are literally thousands of different software update systems in -common use today. (In fact the average Windows user has about `two -dozen `_ -different software updaters on their machine!) - -We are building a library that can be universally (and in most cases -transparently) used to secure software update systems. - -Overview --------- - -At the highest level, TUF simply provides applications with a secure -method of obtaining files and knowing when new versions of files are -available. We call these files, the ones that are supposed to be -downloaded, "target files". The most common need for these abilities is -in software update systems and that's what we had in mind when creating -TUF. - -On the surface, this all sounds simple. Securely obtaining updates just -means: - -- Knowing when an update exists. -- Downloading the updated file. - -The problem is that this is only simple when there are no malicious -parties involved. If an attacker is trying to interfere with these -seemingly simple steps, there is plenty they can do. - -Background ----------- - -Let's assume you take the approach that most systems do (at least, the -ones that even try to be secure). You download both the file you want -and a cryptographic signature of the file. You already know which key -you trust to make the signature. You check that the signature is correct -and was made by this trusted key. All seems well, right? Wrong. You are -still at risk in many ways, including: - -- An attacker keeps giving you the same file, so you never realize - there is an update. -- An attacker gives you an older, insecure version of a file that you - already have, so you download that one and blindly use it thinking - it's newer. -- An attacker gives you a newer version of a file you have but it's not - the newest one. It's newer to you, but it may be insecure and - exploitable by the attacker. -- An attacker compromises the key used to sign these files and now you - download a malicious file that is properly signed. - -These are just some of the attacks software update systems are -vulnerable to when only using signed files. See -`Security `_ for a full list of attacks and updater -weaknesses TUF is designed to prevent. - -The following papers provide detailed information on securing software -updater systems, TUF's design and implementation details, attacks on -package managers, and package management security: - -- `Survivable Key Compromise in Software Update - Systems `_ - -- `A Look In the Mirror: Attacks on Package - Managers `_ - -- `Package Management - Security `_ - -What TUF Does -------------- - -In order to securely download and verify target files, TUF requires a -few extra files to exist on a repository. These are called metadata -files. TUF metadata files contain additional information, including -information about which keys are trusted, the cryptographic hashes of -files, signatures on the metadata, metadata version numbers, and the -date after which the metadata should be considered expired. - -When a software update system using TUF wants to check for updates, it -asks TUF to do the work. That is, your software update system never has -to deal with this additional metadata or understand what's going on -underneath. If TUF reports back that there are updates available, your -software update system can then ask TUF to download these files. TUF -downloads them and checks them against the TUF metadata that it also -downloads from the repository. If the downloaded target files are -trustworthy, TUF hands them over to your software update system. See -`Metadata `_ for more information and examples. - -TUF specification document is also available: - -- `The Update Framework Specification `_ - -TUF Home Page -------------- - -The home page for the TUF project can be found at: -https://updateframework.com - -Mailing List ------------- -Please visit `https://groups.google.com/forum/?fromgroups#!forum/theupdateframework `_ if you would like to contact the TUF team. Questions, feedback, and suggestions are welcomed in this low-volume mailing list. - -A group feed is available at: https://groups.google.com/forum/feed/theupdateframework/msgs/atom.xml?num=50 - - -Installation ------------- - -:: - - pip - installing and managing Python packages (recommended) - - Installing from Python Package Index (https://pypi.python.org/pypi). - Note: Please use "pip install --no-use-wheel tuf" if your version - of pip <= 1.5.6 - $ pip install tuf - - Installing from local source archive. - $ pip install - - Or from the root directory of the unpacked archive. - $ pip install . - -Installation of Optional Requirements (after minimal install) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The optional ``tuf[tools]`` can be installed by repository maintainers -that need to generate TUF repository files, such as metadata, -cryptographic keys, and signatures. Whereas the minimal install can only -verify ed25519 signatures and is intended for sofware updater clients, -``tuf[tools]`` provides repository maintainers secure ed25519 key and -signature generation with PyNaCl / libsodium. - -TUF tools also enable general-purpose cryptography with PyCrypto. -Software updaters that want to support verification of RSASSA-PSS -signatures should require their clients to install ``tuf[tools]``. - -Installing extras does not work if minimal install was a wheel (pip <= 1.5.6.) -`https://github.com/pypa/pip/issues/1885 `_ - -:: - - $ pip install --no-use-wheel tuf - $ pip install tuf[tools] - -Instructions for Contributors -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Development: `https://github.com/theupdateframework/tuf `_ - -`Virtualenv `_ -is a tool to create isolated Python environments. It also includes -``pip`` and ``setuptools``, Python packages used to install TUF and its -dependencies. All installation methods of virtualenv are outlined in the -`installation -section `_ -and instructions for installing locally from source here: -:: - - $ curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.11.6.tar.gz - $ tar xvfz virtualenv-1.11.6.tar.gz - $ cd virtualenv-1.11.6 - $ python virtualenv.py myVE - - -PyCrypto and PyNaCl (third-party dependencies needed by the repository -tools) require Python and FFI (Foreign Function Interface) development -header files. Debian-based distributions can install these header -libraries with apt (Advanced Package Tool.) -:: - - $ apt-get install python-dev - $ apt-get install libffi-dev - -Installation of minimal, optional, development, and testing requirements -can then be accomplished with one command: -:: - - $ pip install -r dev-requirements.txt - -The Update Framework's unit tests can be executed by invoking -`tox `_. All supported Python versions are -tested, but must already be installed locally. -:: - - $ tox - -Using TUF ---------- - -TUF has four major classes of users: clients, for whom TUF is largely -transparent; mirrors, who will (in most cases) have nothing at all to do -with TUF; upstream servers, who will largely be responsible for care and -feeding of repositories; and integrators, who do the work of putting TUF -into existing projects. - -A low-level integration requires importing a single module and calling -particular methods to perform updates. A high-level integration, on the -other hand, can handle TUF-related updates transparently. The client -populates a configuration file and the library interposes on urllib calls. -Generating metadata files stored on upstream servers can be handled by the -repository tool, covered in ``Creating a Repository``. - - -- `Creating a Repository `_ - -- `Low-level Integration `_ - -- `High-level Integration `_ - -Acknowledgements ----------------- - -This material is based upon work supported by the National Science -Foundation under Grant No. CNS-1345049 and CNS-0959138. Any opinions, -findings, and conclusions or recommendations expressed in this material -are those of the author(s) and do not necessarily reflect the views of -the National Science Foundation. From cb1591e63da686842ba1e92bec652964786bc3d6 Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Wed, 11 Nov 2015 12:55:30 -0500 Subject: [PATCH 04/20] Update tuf-spec.txt Fix for issue #296. We should document which underlying hash function is used with PKCS #1 RSA PSS signatures to maintain compatibility with different clients, or integrators that wish to verify/test our metadata. --- docs/tuf-spec.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/tuf-spec.txt b/docs/tuf-spec.txt index d78a3785..aed3148f 100644 --- a/docs/tuf-spec.txt +++ b/docs/tuf-spec.txt @@ -433,6 +433,7 @@ Version 0.9 key type, or cryptographic library: "RSASSA-PSS" : RSA Probabilistic signature scheme with appendix. + The underlying hash function is SHA256. "ed25519" : Elliptic curve digital signature algorithm based on Twisted Edwards curves. From b60b741261bd6965e5d7f633160d9493f4941bb7 Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Tue, 17 Nov 2015 15:19:24 -0500 Subject: [PATCH 05/20] Update tuf-spec.txt * @jawi recommended that the specification say that it is up to implementers to decide how keys should be securely stored. * Python implementation -> reference implementation: for clarity and to be consistent with other documents. --- docs/tuf-spec.txt | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/tuf-spec.txt b/docs/tuf-spec.txt index aed3148f..0867dce5 100644 --- a/docs/tuf-spec.txt +++ b/docs/tuf-spec.txt @@ -39,7 +39,7 @@ Version 0.9 (http://www.geni.net/) (http://www.nsf.gov/) - TUF's Python implementation is based heavily on Thandy, the application + TUF's reference implementation is based heavily on Thandy, the application updater for Tor (http://www.torproject.org/). Its design and this spec are also largely based on Thandy's, with many parts being directly borrowed from Thandy. The Thandy spec can be found here: @@ -428,7 +428,7 @@ Version 0.9 METHOD is the key signing method used to generate the signature. SIGNATURE is a signature of the canonical JSON form of ROLE. - The current Python implementation of TUF defines two signing methods, + The current reference implementation of TUF defines two signing methods, although TUF is not restricted to any particular key signing method, key type, or cryptographic library: @@ -955,9 +955,11 @@ Version 0.9 6.1. Key management and migration - All keys except the timestamp file signing key and the mirror list signing - key should be stored securely offline (e.g. encrypted and on a separate - machine, in special-purpose hardware, etc.). + All keys, except those for the timestamp and mirrors roles, should be + stored securely offline (e.g. encrypted and on a separate machine, in + special-purpose hardware, etc.). This document does not prescribe how keys + should be encrypted and stored, and so it is left to implementers of + this document to decide how best to secure them. To replace a compromised root key or any other top-level role key, the root role signs a new root.json file that lists the updated trusted keys for the From 1f4f7d4b63e8085f528981212acb6e66dc0d9fe5 Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Tue, 17 Nov 2015 15:28:18 -0500 Subject: [PATCH 06/20] Add a README to the "examples" directory. Also warn users that the examples might not be fully up-to-date, and to please consult the examples used by the unit tests. --- examples/README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 examples/README.md diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 00000000..220f4725 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,12 @@ +This directory contains an example of a TUF repository, metadata, and key and +client files. + +## WARNING ## +These examples were last updated 2 years ago. We have since made changes to the +format of our metadata and key files, and will need to regenerate them so the +new tools can properly load them. We are currently working on a 1.0 release +that will make further tweaks to the format of metadata and key files, so these +examples will be modified once again. + +Note: The examples that are up-to-date and normally tested are located here: +https://github.com/theupdateframework/tuf/tree/develop/tests/repository_data/ From 4172479d3934d869e9c8c2bf2f0745973f044c04 Mon Sep 17 00:00:00 2001 From: David Halls Date: Mon, 4 Jan 2016 22:00:13 +0000 Subject: [PATCH 07/20] Fix Python 3 errors - Signing data should be byte strings - Don't log exception outside handler (#300) --- tuf/client/updater.py | 4 ++-- tuf/formats.py | 7 +++++-- tuf/pyca_crypto_keys.py | 6 +++--- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/tuf/client/updater.py b/tuf/client/updater.py index bb4eb375..dd2cb01f 100755 --- a/tuf/client/updater.py +++ b/tuf/client/updater.py @@ -1226,8 +1226,8 @@ def _get_file(self, filepath, verify_file_function, file_type, return file_object else: - logger.exception('Failed to update {0} from all mirrors: {1}'.format( - filepath, file_mirror_errors)) + logger.error('Failed to update {0} from all mirrors: {1}'.format( + filepath, file_mirror_errors)) raise tuf.NoWorkingMirrorError(file_mirror_errors) diff --git a/tuf/formats.py b/tuf/formats.py index 66d4d4a0..24e38a95 100755 --- a/tuf/formats.py +++ b/tuf/formats.py @@ -151,8 +151,8 @@ NAME_SCHEMA = SCHEMA.AnyString() NAMES_SCHEMA = SCHEMA.ListOf(NAME_SCHEMA) -# A string representing data. -DATA_SCHEMA = SCHEMA.AnyString() +# A byte string representing data. +DATA_SCHEMA = SCHEMA.AnyBytes() # Supported hash algorithms. HASHALGORITHMS_SCHEMA = SCHEMA.ListOf(SCHEMA.OneOf( @@ -187,6 +187,9 @@ # A PyCrypto signature. PYCRYPTOSIGNATURE_SCHEMA = SCHEMA.AnyBytes() +# A pyca-cryptography signature. +PYCACRYPTOSIGNATURE_SCHEMA = SCHEMA.AnyBytes() + # An RSA key in PEM format. PEMRSA_SCHEMA = SCHEMA.AnyString() diff --git a/tuf/pyca_crypto_keys.py b/tuf/pyca_crypto_keys.py index 64600610..6b854f8c 100755 --- a/tuf/pyca_crypto_keys.py +++ b/tuf/pyca_crypto_keys.py @@ -264,7 +264,7 @@ def create_rsa_signature(private_key, data): True >>> method == 'RSASSA-PSS' True - >>> tuf.formats.PYCRYPTOSIGNATURE_SCHEMA.matches(signature) + >>> tuf.formats.PYCACRYPTOSIGNATURE_SCHEMA.matches(signature) True @@ -404,7 +404,7 @@ def verify_rsa_signature(signature, signature_method, public_key, data): tuf.formats.NAME_SCHEMA.check_match(signature_method) # Does 'signature' have the correct format? - tuf.formats.PYCRYPTOSIGNATURE_SCHEMA.check_match(signature) + tuf.formats.PYCACRYPTOSIGNATURE_SCHEMA.check_match(signature) # What about 'data'? tuf.formats.DATA_SCHEMA.check_match(data) @@ -428,7 +428,7 @@ def verify_rsa_signature(signature, signature_method, public_key, data): salt_length=hashes.SHA256().digest_size), hashes.SHA256()) - verifier.update(data.encode('utf-8')) + verifier.update(data) # verify() raises 'cryptograpahy.exceptions.InvalidSignature' if the # signature is invalid. From c859eeb2588393b26903f21822a4d28546511efc Mon Sep 17 00:00:00 2001 From: David Halls Date: Wed, 13 Jan 2016 23:02:17 +0000 Subject: [PATCH 08/20] Address code review comments --- tests/test_pyca_crypto_keys.py | 2 +- tests/test_pycrypto_keys.py | 3 ++- tuf/pycrypto_keys.py | 4 ++-- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/tests/test_pyca_crypto_keys.py b/tests/test_pyca_crypto_keys.py index f16e855f..fbfe2387 100755 --- a/tests/test_pyca_crypto_keys.py +++ b/tests/test_pyca_crypto_keys.py @@ -84,7 +84,7 @@ def test_create_rsa_signature(self): crypto_keys.create_rsa_signature, '', data) # Check for invalid 'data'. - self.assertRaises(TypeError, + self.assertRaises(tuf.FormatError, crypto_keys.create_rsa_signature, private_rsa, '') self.assertRaises(tuf.FormatError, diff --git a/tests/test_pycrypto_keys.py b/tests/test_pycrypto_keys.py index 14d00245..4f6c36e2 100755 --- a/tests/test_pycrypto_keys.py +++ b/tests/test_pycrypto_keys.py @@ -84,7 +84,8 @@ def test_create_rsa_signature(self): pycrypto.create_rsa_signature, '', data) # Check for invalid 'data'. - pycrypto.create_rsa_signature(private_rsa, '') + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_signature, private_rsa, '') # create_rsa_signature should reject non-string data. self.assertRaises(tuf.FormatError, diff --git a/tuf/pycrypto_keys.py b/tuf/pycrypto_keys.py index 0fb2cb27..789a6ab1 100755 --- a/tuf/pycrypto_keys.py +++ b/tuf/pycrypto_keys.py @@ -294,7 +294,7 @@ def create_rsa_signature(private_key, data): # If the passphrase is incorrect, PyCrypto returns: "RSA key format is not # supported". try: - sha256_object = Crypto.Hash.SHA256.new(data.encode('utf-8')) + sha256_object = Crypto.Hash.SHA256.new(data) rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) except (ValueError, IndexError, TypeError) as e: @@ -396,7 +396,7 @@ def verify_rsa_signature(signature, signature_method, public_key, data): try: rsa_key_object = Crypto.PublicKey.RSA.importKey(public_key) pkcs1_pss_verifier = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) - sha256_object = Crypto.Hash.SHA256.new(data.encode('utf')) + sha256_object = Crypto.Hash.SHA256.new(data) valid_signature = pkcs1_pss_verifier.verify(sha256_object, signature) except (ValueError, IndexError, TypeError) as e: From 47cae502fa597d74a277bc7f348508951b3f001f Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Thu, 14 Jan 2016 17:25:08 -0500 Subject: [PATCH 09/20] Allow users to specify the path (via conf.py) to save log files --- tests/test_log.py | 3 +++ tuf/conf.py | 8 ++++++++ tuf/log.py | 22 +++++++++++++++------- 3 files changed, 26 insertions(+), 7 deletions(-) diff --git a/tests/test_log.py b/tests/test_log.py index aaf70155..c46af756 100755 --- a/tests/test_log.py +++ b/tests/test_log.py @@ -112,6 +112,9 @@ def test_add_console_handler(self): # Test for invalid argument. self.assertRaises(tuf.FormatError, tuf.log.add_console_handler, 51) + # Test that an exception is printed to the console. Note: A stack trace + # is not included in the exception output because 'log.py' applies a filter + # to minimize the amount of output to the console. try: raise TypeError('Test exception output in the console.') diff --git a/tuf/conf.py b/tuf/conf.py index 1f347603..61dc2f55 100755 --- a/tuf/conf.py +++ b/tuf/conf.py @@ -46,6 +46,14 @@ # http://docs.python.org/2/library/ssl.html#certificates ssl_certificates = None +# The 'log.py' module manages TUF's logging system. Users have the option to +# enable/disable logging to a file via 'ENABLE_FILE_LOGGING' +ENABLE_FILE_LOGGING = True + +# If file logging is enabled via 'ENABLE_FILE_LOGGING', TUF log messages will +# be saved to 'LOG_FILENAME' +LOG_FILENAME = 'tuf.log' + # Since the timestamp role does not have signed metadata about itself, we set a # default but sane upper bound for the number of bytes required to download it. DEFAULT_TIMESTAMP_REQUIRED_LENGTH = 16384 #bytes diff --git a/tuf/log.py b/tuf/log.py index 92281f31..806576b2 100755 --- a/tuf/log.py +++ b/tuf/log.py @@ -66,15 +66,16 @@ import logging import time +import os import tuf import tuf.formats +import tuf.conf # Setting a handler's log level filters only logging messages of that level # (and above). For example, setting the built-in StreamHandler's log level to # 'logging.WARNING' will cause the stream handler to only process messages # of levels: WARNING, ERROR, and CRITICAL. -_DEFAULT_LOG_FILENAME = 'tuf.log' _DEFAULT_LOG_LEVEL = logging.DEBUG _DEFAULT_CONSOLE_LOG_LEVEL = logging.INFO _DEFAULT_FILE_LOG_LEVEL = logging.DEBUG @@ -104,18 +105,25 @@ console_handler = None # Set the built-in file handler. Messages will be logged to -# '_DEFAULT_LOG_FILENAME', and only those messages with a log level of +# 'tuf.conf.LOG_FILENAME', and only those messages with a log level of # '_DEFAULT_LOG_LEVEL'. The log level of messages handled by 'file_handler' -# may be modified with 'set_filehandler_log_level()'. '_DEFAULT_LOG_FILENAME' +# may be modified with 'set_filehandler_log_level()'. 'tuf.conf.LOG_FILENAME' # will be opened in append mode. -file_handler = logging.FileHandler(_DEFAULT_LOG_FILENAME) -file_handler.setLevel(_DEFAULT_FILE_LOG_LEVEL) -file_handler.setFormatter(formatter) +if tuf.conf.ENABLE_FILE_LOGGING: + file_handler = logging.FileHandler(tuf.conf.LOG_FILENAME) + file_handler.setLevel(_DEFAULT_FILE_LOG_LEVEL) + file_handler.setFormatter(formatter) +else: + pass + # Set the logger and its settings. logger = logging.getLogger('tuf') logger.setLevel(_DEFAULT_LOG_LEVEL) -logger.addHandler(file_handler) +if tuf.conf.ENABLE_FILE_LOGGING: + logger.addHandler(file_handler) +else: + pass # Silently ignore logger exceptions. logging.raiseExceptions = False From d992c5dd3e56108e4ab91a42cc4dbccb833b562c Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Thu, 14 Jan 2016 17:58:46 -0500 Subject: [PATCH 10/20] Set the logger (prior to adding a potential file handler) to avoid multiple if-else statements --- tuf/log.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/tuf/log.py b/tuf/log.py index 806576b2..a22bed2c 100755 --- a/tuf/log.py +++ b/tuf/log.py @@ -104,6 +104,10 @@ # set by default. console_handler = None +# Set the logger and its settings. +logger = logging.getLogger('tuf') +logger.setLevel(_DEFAULT_LOG_LEVEL) + # Set the built-in file handler. Messages will be logged to # 'tuf.conf.LOG_FILENAME', and only those messages with a log level of # '_DEFAULT_LOG_LEVEL'. The log level of messages handled by 'file_handler' @@ -113,14 +117,6 @@ file_handler = logging.FileHandler(tuf.conf.LOG_FILENAME) file_handler.setLevel(_DEFAULT_FILE_LOG_LEVEL) file_handler.setFormatter(formatter) -else: - pass - - -# Set the logger and its settings. -logger = logging.getLogger('tuf') -logger.setLevel(_DEFAULT_LOG_LEVEL) -if tuf.conf.ENABLE_FILE_LOGGING: logger.addHandler(file_handler) else: pass From 015deb45707024e0af63bea7e739ebdfbbf8b3a8 Mon Sep 17 00:00:00 2001 From: David Halls Date: Fri, 15 Jan 2016 09:18:25 +0000 Subject: [PATCH 11/20] Fix Python 3 encoding issues in pcyca_crypto_keys.py --- tuf/pyca_crypto_keys.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tuf/pyca_crypto_keys.py b/tuf/pyca_crypto_keys.py index 6b854f8c..8067024f 100755 --- a/tuf/pyca_crypto_keys.py +++ b/tuf/pyca_crypto_keys.py @@ -917,7 +917,7 @@ def _encrypt(key_data, derived_key_information): # Encrypt the plaintext and get the associated ciphertext. # Do we need to check for any exceptions? - ciphertext = encryptor.update(key_data) + encryptor.finalize() + ciphertext = encryptor.update(key_data.encode('utf-8')) + encryptor.finalize() # Generate the hmac of the ciphertext to ensure it has not been modified. # The decryption routine may verify a ciphertext without having to perform @@ -942,7 +942,7 @@ def _encrypt(key_data, derived_key_information): # of the fields it is separating. return binascii.hexlify(salt).decode() + _ENCRYPTION_DELIMITER + \ str(iterations) + _ENCRYPTION_DELIMITER + \ - hmac_value + _ENCRYPTION_DELIMITER + \ + hmac_value.decode() + _ENCRYPTION_DELIMITER + \ binascii.hexlify(iv).decode() + _ENCRYPTION_DELIMITER + \ binascii.hexlify(ciphertext).decode() @@ -993,7 +993,7 @@ def _decrypt(file_contents, password): generated_hmac = binascii.hexlify(generated_hmac_object.finalize()) - if not tuf.util.digests_are_equal(generated_hmac, hmac): + if not tuf.util.digests_are_equal(generated_hmac.decode(), hmac): raise tuf.CryptoError('Decryption failed.') # Construct a Cipher object, with the key and iv. From 86b7af94f2dc4d79140f0eb5ad0df9dbf0cc7602 Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Fri, 15 Jan 2016 11:04:21 -0500 Subject: [PATCH 12/20] Remove 'import os' statement (the os module is not needed) --- tuf/log.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tuf/log.py b/tuf/log.py index a22bed2c..48e8639d 100755 --- a/tuf/log.py +++ b/tuf/log.py @@ -66,7 +66,6 @@ import logging import time -import os import tuf import tuf.formats From 1c51b8d732d553e87a3b4555b369294c75f2389e Mon Sep 17 00:00:00 2001 From: David Halls Date: Tue, 19 Jan 2016 07:12:07 +0000 Subject: [PATCH 13/20] Don't import tuf.log in __init__.py so importing tuf.conf doesn't start logging --- tuf/__init__.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/tuf/__init__.py b/tuf/__init__.py index 4add6e45..b3abc8a2 100755 --- a/tuf/__init__.py +++ b/tuf/__init__.py @@ -28,14 +28,8 @@ from __future__ import division from __future__ import unicode_literals -import logging - -import tuf.log - import six -logging = logging.getLogger('tuf.__init__') - # Import 'tuf.formats' if a module tries to import the # entire tuf package (i.e., from tuf import *). __all__ = ['formats'] From cb687b4c2ff4ff37066cd4b7a193690035cd6498 Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Wed, 20 Jan 2016 10:29:58 -0500 Subject: [PATCH 14/20] Import tuf.log in test_init and fix logger name. Issue #303. In PR #307, tuf.log is no longer imported by __init__.py. Add an import statement for tuf.log to fix missing handler warning in 'test_init.py. The logger name specified is also incorrect. --- tests/test_init.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/test_init.py b/tests/test_init.py index 07d45f9e..b4550443 100755 --- a/tests/test_init.py +++ b/tests/test_init.py @@ -29,8 +29,9 @@ import logging import tuf +import tuf.log -logger = logging.getLogger('tuf.test_keys') +logger = logging.getLogger('tuf.test_init') class TestInit(unittest.TestCase): def setUp(self): From 294def9ade2b242990a91ab0eb685a77804325df Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Thu, 21 Jan 2016 11:23:32 -0500 Subject: [PATCH 15/20] Update tox.ini --- tox.ini | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tox.ini b/tox.ini index 029452da..39f1c978 100644 --- a/tox.ini +++ b/tox.ini @@ -4,8 +4,7 @@ # and then run "tox" from this directory. [tox] -#envlist = py26, py27, py32, py33, py34 -envlist = py27 +envlist = py26, py27, py32, py33, py34 [testenv] changedir = tests From 5d199750295bc321ac47274cf1128add307818b5 Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Thu, 21 Jan 2016 13:04:29 -0500 Subject: [PATCH 16/20] Update tox.ini Drop support for Python 3.2. No longer supported by coverage and others. --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 39f1c978..0e254846 100644 --- a/tox.ini +++ b/tox.ini @@ -4,7 +4,7 @@ # and then run "tox" from this directory. [tox] -envlist = py26, py27, py32, py33, py34 +envlist = py26, py27, py33, py34 [testenv] changedir = tests From ddec8ab44ab02badb572c2f8ddb935cd076236ac Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Thu, 21 Jan 2016 13:25:58 -0500 Subject: [PATCH 17/20] Update aggregate_tests.py Decrease the verbosity of the test runner so that only test failures are shown. --- tests/aggregate_tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/aggregate_tests.py b/tests/aggregate_tests.py index 12e14e37..f0d911ba 100755 --- a/tests/aggregate_tests.py +++ b/tests/aggregate_tests.py @@ -64,6 +64,6 @@ if __name__ == '__main__': suite = unittest.TestLoader().loadTestsFromNames(tests_without_extension) - all_tests_passed = unittest.TextTestRunner(verbosity=2).run(suite).wasSuccessful() + all_tests_passed = unittest.TextTestRunner(verbosity=1).run(suite).wasSuccessful() if not all_tests_passed: sys.exit(1) From b2177aaaad972dc332e0e1fc426be4e43648263e Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Fri, 22 Jan 2016 10:35:59 -0500 Subject: [PATCH 18/20] Update tox.ini Test and build against Python 3.5 --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 0e254846..2c3cd128 100644 --- a/tox.ini +++ b/tox.ini @@ -4,7 +4,7 @@ # and then run "tox" from this directory. [tox] -envlist = py26, py27, py33, py34 +envlist = py26, py27, py33, py34, py35 [testenv] changedir = tests From c83cc3257802a76a963bb0a4c36316ea004afd72 Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Fri, 22 Jan 2016 10:59:21 -0500 Subject: [PATCH 19/20] Update setup.py * Bump version number to 0.10.0 (pre-release) * Update URL * Remove classifier for Python 3.2 --- setup.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/setup.py b/setup.py index 1b73af6f..c53879df 100755 --- a/setup.py +++ b/setup.py @@ -80,12 +80,12 @@ setup( name = 'tuf', - version = '0.9.9', + version = '0.10.0', description = 'A secure updater framework for Python', long_description = long_description, - author = 'http://www.theupdateframework.com', + author = 'https://www.updateframework.com', author_email = 'theupdateframework@googlegroups.com', - url = 'http://www.theupdateframework.com', + url = 'https://www.updateframework.com', keywords = 'update updater secure authentication key compromise revocation', classifiers = [ 'Development Status :: 4 - Beta', @@ -101,9 +101,9 @@ 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', + 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: Implementation :: CPython', 'Topic :: Security', 'Topic :: Software Development' From 2517bd1341d1f169b40cce13f1dcaa0016419162 Mon Sep 17 00:00:00 2001 From: Vladimir Diaz Date: Fri, 22 Jan 2016 11:09:55 -0500 Subject: [PATCH 20/20] Update MANIFEST.in Remove iso8601 entry --- MANIFEST.in | 1 - 1 file changed, 1 deletion(-) diff --git a/MANIFEST.in b/MANIFEST.in index 82813772..f015228a 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -11,7 +11,6 @@ include tests/repository_data/keystore/targets_key include tests/repository_data/keystore/timestamp_key include tuf/_vendor/ed25519/test_data/ed25519 include tuf/_vendor/ed25519/LICENSE -include tuf/_vendor/iso8601/LICENSE recursive-include docs *.txt recursive-include docs/papers *.pdf