diff --git a/METADATA.md b/METADATA.md new file mode 100644 index 00000000..b1875209 --- /dev/null +++ b/METADATA.md @@ -0,0 +1,81 @@ +# Metadata + +Metadata files provide information that clients can use to make update decisions. Different metadata files provide different information. The various metadata files are signed by different roles. The concept of roles allows TUF to only trust information that a role is trusted to provide. + +The signed metadata files always include the time they were created and their expiration dates. This ensures that outdated metadata will be detected and that clients can refuse to accept metadata older than that which they've already seen. + +All TUF metadata uses a subset of the JSON object format. When calculating the digest of an object, we use the [Canonical JSON](http://wiki.laptop.org/go/Canonical_JSON) format. Implementation-level detail about the metadata can be found in the [spec](docs/tuf-spec.txt). + +There are four required top-level roles and one optional top-level role, each with their own metadata file. + +Required: + +* Root +* Targets +* Release +* Timestamp + +Optional: + +* Mirrors + +There may also be any number of delegated target roles. + +## Root Metadata (root.txt) + +Signed by: Root role. + +Specifies the other top-level roles. When specifying these roles, the trusted keys for each role are listed along with the minimum number of those keys which are required to sign the role's metadata. We call this number the signature threshold. + +See [example](http://mirror1.poly.edu/test-pypi/metadata/root.txt). + +## Targets Metadata (targets.txt) + +Signed by: Targets role. + +The targets.txt metadata file lists hashes and sizes of target files. Target files are the actual files that clients are intending to download (for example, the software updates they are trying to obtain). + +This file can optionally define other roles to which it delegates trust. Delegating trust means that the delegated role is trusted for some or all of the target files available from the repository. When delegated roles are specified, they are specified in a similar way to how the Root role specifies the top-level roles: the trusted keys and signature threshold for each role is given. Additionally, one or more patterns are specified which indicate the target file paths for which clients should trust each delegated role. + +See [example](http://mirror1.poly.edu/test-pypi/metadata/targets.txt). + +## Delegated Targets Metadata (targets/foo.txt) + +Signed by: A delegated targets role. + +The metadata files provided by delegated targets roles follow exactly the same format as the metadata file provided by the top-level Targets role. + +The location of the metadata file for each delegated target role is based on the delegation ancestry of the role. If the top-level Targets role defines a role named foo, then the delegated target role's full name would be targets/foo and its metadata file will be available on the repository at the path targets/foo.txt (this is relative to the base directory from which all metadata is available). This path is just the full name of the role followed by a file extension. + +If this delegated role foo further delegates to a role bar, then the result is a role whose full name is targets/foo/bar and whose signed metadata file is made available on the repository at targets/foo/bar.txt. + +See [example](http://mirror1.poly.edu/test-pypi/metadata/targets/unclaimed.txt). + +## Release Metadata (release.txt) + +Signed by: Release role. + +The release.txt metadata file lists hashes and sizes of all metadata files other than timestamp.txt. This file ensures that clients will see a consistent view of the files on the repository. That is, metadata files (and thus target file) that existed on the repository at different times cannot be combined and presented to clients by an attacker. + +​See [example](http://mirror1.poly.edu/test-pypi/metadata/release.txt). + +## Timestamp Metadata (timestamp.txt) + +Signed by: Timestamp role. + +The timestamp.txt metadata file lists the hashes and size of the release.txt file. This is the first and potentially only file that needs to be downloaded when clients poll for the existence of updates. This file is frequently resigned and has a short expiration date, thus allowing clients to quickly detect if they are being prevented from obtaining the most recent metadata. An online key is generally used to automatically resign this file at regular intervals. + +There are two primary reasons that the timestamp.txt file doesn't contain all of the information that the release.txt file does. + +* The timestamp.txt file is downloaded very frequently and so should be kept as small as possible, especially considering that the release.txt file grows in size in proportion to the number of delegated target roles. +* As the Timestamp role's key is an online key and thus at high risk, separate keys should be used for signing the release.txt metadata file so that the Release role's keys can be kept offline and thus more secure. + +See [example](http://mirror1.poly.edu/test-pypi/metadata/timestamp.txt). + +## Mirrors Metadata (mirrors.txt) + +Optionally signed by: Mirrors role. + +The mirrors.txt file provides an optional way to provide mirror list updates to TUF clients. Mirror lists can alternatively be provided directly by the software update system and obtained in any way the system sees fit, including being hard coded if that is what an applications wants to do. + +No example available. At the time of writing, this hasn't been implemented in TUF. Currently mirrors are specified by the client code. diff --git a/README.md b/README.md index 41007030..5f57b330 100644 --- a/README.md +++ b/README.md @@ -15,9 +15,8 @@ completely new software. Three major classes of software update systems are: -* **Application updaters** which are used by applications use to update -themselves. For example, Firefox updates itself through its own application -updater. +* **Application updaters** which are used by applications to update themselves. +For example, Firefox updates itself through its own application updater. * **Library package managers** such as those offered by many programming languages for installing additional libraries. These are systems such as @@ -30,8 +29,63 @@ YaST are examples of these. ## Our Approach There are literally thousands of different software update systems in common -use today. (In fact the average Windows user has about two dozen different +use today. (In fact the average Windows user has about [two dozen](http://secunia.com/gfx/pdf/Secunia_RSA_Software_Portfolio_Security_Exposure.pdf) different software updaters on their machine!) We are building a library that can be universally (and in most cases transparently) used to secure software update systems. + +## Overview + +At the highest level, TUF simply provides applications with a secure method of obtaining files and knowing when new versions of files are available. We call these files, the ones that are supposed to be downloaded, "target files". The most common need for these abilities is in software update systems and that's what we had in mind when creating TUF. + +On the surface, this all sounds simple. Securely obtaining updates just means: + +* Knowing when an update exists. +* Downloading the updated file. + +The problem is that this is only simple when there are no malicious parties involved. If an attacker is trying to interfere with these seemingly simple steps, there is plenty they can do. + +## Background + +Let's assume you take the approach that most systems do (at least, the ones that even try to be secure). You download both the file you want and a cryptographic signature of the file. You already know which key you trust to make the signature. You check that the signature is correct and was made by this trusted key. All seems well, right? Wrong. You are still at risk in many ways, including: + +* An attacker keeps giving you the same file, so you never realize there is an update. +* An attacker gives you an older, insecure version of a file that you already have, so you download that one and blindly use it thinking it's newer. +* An attacker gives you a newer version of a file you have but it's not the newest one. It's newer to you, but it may be insecure and exploitable by the attacker. +* An attacker compromises the key used to sign these files and now you download a malicious file that is properly signed. + +These are just some of the attacks software update systems are vulnerable to when only using signed files. +See [Security](SECURITY.md) for a full list of attacks and updater weaknesses TUF is designed to prevent. + +The following papers provide detailed information on securing software updater systems, TUF's design and implementation details, attacks on package managers, and package management security: + +* [Survivable Key Compromise in Software Update Systems](docs/papers/survivable-key-compromise-ccs2010.pdf?raw=true) + +* [A Look In the Mirror: Attacks on Package Managers](docs/papers/package-management-security-tr08-02.pdf?raw=true) + +* [Package Management Security](docs/papers/attacks-on-package-managers-ccs2008.pdf?raw=true) + + +##What TUF Does + +In order to securely download and verify target files, TUF requires a few extra files to exist on a repository. These are called metadata files. TUF metadata files contain additional information, including information about which keys are trusted, the cryptographic hashes of files, signatures on the metadata, metadata version numbers, and the date after which the metadata should be considered expired. + +When a software update system using TUF wants to check for updates, it asks TUF to do the work. That is, your software update system never has to deal with this additional metadata or understand what's going on underneath. If TUF reports back that there are updates available, your software update system can then ask TUF to download these files. TUF downloads them and checks them against the TUF metadata that it also downloads from the repository. If the downloaded target files are trustworthy, TUF hands them over to your software update system. +See [Metadata](METADATA.md) for more information and examples. + +TUF specification document is also available: + +* [The Update Framework Specification](docs/tuf-spec.txt?raw=true) + + + +##Using TUF + +TUF has four major classes of users: clients, for whom TUF is largely transparent; mirrors, who will (in most cases) have nothing at all to do with TUF; upstream servers, who will largely be responsible for care and feeding of repositories; and integrators, who do the work of putting TUF into existing projects. + +* [Creating a Repository](tuf/README.md) + +* [Low-level Integration](tuf/client/README.md) + +* [High-level Integration](tuf/interposition/README.md) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..be58249d --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,69 @@ +#Security + +Generally, a software update system is secure if it can be sure that it knows about the latest available updates in a timely manner, any files it downloads are the correct files, and no harm results from checking or downloading files. The details of making this happen are complicated by various attacks that can be carried out against software update systems. + +## Attacks and Weaknesses + +The following are some of the known attacks on software update systems, including weaknesses that make attacks possible. In order to design a secure software update framework, these need to be understood and protected against. Some of these issues are or can be related depending on the design and implementation of a software update system. + +* **Arbitrary software installation**. An attacker installs anything they want on the client system. That is, an attacker can provide arbitrary files in response to download requests and the files will not be detected as illegitimate. + +* **Rollback attacks**. An attacker presents a software update system with older files than those the client has already seen, causing the client to use files older than those the client knows about. + +* **Indefinite freeze attacks**. An attacker continues to present a software update system with the same files the client has already seen. The result is that the client does not know that new files are available. + +* **Endless data attacks**. An attacker responds to a file download request with an endless stream of data, causing harm to clients (e.g. a disk partition filling up or memory exhaustion). + +* **Slow retrieval attacks**. An attacker responds to clients with a very slow stream of data that essentially results in the client never continuing the update process. + +* **Extraneous dependencies attacks**. An attacker indicates to clients that in order to install the software they wanted, they also need to install unrelated software. This unrelated software can be from a trusted source but may have known vulnerabilities that are exploitable by the attacker. + +* **Mix-and-match attacks**. An attacker presents clients with a view of a repository that includes files that never existed together on the repository at the same time. This can result in, for example, outdated versions of dependencies being installed. + +* **Wrong software installation**. An attacker provides a client with a trusted file that is not the one the client wanted. + +* **Malicious mirrors preventing updates**. An attacker in control of one repository mirror is able to prevent users from obtaining updates from other, good mirrors. + +* **Vulnerability to key compromises**. At attacker who is able to compromise a single key or less than a given threshold of keys can compromise clients. This includes relying on a single online key (such as only being protected by SSL) or a single offline key (such as most software update systems use to sign files). + +##Design Concepts + +The design and implementation of TUF aims to be secure against all of the above attacks. A few general ideas drive much of the security of TUF. + +For the details of how TUF conveys the information discussed below, see the [Metadata documentation](METADATA.md). + +## Trust + +Trusting downloaded files really means trusting that the files were provided by some trusted party. Two frequently overlooked aspects of trust in a secure software update system are: + +* Trust should not be granted forever. Trust should expire if it is not renewed. +* Compartmentalized trust. A trusted party should only be trusted for files that it is supposed to provide. + +## Mitigated Key Risk + +Cryptographic signatures are a necessary component in securing a software update system. The safety of the keys that are used to create these signatures affects the security of clients. Rather than incorrectly assume that private keys are always safe from compromise, a secure software update system must strive to keep clients as safe as possible even when compromises happen. + +Keeping clients safe despite dangers to keys involves: + +* Fast and secure key replacement and revocation. +* Minimally trusting keys that are at high risk. Keys that are kept online or used in an automated fashion shouldn't pose immediate risk to clients if compromised. +* Supporting the use of multiple keys with threshold/quorum signatures trust. + +## Integrity + +File integrity is important both with respect to single files as well as collections of files. It's fairly obvious that clients must verify that individual downloaded files are correct. Not as obvious but still very important is the need for clients to be certain that their entire view of a repository is correct. For example, if a trusted party is providing two files, a software update system should see the latest versions of both of those files, not just one of the files and not versions of the two files that were never provided together. + +## Freshness + +As software updates often fix security bugs, it is important that software update systems be able to obtain the latest versions of files that are available. An attacker may want to trick a client into installing outdated versions of software or even just convince a client that no updates are available. + +Ensuring freshness means to: + +* Never accept files older than those that have been seen previously. +* Recognize when there may be a problem obtaining updates. + +Note that it won't always be possible for a client to successfully update if an attacker is responding to their requests. However, a client should be able to recognize that updates may exist that they haven't been able to obtain. + +## Implementation Safety + +In addition to a secure design, TUF also works to be secure against implementation vulnerabilities including those common to software update systems. In some cases this is assisted by the inclusion of additional information in metadata. For example, knowing the expected size of a target file that is to be downloaded allows TUF to limit the amount of data it will download when retrieving the file. As a result, TUF is secure against endless data attacks (discussed above). diff --git a/docs/images/libtuf-diagram.png b/docs/images/libtuf-diagram.png new file mode 100644 index 00000000..730479d2 Binary files /dev/null and b/docs/images/libtuf-diagram.png differ diff --git a/docs/latex/tuf-client-spec.pdf.old b/docs/latex/tuf-client-spec.pdf.old new file mode 100644 index 00000000..4228832d Binary files /dev/null and b/docs/latex/tuf-client-spec.pdf.old differ diff --git a/docs/latex/tuf-client-spec.tex b/docs/latex/tuf-client-spec.tex index a8148e1e..3c58ea64 100644 --- a/docs/latex/tuf-client-spec.tex +++ b/docs/latex/tuf-client-spec.tex @@ -1,8 +1,35 @@ % tuf_client_spec.tex -\documentclass{letter} -\usepackage{listings} +\documentclass{article} \setlength\parindent{0pt} -\lstset{frameshape={ynn}{nny}{nnn}{yyn}, showstringspaces=false, basicstyle=\ttfamily} +\usepackage{listings} +\usepackage{hyperref} +\usepackage{color} +\usepackage{textcomp} +\definecolor{listinggray}{gray}{0.9} +\definecolor{lbcolor}{rgb}{0.9,0.9,0.9} +\lstset{ + backgroundcolor=\color{lbcolor}, + tabsize=4, + rulecolor=, + language=matlab, + basicstyle=\scriptsize, + upquote=true, + aboveskip={1.5\baselineskip}, + columns=fixed, + showstringspaces=false, + extendedchars=true, + breaklines=true, + prebreak = \raisebox{0ex}[0ex][0ex]{\ensuremath{\hookleftarrow}}, + frame=single, + showtabs=false, + showspaces=false, + showstringspaces=false, + identifierstyle=\ttfamily, + keywordstyle=\color[rgb]{0,0,1}, + commentstyle=\color[rgb]{0.133,0.545,0.133}, + stringstyle=\color[rgb]{0.627,0.126,0.941}, +} + \begin{document} %--------------------------------- Header -------------------------------------- @@ -44,7 +71,23 @@ functional TUF system in which to do so. \subsection{Setting up the Repository} You'll need to run the following steps to set the stage: -\lstinputlisting[language=csh]{tufsetup.sh} + +\begin{lstlisting} +#! /bin/sh + +# create the relevant directories +mkdir tufdemo +cd tufdemo +mkdir demorepo +mkdir demoproject + +# add a file to the project +echo "#! /usr/bin/env python" > demoproject/helloworld.py +echo "print 'hello, world!'" >> demoproject/helloworld.py + +# run the quickstart script +quickstart.py -t 1 -k keystore -l demorepo -r demoproject +\end{lstlisting} This will prompt you for a password for your keystore and an expiration date. Choosing your expiration date is something of a balancing act: on the one hand, @@ -72,7 +115,14 @@ TUF isn't designed as a replacement for package managers so it doesn't provide a mechanism with which to perform the initial installation of our demo project's metadata. To do that, open up another terminal and run the following: -\lstinputlisting[language=csh]{tufclientsetup.sh} +\begin{lstlisting} +#! /bin/sh + +mkdir democlient +cp -r demorepo/meta democlient/cur +cp -r democlient/cur democlient/prev +\end{lstlisting} + Once we've installed our metadata, getting the software is a simple matter of running the demonstration client, found with TUF's source at diff --git a/docs/tuf-server-spec.pdf b/docs/latex/tuf-server-spec.pdf.old similarity index 59% rename from docs/tuf-server-spec.pdf rename to docs/latex/tuf-server-spec.pdf.old index e1633ba6..bdb48938 100644 Binary files a/docs/tuf-server-spec.pdf and b/docs/latex/tuf-server-spec.pdf.old differ diff --git a/docs/latex/tuf-server-spec.tex b/docs/latex/tuf-server-spec.tex index 104ec8a2..d9f7c383 100644 --- a/docs/latex/tuf-server-spec.tex +++ b/docs/latex/tuf-server-spec.tex @@ -1,8 +1,35 @@ -% tuf_repo_spec.tex +% tuf_server_spec.tex \documentclass{article} \setlength\parindent{0pt} \usepackage{listings} -\lstset{frameshape={ynn}{nny}{nnn}{yyn}, showstringspaces=false, basicstyle=\ttfamily} +\usepackage{hyperref} +\usepackage{color} +\usepackage{textcomp} +\definecolor{listinggray}{gray}{0.9} +\definecolor{lbcolor}{rgb}{0.9,0.9,0.9} +\lstset{ + backgroundcolor=\color{lbcolor}, + tabsize=4, + rulecolor=, + language=matlab, + basicstyle=\scriptsize, + upquote=true, + aboveskip={1.5\baselineskip}, + columns=fixed, + showstringspaces=false, + extendedchars=true, + breaklines=true, + prebreak = \raisebox{0ex}[0ex][0ex]{\ensuremath{\hookleftarrow}}, + frame=single, + showtabs=false, + showspaces=false, + showstringspaces=false, + identifierstyle=\ttfamily, + keywordstyle=\color[rgb]{0,0,1}, + commentstyle=\color[rgb]{0.133,0.545,0.133}, + stringstyle=\color[rgb]{0.627,0.126,0.941}, +} + \begin{document} %--------------------------------- Header -------------------------------------- diff --git a/docs/tuf-client-spec.pdf b/docs/tuf-client-spec.pdf deleted file mode 100644 index 2ba16599..00000000 Binary files a/docs/tuf-client-spec.pdf and /dev/null differ diff --git a/docs/tuf-spec.txt b/docs/tuf-spec.txt index 7437ade5..6e05e68c 100644 --- a/docs/tuf-spec.txt +++ b/docs/tuf-spec.txt @@ -1,4 +1,4 @@ - TUF: The Update Framework + The Update Framework Specification 1. Introduction @@ -680,9 +680,10 @@ 5.1. The client application - Note: At any point in the following process there is a problem (e.g. only - expired metadata can be retrieved), the software update system using the - framework must decide how to proceed. + Note: If at any point in the following process there is a problem (e.g., only + expired metadata can be retrieved), the Root file is downloaded and the process + starts over. Optionally, the software update system using the framework can + decide how to proceed rather than automatically downloading a new Root file. The client code instructs the framework to check for updates. The framework downloads the timestamp.txt file from a mirror and checks that the file is diff --git a/ed25519/ed25519.py b/ed25519/ed25519.py deleted file mode 100755 index 7f8613b8..00000000 --- a/ed25519/ed25519.py +++ /dev/null @@ -1,104 +0,0 @@ -import hashlib - -b = 256 -q = 2**255 - 19 -l = 2**252 + 27742317777372353535851937790883648493 - -def H(m): - return hashlib.sha512(m).digest() - -def expmod(b,e,m): - if e == 0: return 1 - t = expmod(b,e/2,m)**2 % m - if e & 1: t = (t*b) % m - return t - -def inv(x): - return expmod(x,q-2,q) - -d = -121665 * inv(121666) -I = expmod(2,(q-1)/4,q) - -def xrecover(y): - xx = (y*y-1) * inv(d*y*y+1) - x = expmod(xx,(q+3)/8,q) - if (x*x - xx) % q != 0: x = (x*I) % q - if x % 2 != 0: x = q-x - return x - -By = 4 * inv(5) -Bx = xrecover(By) -B = [Bx % q,By % q] - -def edwards(P,Q): - x1 = P[0] - y1 = P[1] - x2 = Q[0] - y2 = Q[1] - x3 = (x1*y2+x2*y1) * inv(1+d*x1*x2*y1*y2) - y3 = (y1*y2+x1*x2) * inv(1-d*x1*x2*y1*y2) - return [x3 % q,y3 % q] - -def scalarmult(P,e): - if e == 0: return [0,1] - Q = scalarmult(P,e/2) - Q = edwards(Q,Q) - if e & 1: Q = edwards(Q,P) - return Q - -def encodeint(y): - bits = [(y >> i) & 1 for i in range(b)] - return ''.join([chr(sum([bits[i * 8 + j] << j for j in range(8)])) for i in range(b/8)]) - -def encodepoint(P): - x = P[0] - y = P[1] - bits = [(y >> i) & 1 for i in range(b - 1)] + [x & 1] - return ''.join([chr(sum([bits[i * 8 + j] << j for j in range(8)])) for i in range(b/8)]) - -def bit(h,i): - return (ord(h[i/8]) >> (i%8)) & 1 - -def publickey(sk): - h = H(sk) - a = 2**(b-2) + sum(2**i * bit(h,i) for i in range(3,b-2)) - A = scalarmult(B,a) - return encodepoint(A) - -def Hint(m): - h = H(m) - return sum(2**i * bit(h,i) for i in range(2*b)) - -def signature(m,sk,pk): - h = H(sk) - a = 2**(b-2) + sum(2**i * bit(h,i) for i in range(3,b-2)) - r = Hint(''.join([h[i] for i in range(b/8,b/4)]) + m) - R = scalarmult(B,r) - S = (r + Hint(encodepoint(R) + pk + m) * a) % l - return encodepoint(R) + encodeint(S) - -def isoncurve(P): - x = P[0] - y = P[1] - return (-x*x + y*y - 1 - d*x*x*y*y) % q == 0 - -def decodeint(s): - return sum(2**i * bit(s,i) for i in range(0,b)) - -def decodepoint(s): - y = sum(2**i * bit(s,i) for i in range(0,b-1)) - x = xrecover(y) - if x & 1 != bit(s,b-1): x = q-x - P = [x,y] - if not isoncurve(P): raise Exception("decoding point that is not on curve") - return P - -def checkvalid(s,m,pk): - if len(s) != b/4: raise Exception("signature length is wrong") - if len(pk) != b/8: raise Exception("public-key length is wrong") - R = decodepoint(s[0:b/8]) - A = decodepoint(pk) - S = decodeint(s[b/8:b/4]) - h = Hint(encodepoint(R) + pk + m) - if scalarmult(B,S) != edwards(R,scalarmult(A,h)): - raise Exception("signature does not pass verification") diff --git a/setup.py b/setup.py index d035942f..99422bea 100755 --- a/setup.py +++ b/setup.py @@ -69,6 +69,7 @@ url='https://www.updateframework.com', install_requires=['pycrypto>=2.6'], packages=[ + 'ed25519', 'tuf', 'tuf.client', 'tuf.compatibility', @@ -82,6 +83,7 @@ 'tuf/repo/quickstart.py', 'tuf/pushtools/push.py', 'tuf/pushtools/receivetools/receive.py', - 'tuf/repo/signercli.py' + 'tuf/repo/signercli.py', + 'tuf/client/basic_client.py' ] ) diff --git a/tests/unit/test_ed25519_keys.py b/tests/unit/test_ed25519_keys.py new file mode 100755 index 00000000..a65a85dc --- /dev/null +++ b/tests/unit/test_ed25519_keys.py @@ -0,0 +1,123 @@ +""" + + test_ed25519_keys.py + + + Vladimir Diaz + + + October 11, 2013. + + + See LICENSE for licensing information. + + + Test cases for test_ed25519_keys.py. +""" + +import unittest +import logging + +import tuf +import tuf.log +import tuf.formats +import tuf.ed25519_keys as ed25519 + +logger = logging.getLogger('tuf.test_ed25519_keys') + +public, private = ed25519.generate_public_and_private() +FORMAT_ERROR_MSG = 'tuf.FormatError raised. Check object\'s format.' + + +class TestEd25519_keys(unittest.TestCase): + def setUp(self): + pass + + + def test_generate_public_and_private(self): + pub, priv = ed25519.generate_public_and_private() + + # Check format of 'pub' and 'priv'. + self.assertEqual(True, tuf.formats.ED25519PUBLIC_SCHEMA.matches(pub)) + self.assertEqual(True, tuf.formats.ED25519SEED_SCHEMA.matches(priv)) + + # Check for invalid argument. + self.assertRaises(tuf.FormatError, + ed25519.generate_public_and_private, 'True') + + self.assertRaises(tuf.FormatError, + ed25519.generate_public_and_private, 2048) + + + def test_create_signature(self): + global public + global private + data = 'The quick brown fox jumps over the lazy dog' + signature, method = ed25519.create_signature(public, private, data) + + # Verify format of returned values. + self.assertEqual(True, + tuf.formats.ED25519SIGNATURE_SCHEMA.matches(signature)) + + self.assertEqual(True, tuf.formats.NAME_SCHEMA.matches(method)) + self.assertEqual('ed25519', method) + + # Check for improperly formatted argument. + self.assertRaises(tuf.FormatError, + ed25519.create_signature, 123, private, data) + + self.assertRaises(tuf.FormatError, + ed25519.create_signature, public, 123, data) + + # Check for invalid 'data'. + self.assertRaises(tuf.CryptoError, + ed25519.create_signature, public, private, 123) + + + def test_verify_signature(self): + global public + global private + data = 'The quick brown fox jumps over the lazy dog' + signature, method = ed25519.create_signature(public, private, data) + + valid_signature = ed25519.verify_signature(public, method, signature, data) + self.assertEqual(True, valid_signature) + + # Check for improperly formatted arguments. + self.assertRaises(tuf.FormatError, ed25519.verify_signature, 123, method, + signature, data) + + # Signature method improperly formatted. + self.assertRaises(tuf.FormatError, ed25519.verify_signature, public, 123, + signature, data) + + # Signature not a string. + self.assertRaises(tuf.FormatError, ed25519.verify_signature, public, method, + 123, data) + + # Invalid signature length, which must be exactly 64 bytes.. + self.assertRaises(tuf.FormatError, ed25519.verify_signature, public, method, + 'bad_signature', data) + + # Check for invalid signature and data. + # Mismatched data. + self.assertEqual(False, ed25519.verify_signature(public, method, + signature, '123')) + + # Mismatched signature. + bad_signature = 'a'*64 + self.assertEqual(False, ed25519.verify_signature(public, method, + bad_signature, data)) + + # Generated signature created with different data. + new_signature, method = ed25519.create_signature(public, private, + 'mismatched data') + + self.assertEqual(False, ed25519.verify_signature(public, method, + new_signature, data)) + + + +# Run the unit tests. +if __name__ == '__main__': + unittest.main() diff --git a/tests/unit/test_formats.py b/tests/unit/test_formats.py index 1e375ca3..49b8c608 100755 --- a/tests/unit/test_formats.py +++ b/tests/unit/test_formats.py @@ -76,7 +76,7 @@ def test_schemas(self): 'NAME_SCHEMA': (tuf.formats.NAME_SCHEMA, 'Marty McFly'), - 'TOGGLE_SCHEMA': (tuf.formats.TOGGLE_SCHEMA, True), + 'BOOLEAN_SCHEMA': (tuf.formats.BOOLEAN_SCHEMA, True), 'THRESHOLD_SCHEMA': (tuf.formats.THRESHOLD_SCHEMA, 1), diff --git a/tests/unit/test_keydb.py b/tests/unit/test_keydb.py index c7330875..fe57b776 100755 --- a/tests/unit/test_keydb.py +++ b/tests/unit/test_keydb.py @@ -13,7 +13,6 @@ Unit test for 'keydb.py'. - """ import unittest @@ -21,7 +20,7 @@ import tuf import tuf.formats -import tuf.rsa_key +import tuf.keys import tuf.keydb import tuf.log @@ -31,7 +30,7 @@ # Generate the three keys to use in our test cases. KEYS = [] for junk in range(3): - KEYS.append(tuf.rsa_key.generate(2048)) + KEYS.append(tuf.keys.generate_rsa_key(2048)) @@ -89,7 +88,7 @@ def test_get_key(self): - def test_add_rsakey(self): + def test_add_key(self): # Test conditions using valid 'keyid' arguments. rsakey = KEYS[0] keyid = KEYS[0]['keyid'] @@ -97,9 +96,9 @@ def test_add_rsakey(self): keyid2 = KEYS[1]['keyid'] rsakey3 = KEYS[2] keyid3 = KEYS[2]['keyid'] - self.assertEqual(None, tuf.keydb.add_rsakey(rsakey, keyid)) - self.assertEqual(None, tuf.keydb.add_rsakey(rsakey2, keyid2)) - self.assertEqual(None, tuf.keydb.add_rsakey(rsakey3)) + self.assertEqual(None, tuf.keydb.add_key(rsakey, keyid)) + self.assertEqual(None, tuf.keydb.add_key(rsakey2, keyid2)) + self.assertEqual(None, tuf.keydb.add_key(rsakey3)) self.assertEqual(rsakey, tuf.keydb.get_key(keyid)) self.assertEqual(rsakey2, tuf.keydb.get_key(keyid2)) @@ -109,26 +108,26 @@ def test_add_rsakey(self): tuf.keydb.clear_keydb() rsakey3['keytype'] = 'bad_keytype' - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, None, keyid) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, '', keyid) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, ['123'], keyid) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, {'a': 'b'}, keyid) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey, {'keyid': ''}) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey, 123) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey, False) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey, ['keyid']) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey3, keyid3) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, None, keyid) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, '', keyid) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, ['123'], keyid) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, {'a': 'b'}, keyid) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey, {'keyid': ''}) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey, 123) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey, False) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey, ['keyid']) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey3, keyid3) rsakey3['keytype'] = 'rsa' # Test conditions where keyid does not match the rsakey. - self.assertRaises(tuf.Error, tuf.keydb.add_rsakey, rsakey, keyid2) - self.assertRaises(tuf.Error, tuf.keydb.add_rsakey, rsakey2, keyid) + self.assertRaises(tuf.Error, tuf.keydb.add_key, rsakey, keyid2) + self.assertRaises(tuf.Error, tuf.keydb.add_key, rsakey2, keyid) # Test conditions using keyids that have already been added. - tuf.keydb.add_rsakey(rsakey, keyid) - tuf.keydb.add_rsakey(rsakey2, keyid2) - self.assertRaises(tuf.KeyAlreadyExistsError, tuf.keydb.add_rsakey, rsakey) - self.assertRaises(tuf.KeyAlreadyExistsError, tuf.keydb.add_rsakey, rsakey2) + tuf.keydb.add_key(rsakey, keyid) + tuf.keydb.add_key(rsakey2, keyid2) + self.assertRaises(tuf.KeyAlreadyExistsError, tuf.keydb.add_key, rsakey) + self.assertRaises(tuf.KeyAlreadyExistsError, tuf.keydb.add_key, rsakey2) @@ -140,12 +139,13 @@ def test_remove_key(self): keyid2 = KEYS[1]['keyid'] rsakey3 = KEYS[2] keyid3 = KEYS[2]['keyid'] - tuf.keydb.add_rsakey(rsakey, keyid) - tuf.keydb.add_rsakey(rsakey2, keyid2) - tuf.keydb.add_rsakey(rsakey3, keyid3) + tuf.keydb.add_key(rsakey, keyid) + tuf.keydb.add_key(rsakey2, keyid2) + tuf.keydb.add_key(rsakey3, keyid3) self.assertEqual(None, tuf.keydb.remove_key(keyid)) self.assertEqual(None, tuf.keydb.remove_key(keyid2)) + # Ensure the keys were actually removed. self.assertRaises(tuf.UnknownKeyError, tuf.keydb.get_key, keyid) self.assertRaises(tuf.UnknownKeyError, tuf.keydb.get_key, keyid2) diff --git a/tests/unit/test_keys.py b/tests/unit/test_keys.py new file mode 100755 index 00000000..758c1009 --- /dev/null +++ b/tests/unit/test_keys.py @@ -0,0 +1,193 @@ +""" + + test_keys.py + + + Vladimir Diaz + + + October 10, 2013. + + + See LICENSE for licensing information. + + + Test cases for test_keys.py. + TODO: test case for ed25519 key generation and refactor. +""" + +import unittest +import logging + +import tuf +import tuf.log +import tuf.formats +import tuf.keys + +logger = logging.getLogger('tuf.test_keys') + +KEYS = tuf.keys +FORMAT_ERROR_MSG = 'tuf.FormatError was raised! Check object\'s format.' +DATA = 'SOME DATA REQUIRING AUTHENTICITY.' + + +rsakey_dict = KEYS.generate_rsa_key() +temp_key_info_vals = rsakey_dict.values() +temp_key_vals = rsakey_dict['keyval'].values() + + +class TestKeys(unittest.TestCase): + def setUp(self): + rsakey_dict['keytype']=temp_key_info_vals[0] + rsakey_dict['keyid']=temp_key_info_vals[1] + rsakey_dict['keyval']=temp_key_info_vals[2] + rsakey_dict['keyval']['public']=temp_key_vals[0] + rsakey_dict['keyval']['private']=temp_key_vals[1] + + + def test_generate_rsa_key(self): + _rsakey_dict = KEYS.generate_rsa_key() + + # Check if the format of the object returned by generate() corresponds + # to RSAKEY_SCHEMA format. + self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(_rsakey_dict), + FORMAT_ERROR_MSG) + + # Passing a bit value that is <2048 to generate() - should raise + # 'tuf.FormatError'. + self.assertRaises(tuf.FormatError, KEYS.generate_rsa_key, 555) + + # Passing a string instead of integer for a bit value. + self.assertRaises(tuf.FormatError, KEYS.generate_rsa_key, 'bits') + + # NOTE if random bit value >=2048 (not 4096) is passed generate(bits) + # does not raise any errors and returns a valid key. + self.assertTrue(tuf.formats.RSAKEY_SCHEMA.matches(KEYS.generate_rsa_key(2048))) + self.assertTrue(tuf.formats.RSAKEY_SCHEMA.matches(KEYS.generate_rsa_key(4096))) + + + def test_format_keyval_to_metadata(self): + keyvalue = rsakey_dict['keyval'] + keytype = rsakey_dict['keytype'] + key_meta = KEYS.format_keyval_to_metadata(keytype, keyvalue) + + # Check if the format of the object returned by this function corresponds + # to KEY_SCHEMA format. + self.assertEqual(None, + tuf.formats.KEY_SCHEMA.check_match(key_meta), + FORMAT_ERROR_MSG) + key_meta = KEYS.format_keyval_to_metadata(keytype, keyvalue, private=True) + + # Check if the format of the object returned by this function corresponds + # to KEY_SCHEMA format. + self.assertEqual(None, tuf.formats.KEY_SCHEMA.check_match(key_meta), + FORMAT_ERROR_MSG) + + # Supplying a 'bad' keyvalue. + self.assertRaises(tuf.FormatError, KEYS.format_keyval_to_metadata, + 'bad_keytype', keyvalue) + + del keyvalue['public'] + self.assertRaises(tuf.FormatError, KEYS.format_keyval_to_metadata, + keytype, keyvalue) + + + def test_format_metadata_to_key(self): + # Reconfiguring rsakey_dict to conform to KEY_SCHEMA + # i.e. {keytype: 'rsa', keyval: {public: pub_key, private: priv_key}} + #keyid = rsakey_dict['keyid'] + del rsakey_dict['keyid'] + + rsakey_dict_from_meta = KEYS.format_metadata_to_key(rsakey_dict) + + # Check if the format of the object returned by this function corresponds + # to RSAKEY_SCHEMA format. + self.assertEqual(None, + tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict_from_meta), + FORMAT_ERROR_MSG) + + # Supplying a wrong number of arguments. + self.assertRaises(TypeError, KEYS.format_metadata_to_key) + args = (rsakey_dict, rsakey_dict) + self.assertRaises(TypeError, KEYS.format_metadata_to_key, *args) + + # Supplying a malformed argument to the function - should get FormatError + del rsakey_dict['keyval'] + self.assertRaises(tuf.FormatError, KEYS.format_metadata_to_key, + rsakey_dict) + + + def test_helper_get_keyid(self): + keytype = rsakey_dict['keytype'] + keyvalue = rsakey_dict['keyval'] + + # Check format of 'keytype'. + self.assertEqual(None, tuf.formats.KEYTYPE_SCHEMA.check_match(keytype), + FORMAT_ERROR_MSG) + + # Check format of 'keyvalue'. + self.assertEqual(None, tuf.formats.KEYVAL_SCHEMA.check_match(keyvalue), + FORMAT_ERROR_MSG) + + keyid = KEYS._get_keyid(keytype, keyvalue) + + # Check format of 'keyid' - the output of '_get_keyid()' function. + self.assertEqual(None, tuf.formats.KEYID_SCHEMA.check_match(keyid), + FORMAT_ERROR_MSG) + + + def test_create_signature(self): + # Creating a signature for 'DATA'. + signature = KEYS.create_signature(rsakey_dict, DATA) + + # Check format of output. + self.assertEqual(None, + tuf.formats.SIGNATURE_SCHEMA.check_match(signature), + FORMAT_ERROR_MSG) + + # Removing private key from 'rsakey_dict' - should raise a TypeError. + rsakey_dict['keyval']['private'] = '' + + args = (rsakey_dict, DATA) + self.assertRaises(TypeError, KEYS.create_signature, *args) + + # Supplying an incorrect number of arguments. + self.assertRaises(TypeError, KEYS.create_signature) + + + def test_verify_signature(self): + # Creating a signature 'signature' of 'DATA' to be verified. + signature = KEYS.create_signature(rsakey_dict, DATA) + + # Verifying the 'signature' of 'DATA'. + verified = KEYS.verify_signature(rsakey_dict, signature, DATA) + self.assertTrue(verified, "Incorrect signature.") + + # Testing an invalid 'signature'. Same 'signature' is passed, with + # 'DATA' different than the original 'DATA' that was used + # in creating the 'signature'. Function should return 'False'. + + # Modifying 'DATA'. + _DATA = '1111'+DATA+'1111' + + # Verifying the 'signature' of modified '_DATA'. + verified = KEYS.verify_signature(rsakey_dict, signature, _DATA) + self.assertFalse(verified, + 'Returned \'True\' on an incorrect signature.') + + # Modifying 'signature' to pass an incorrect method since only + # 'PyCrypto-PKCS#1 PSS' + # is accepted. + signature['method'] = 'Biff' + + args = (rsakey_dict, signature, DATA) + self.assertRaises(tuf.UnknownMethodError, KEYS.verify_signature, *args) + + # Passing incorrect number of arguments. + self.assertRaises(TypeError, KEYS.verify_signature) + + + +# Run the unit tests. +if __name__ == '__main__': + unittest.main() diff --git a/tests/unit/test_keystore.py b/tests/unit/test_keystore.py index 5eed223a..e1a8c7f2 100755 --- a/tests/unit/test_keystore.py +++ b/tests/unit/test_keystore.py @@ -13,7 +13,6 @@ Unit test for keystore.py. - """ import unittest @@ -25,7 +24,7 @@ import tuf import tuf.repo.keystore -import tuf.rsa_key +import tuf.keys import tuf.formats import tuf.util import tuf.log @@ -56,7 +55,7 @@ for i in range(3): # Populating the original 'RSAKEYS' and 'PASSWDS' lists. - RSAKEYS.append(tuf.rsa_key.generate()) + RSAKEYS.append(tuf.keys.generate_rsa_key()) PASSWDS.append('passwd_'+str(i)) # Saving original copies of 'RSAKEYS' and 'PASSWDS' to temp variables @@ -350,6 +349,7 @@ def tearDownModule(): tuf.repo.keystore.clear_keystore() + # Run the unit tests. if __name__ == '__main__': unittest.main() diff --git a/tests/unit/test_pycrypto_keys.py b/tests/unit/test_pycrypto_keys.py new file mode 100755 index 00000000..f6adf7d0 --- /dev/null +++ b/tests/unit/test_pycrypto_keys.py @@ -0,0 +1,198 @@ +""" + + test_pycrypto_keys.py + + + Vladimir Diaz + + + October 10, 2013. + + + See LICENSE for licensing information. + + + Test cases for test_pycrypto_keys.py. +""" + +import unittest +import logging + +import tuf +import tuf.log +import tuf.formats +import tuf.pycrypto_keys as pycrypto + +logger = logging.getLogger('tuf.test_pycrypto_keys') + +public_rsa, private_rsa = pycrypto.generate_rsa_public_and_private() +FORMAT_ERROR_MSG = 'tuf.FormatError raised. Check object\'s format.' + + +class TestPycrypto_keys(unittest.TestCase): + def setUp(self): + pass + + + def test_generate_rsa_public_and_private(self): + pub, priv = pycrypto.generate_rsa_public_and_private() + + # Check format of 'pub' and 'priv'. + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(pub), + FORMAT_ERROR_MSG) + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(priv), + FORMAT_ERROR_MSG) + + # Check for invalid bits argument. bit >= 2048 and a multiple of 256. + self.assertRaises(tuf.FormatError, + pycrypto.generate_rsa_public_and_private, 1024) + + self.assertRaises(ValueError, + pycrypto.generate_rsa_public_and_private, 2049) + + self.assertRaises(tuf.FormatError, + pycrypto.generate_rsa_public_and_private, '2048') + + + def test_create_rsa_signature(self): + global private_rsa + data = 'The quick brown fox jumps over the lazy dog' + signature, method = pycrypto.create_rsa_signature(private_rsa, data) + + # Verify format of returned values. + self.assertNotEqual(None, signature) + self.assertEqual(None, tuf.formats.NAME_SCHEMA.check_match(method), + FORMAT_ERROR_MSG) + self.assertEqual('RSASSA-PSS', method) + + # Check for improperly formatted argument. + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_signature, 123, data) + + # Check for invalid 'data'. + self.assertRaises(tuf.CryptoError, + pycrypto.create_rsa_signature, private_rsa, 123) + + + def test_verify_rsa_signature(self): + global public_rsa + global private_rsa + data = 'The quick brown fox jumps over the lazy dog' + signature, method = pycrypto.create_rsa_signature(private_rsa, data) + + valid_signature = pycrypto.verify_rsa_signature(signature, method, public_rsa, + data) + self.assertEqual(True, valid_signature) + + # Check for improperly formatted arguments. + self.assertRaises(tuf.FormatError, pycrypto.verify_rsa_signature, signature, + 123, public_rsa, data) + + self.assertRaises(tuf.FormatError, pycrypto.verify_rsa_signature, signature, + method, 123, data) + + self.assertRaises(tuf.FormatError, pycrypto.verify_rsa_signature, 123, method, + public_rsa, data) + + # Check for invalid signature and data. + self.assertRaises(tuf.CryptoError, pycrypto.verify_rsa_signature, signature, + method, public_rsa, 123) + + self.assertEqual(False, pycrypto.verify_rsa_signature(signature, method, + public_rsa, 'mismatched data')) + + mismatched_signature, method = pycrypto.create_rsa_signature(private_rsa, + 'mismatched data') + + self.assertEqual(False, pycrypto.verify_rsa_signature(mismatched_signature, + method, public_rsa, data)) + + + + def test_create_rsa_encrypted_pem(self): + global public_rsa + global private_rsa + passphrase = 'pw' + + # Check format of 'public_rsa'. + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(public_rsa), + FORMAT_ERROR_MSG) + + # Check format of 'passphrase'. + self.assertEqual(None, tuf.formats.PASSWORD_SCHEMA.check_match(passphrase), + FORMAT_ERROR_MSG) + + # Generate the encrypted PEM string of 'public_rsa'. + pem_rsakey = pycrypto.create_rsa_encrypted_pem(private_rsa, passphrase) + + # Check format of 'pem_rsakey'. + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(pem_rsakey), + FORMAT_ERROR_MSG) + + # Check for invalid arguments. + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_encrypted_pem, 1, passphrase) + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_encrypted_pem, private_rsa, ['pw']) + + + def test_create_rsa_public_and_private_from_encrypted_pem(self): + global private_rsa + passphrase = 'pw' + + # Generate the encrypted PEM string of 'private_rsa'. + pem_rsakey = pycrypto.create_rsa_encrypted_pem(private_rsa, passphrase) + + # Check format of 'passphrase'. + self.assertEqual(None, tuf.formats.PASSWORD_SCHEMA.check_match(passphrase), + FORMAT_ERROR_MSG) + + # Decrypt 'pem_rsakey' and verify the decrypted object is properly + # formatted. + public_decrypted, private_decrypted = \ + pycrypto.create_rsa_public_and_private_from_encrypted_pem(pem_rsakey, + passphrase) + self.assertEqual(None, + tuf.formats.PEMRSA_SCHEMA.check_match(public_decrypted), + FORMAT_ERROR_MSG) + + self.assertEqual(None, + tuf.formats.PEMRSA_SCHEMA.check_match(private_decrypted), + FORMAT_ERROR_MSG) + + # Does 'public_decrypted' and 'private_decrypted' match the originals? + self.assertEqual(public_rsa, public_decrypted) + self.assertEqual(private_rsa, private_decrypted) + + # Attempt decryption of 'pem_rsakey' using an incorrect passphrase. + self.assertRaises(tuf.CryptoError, + pycrypto.create_rsa_public_and_private_from_encrypted_pem, + pem_rsakey, 'bad_pw') + + # Check for non-encrypted PEM strings. + # create_rsa_public_and_private_from_encrypted_pem() + # returns a tuple of tuf.formats.PEMRSA_SCHEMA objects if the PEM formatted + # string is not actually encrypted but still a valid PEM string. + pub, priv = pycrypto.create_rsa_public_and_private_from_encrypted_pem( + private_rsa, passphrase) + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(pub), + FORMAT_ERROR_MSG) + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(priv), + FORMAT_ERROR_MSG) + + # Check for invalid arguments. + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_public_and_private_from_encrypted_pem, + 123, passphrase) + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_public_and_private_from_encrypted_pem, + pem_rsakey, ['pw']) + self.assertRaises(tuf.CryptoError, + pycrypto.create_rsa_public_and_private_from_encrypted_pem, + 'invalid_pem', passphrase) + + + +# Run the unit tests. +if __name__ == '__main__': + unittest.main() diff --git a/tests/unit/test_quickstart.py b/tests/unit/test_quickstart.py index 00ddf44e..e1bca8c4 100755 --- a/tests/unit/test_quickstart.py +++ b/tests/unit/test_quickstart.py @@ -18,7 +18,6 @@ Given that all message prompts don't change - this will work pretty well for running quickstart without having to manually enter input to prompts every time you want to run quickstart. - """ import os diff --git a/tests/unit/test_roledb.py b/tests/unit/test_roledb.py index 0b0259b5..cb048854 100755 --- a/tests/unit/test_roledb.py +++ b/tests/unit/test_roledb.py @@ -13,7 +13,6 @@ Unit test for 'roledb.py'. - """ @@ -22,7 +21,7 @@ import tuf import tuf.formats -import tuf.rsa_key +import tuf.keys import tuf.roledb import tuf.log @@ -32,7 +31,7 @@ # Generate the three keys to use in our test cases. KEYS = [] for junk in range(3): - KEYS.append(tuf.rsa_key.generate(2048)) + KEYS.append(tuf.keys.generate_rsa_key(2048)) diff --git a/tests/unit/test_rsa_key.py b/tests/unit/test_rsa_key.py deleted file mode 100755 index 881ee1d7..00000000 --- a/tests/unit/test_rsa_key.py +++ /dev/null @@ -1,258 +0,0 @@ -""" - - test_rsa_key.py - - - Konstantin Andrianov - - - April 24, 2012. - - - See LICENSE for licensing information. - - - Test cases for rsa_key.py. - - - I'm using 'global rsakey_dict' - there is no harm in doing so since - in order to modify the global variable in any method, python requires - explicit indication to modify i.e. declaring 'global' in each method - that modifies the global variable 'rsakey_dict'. - -""" - -import unittest -import logging - -import tuf -import tuf.log -import tuf.formats -import tuf.rsa_key - -logger = logging.getLogger('tuf.test_rsa_key') - -RSA_KEY = tuf.rsa_key -FORMAT_ERROR_MSG = 'tuf.FormatError was raised! Check object\'s format.' -DATA = 'SOME DATA REQUIRING AUTHENTICITY.' - - -rsakey_dict = RSA_KEY.generate() -temp_key_info_vals = rsakey_dict.values() -temp_key_vals = rsakey_dict['keyval'].values() - - -class TestRsa_key(unittest.TestCase): - def setUp(self): - rsakey_dict['keytype']=temp_key_info_vals[0] - rsakey_dict['keyid']=temp_key_info_vals[1] - rsakey_dict['keyval']=temp_key_info_vals[2] - rsakey_dict['keyval']['public']=temp_key_vals[0] - rsakey_dict['keyval']['private']=temp_key_vals[1] - - - def test_generate(self): - _rsakey_dict = RSA_KEY.generate() - - # Check if the format of the object returned by generate() corresponds - # to RSAKEY_SCHEMA format. - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(_rsakey_dict), - FORMAT_ERROR_MSG) - - # Passing a bit value that is <2048 to generate() - should raise - # 'tuf.FormatError'. - self.assertRaises(tuf.FormatError, RSA_KEY.generate, 555) - - # Passing a string instead of integer for a bit value. - self.assertRaises(tuf.FormatError, RSA_KEY.generate, 'bits') - - # NOTE if random bit value >=2048 (not 4096) is passed generate(bits) - # does not raise any errors and returns a valid key. - self.assertTrue(tuf.formats.RSAKEY_SCHEMA.matches(RSA_KEY.generate(2048))) - self.assertTrue(tuf.formats.RSAKEY_SCHEMA.matches(RSA_KEY.generate(4096))) - - def test_create_in_metadata_format(self): - key_value = rsakey_dict['keyval'] - key_meta = RSA_KEY.create_in_metadata_format(key_value) - - # Check if the format of the object returned by this function corresponds - # to KEY_SCHEMA format. - self.assertEqual(None, - tuf.formats.KEY_SCHEMA.check_match(key_meta), - FORMAT_ERROR_MSG) - key_meta = RSA_KEY.create_in_metadata_format(key_value, private=True) - - # Check if the format of the object returned by this function corresponds - # to KEY_SCHEMA format. - self.assertEqual(None, tuf.formats.KEY_SCHEMA.check_match(key_meta), - FORMAT_ERROR_MSG) - - # Supplying a 'bad' key_value. - del key_value['public'] - self.assertRaises(tuf.FormatError, RSA_KEY.create_in_metadata_format, - key_value) - - - def test_create_from_metadata_format(self): - # Reconfiguring rsakey_dict to conform to KEY_SCHEMA - # i.e. {keytype: 'rsa', keyval: {public: pub_key, private: priv_key}} - #keyid = rsakey_dict['keyid'] - del rsakey_dict['keyid'] - - rsakey_dict_from_meta = RSA_KEY.create_from_metadata_format(rsakey_dict) - - # Check if the format of the object returned by this function corresponds - # to RSAKEY_SCHEMA format. - self.assertEqual(None, - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict_from_meta), - FORMAT_ERROR_MSG) - - # Supplying a wrong number of arguments. - self.assertRaises(TypeError, RSA_KEY.create_from_metadata_format) - args = (rsakey_dict, rsakey_dict) - self.assertRaises(TypeError, RSA_KEY.create_from_metadata_format, *args) - - # Supplying a malformed argument to the function - should get FormatError - del rsakey_dict['keyval'] - self.assertRaises(tuf.FormatError, RSA_KEY.create_from_metadata_format, - rsakey_dict) - - - def test_helper_get_keyid(self): - key_value = rsakey_dict['keyval'] - - # Check format of 'key_value'. - self.assertEqual(None, tuf.formats.KEYVAL_SCHEMA.check_match(key_value), - FORMAT_ERROR_MSG) - - keyid = RSA_KEY._get_keyid(key_value) - - # Check format of 'keyid' - the output of '_get_keyid()' function. - self.assertEqual(None, tuf.formats.KEYID_SCHEMA.check_match(keyid), - FORMAT_ERROR_MSG) - - - def test_createsignature(self): - # Creating a signature for 'DATA'. - signature = RSA_KEY.create_signature(rsakey_dict, DATA) - - # Check format of output. - self.assertEqual(None, - tuf.formats.SIGNATURE_SCHEMA.check_match(signature), - FORMAT_ERROR_MSG) - - # Removing private key from 'rsakey_dict' - should raise a TypeError. - rsakey_dict['keyval']['private'] = '' - - args = (rsakey_dict, DATA) - self.assertRaises(TypeError, RSA_KEY.create_signature, *args) - - # Supplying an incorrect number of arguments. - self.assertRaises(TypeError, RSA_KEY.create_signature) - - - def test_verify_signature(self): - # Creating a signature 'signature' of 'DATA' to be verified. - signature = RSA_KEY.create_signature(rsakey_dict, DATA) - - # Verifying the 'signature' of 'DATA'. - verified = RSA_KEY.verify_signature(rsakey_dict, signature, DATA) - self.assertTrue(verified, "Incorrect signature.") - - # Testing an invalid 'signature'. Same 'signature' is passed, with - # 'DATA' different than the original 'DATA' that was used - # in creating the 'signature'. Function should return 'False'. - - # Modifying 'DATA'. - _DATA = '1111'+DATA+'1111' - - # Verifying the 'signature' of modified '_DATA'. - verified = RSA_KEY.verify_signature(rsakey_dict, signature, _DATA) - self.assertFalse(verified, - 'Returned \'True\' on an incorrect signature.') - - # Modifying 'signature' to pass an incorrect method since only - # 'PyCrypto-PKCS#1 PSS' - # is accepted. - signature['method'] = 'Biff' - - args = (rsakey_dict, signature, DATA) - self.assertRaises(tuf.UnknownMethodError, RSA_KEY.verify_signature, *args) - - # Passing incorrect number of arguments. - self.assertRaises(TypeError,RSA_KEY.verify_signature) - - - def test_create_encrypted_pem(self): - passphrase = 'pw' - - # Check format of 'rsakey_dict'. - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict), - FORMAT_ERROR_MSG) - - # Check format of 'passphrase'. - self.assertEqual(None, tuf.formats.PASSWORD_SCHEMA.check_match(passphrase), - FORMAT_ERROR_MSG) - - # Generate the encrypted PEM string of 'rsakey_dict'. - pem_rsakey = tuf.rsa_key.create_encrypted_pem(rsakey_dict, passphrase) - - # Check for invalid arguments. - self.assertRaises(tuf.FormatError, - tuf.rsa_key.create_encrypted_pem, 'Biff', passphrase) - self.assertRaises(tuf.FormatError, - tuf.rsa_key.create_encrypted_pem, rsakey_dict, ['pw']) - - - - def test_create_from_encrypted_pem(self): - passphrase = 'pw' - - # Check format of 'rsakey_dict'. - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict), - FORMAT_ERROR_MSG) - - # Check format of 'passphrase'. - self.assertEqual(None, tuf.formats.PASSWORD_SCHEMA.check_match(passphrase), - FORMAT_ERROR_MSG) - - # Generate the encrypted PEM string of 'rsakey_dict'. - pem_rsakey = tuf.rsa_key.create_encrypted_pem(rsakey_dict, passphrase) - - # Decrypt 'pem_rsakey' and verify the decrypted object is properly - # formatted. - decrypted_rsakey = tuf.rsa_key.create_from_encrypted_pem(pem_rsakey, - passphrase) - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(decrypted_rsakey), - FORMAT_ERROR_MSG) - - # Does 'decrypted_rsakey' match the original 'rsakey_dict'. - self.assertEqual(rsakey_dict, decrypted_rsakey) - - # Attempt decryption of 'pem_rsakey' using an incorrect passphrase. - self.assertRaises(tuf.CryptoError, - tuf.rsa_key.create_from_encrypted_pem, pem_rsakey, - 'bad_pw') - # Check for non-encrypted PEM string. create_from_encrypted_pem()/PyCrypto - # returns a tuf.formats.RSAKEY_SCHEMA object if PEM formatted string is - # not actually encrypted but still a valid PEM string. - non_encrypted_private_key = rsakey_dict['keyval']['private'] - decrypted_non_encrypted = tuf.rsa_key.create_from_encrypted_pem( - non_encrypted_private_key, passphrase) - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match( - decrypted_non_encrypted), FORMAT_ERROR_MSG) - - # Check for invalid arguments. - self.assertRaises(tuf.FormatError, - tuf.rsa_key.create_from_encrypted_pem, 123, passphrase) - self.assertRaises(tuf.FormatError, - tuf.rsa_key.create_from_encrypted_pem, pem_rsakey, ['pw']) - self.assertRaises(tuf.CryptoError, - tuf.rsa_key.create_from_encrypted_pem, 'invalid_pem', - passphrase) - - - -# Run the unit tests. -if __name__ == '__main__': - unittest.main() diff --git a/tests/unit/test_sig.py b/tests/unit/test_sig.py index efdbe516..db8fe579 100755 --- a/tests/unit/test_sig.py +++ b/tests/unit/test_sig.py @@ -14,10 +14,8 @@ Test cases for for sig.py. - """ - import unittest import logging @@ -26,7 +24,7 @@ import tuf.formats import tuf.keydb import tuf.roledb -import tuf.rsa_key +import tuf.keys import tuf.sig logger = logging.getLogger('tuf.test_sig') @@ -34,7 +32,7 @@ # Setup the keys to use in our test cases. KEYS = [] for _ in range(3): - KEYS.append(tuf.rsa_key.generate(2048)) + KEYS.append(tuf.keys.generate_rsa_key(2048)) @@ -55,7 +53,7 @@ def test_get_signature_status_no_role(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[0])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) # No specific role we're considering. sig_status = tuf.sig.get_signature_status(signable, None) @@ -82,7 +80,7 @@ def test_get_signature_status_bad_sig(self): signable['signed'], KEYS[0])) signable['signed'] += 'signature no longer matches signed data' - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid']], threshold) @@ -112,7 +110,7 @@ def test_get_signature_status_unknown_method(self): signable['signed'], KEYS[0])) signable['signatures'][0]['method'] = 'fake-sig-method' - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid']], threshold) @@ -142,7 +140,7 @@ def test_get_signature_status_single_key(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[0])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid']], threshold) @@ -171,7 +169,7 @@ def test_get_signature_status_below_threshold(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[0])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 2 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid'], @@ -205,8 +203,8 @@ def test_get_signature_status_below_threshold_unrecognized_sigs(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[2])) - tuf.keydb.add_rsakey(KEYS[0]) - tuf.keydb.add_rsakey(KEYS[1]) + tuf.keydb.add_key(KEYS[0]) + tuf.keydb.add_key(KEYS[1]) threshold = 2 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid'], @@ -242,8 +240,8 @@ def test_get_signature_status_below_threshold_unauthorized_sigs(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[1])) - tuf.keydb.add_rsakey(KEYS[0]) - tuf.keydb.add_rsakey(KEYS[1]) + tuf.keydb.add_key(KEYS[0]) + tuf.keydb.add_key(KEYS[1]) threshold = 2 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid'], KEYS[2]['keyid']], threshold) @@ -278,7 +276,7 @@ def test_check_signatures_no_role(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[0])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) # No specific role we're considering. It's invalid to use the # function tuf.sig.verify() without a role specified because @@ -295,7 +293,7 @@ def test_verify_single_key(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[0])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid']], threshold) @@ -321,8 +319,8 @@ def test_verify_unrecognized_sig(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[2])) - tuf.keydb.add_rsakey(KEYS[0]) - tuf.keydb.add_rsakey(KEYS[1]) + tuf.keydb.add_key(KEYS[0]) + tuf.keydb.add_key(KEYS[1]) threshold = 2 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid'], KEYS[1]['keyid']], threshold) @@ -363,7 +361,7 @@ def test_may_need_new_keys(self): signable['signatures'].append(tuf.sig.generate_rsa_signature( signable['signed'], KEYS[0])) - tuf.keydb.add_rsakey(KEYS[1]) + tuf.keydb.add_key(KEYS[1]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[1]['keyid']], threshold) diff --git a/tuf/README.md b/tuf/README.md new file mode 100644 index 00000000..961df47e --- /dev/null +++ b/tuf/README.md @@ -0,0 +1,349 @@ +#Repository Management +![Repo Tools Diagram 1](https://raw.github.com/theupdateframework/tuf/repository-tools/docs/images/libtuf-diagram.png) +## Create TUF Repository + +The **tuf.libtuf** module can be used to create a TUF repository. It may either be imported into a Python module +or used interactively in a Python interpreter. + +```Bash +$ python +Python 2.7.3 (default, Sep 26 2013, 20:08:41) +[GCC 4.6.3] on linux2 +Type "help", "copyright", "credits" or "license" for more information. +>>> from tuf.libtuf import * +>>> repository = load_repository("path/to/repository") +``` +The **tuf.interposition** package and **tuf.client.updater** module assist in integrating TUF with a software updater. + + +### Keys + +#### Create RSA Keys +```python +from tuf.libtuf import * + +# Generate and write the first of two root keys for the TUF repository. +# The following function creates an RSA key pair, where the private key is saved to +# "path/to/root_key" and the public key to "path/to/root_key.pub". +generate_and_write_rsa_keypair("path/to/root_key", bits=2048, password="password") + +# If the key length is unspecified, it defaults to 3072 bits. A length of less +# than 2048 bits raises an exception. A password may be supplied as an +# argument, otherwise a user prompt is presented. +generate_and_write_rsa_keypair("path/to/root_key2") +Enter a password for the RSA key: +Confirm: +``` +The following four key files should now exist: + +1. root_key +2. root_key.pub +3. root_key2 +4. root_key2.pub + +### Import RSA Keys +```python +from tuf.libtuf import * + +# Import an existing public key. +public_root_key = import_rsa_publickey_from_file("path/to/root_key.pub") + +# Import an existing private key. Importing a private key requires a password, whereas +# importing a public key does not. +private_root_key = import_rsa_privatekey_from_file("path/to/root_key") +Enter a password for the encrypted RSA key: +``` +import_rsa_privatekey_from_file() raises a "tuf.CryptoError" exception if the key/password +is invalid. + +### Create and Import ED25519 Keys +```Python +from tuf.libtuf import * + +# Generate and write an ed25519 key pair. The private key is saved encrypted. +# A 'password' argument may be supplied, otherwise a prompt is presented. +generate_and_write_ed25519_keypair('path/to/ed25519_key') +Enter a password for the ED25519 key: +Confirm: + +# Import the ed25519 public key just created . . . +public_ed25519_key = import_ed25519_publickey_from_file('path/to/ed25519_key.pub') + +# and its corresponding private key. +private_ed25519_key = import_ed25519_privatekey_from_file('path/to/ed25519_key') +Enter a password for the encrypted ED25519 key: +``` + +### Create a new Repository + +#### Create Root +```python +# Continuing from the previous section . . . + +# Create a new Repository object that holds the file path to the repository and the four +# top-level role objects (Root, Targets, Release, Timestamp). Metadata files are created when +# repository.write() is called. The repository directory is created if it does not exist. +repository = create_new_repository("path/to/repository/") + +# The Repository instance, 'repository', initially contains top-level Metadata objects. +# Add one of the public keys, created in the previous section, to the root role. Metadata is +# considered valid if it is signed by the public key's corresponding private key. +repository.root.add_key(public_root_key) + +# Role keys (i.e., the key's keyid) may be queried. Other attributes include: signing_keys, version, +# signatures, expiration, threshold, delegations (Targets role), and compressions. +repository.root.keys +['b23514431a53676595922e955c2d547293da4a7917e3ca243a175e72bbf718df'] + +# Add a second public key to the root role. Although previously generated and saved to a file, +# the second public key must be imported before it can added to a role. +public_root_key2 = import_rsa_publickey_from_file("path/to/root_key2.pub") +repository.root.add_key(public_root_key2) + +# Threshold of each role defaults to 1. Users may change the threshold value, but libtuf.py +# validates thresholds and warns users. Set the threshold of the root role to 2, +# which means the root metadata file is considered valid if it contains at least two valid +# signatures. +repository.root.threshold = 2 +private_root_key2 = import_rsa_privatekey_from_file("path/to/root_key2", password="password") + +# Load the root signing keys to the repository, which write() uses to sign the root metadata. +# The load_signing_key() method SHOULD warn when the key is NOT explicitly allowed to +# sign for it. +repository.root.load_signing_key(private_root_key) +repository.root.load_signing_key(private_root_key2) + +# Print the number of valid signatures and public/private keys of the repository's metadata. +repository.status() +'root' role contains 2 / 2 signatures. +'targets' role contains 0 / 1 public keys. + +try: + repository.write() + +# An exception is raised here by write() because the other top-level roles (targets, release, +# and timestamp) have not been configured with keys. Another option is to call +# repository.write_partial() and generate metadata that may contain an invalid threshold of +# signatures, required public keys, etc. write_partial() allows multiple repository maintainers to +# independently sign metadata and generate them separately. load_repository() can load partially +# written metadata.q +except tuf.Error, e: + print e +Not enough signatures for 'path/to/repository/metadata.staged/targets.txt' + +# In the next section, update the other top-level roles and create a repository with valid metadata. +``` + +#### Create Timestamp, Release, Targets + +```python +# Continuing from the previous section . . . + +# Generate keys for the remaining top-level roles. The root keys have been set above. +# The password argument may be omitted if a password prompt is needed. +generate_and_write_rsa_keypair("path/to/targets_key", password="password") +generate_and_write_rsa_keypair("path/to/release_key", password="password") +generate_and_write_rsa_keypair("path/to/timestamp_key", password="password") + +# Add the public keys of the remaining top-level roles. +repository.targets.add_key(import_rsa_publickey_from_file("path/to/targets_key.pub")) +repository.release.add_key(import_rsa_publickey_from_file("path/to/release_key.pub")) +repository.timestamp.add_key(import_rsa_publickey_from_file("path/to/timestamp_key.pub")) + +# Import the signing keys of the remaining top-level roles. Prompt for passwords. +private_targets_key = import_rsa_privatekey_from_file("path/to/targets_key") +Enter a password for the encrypted RSA key: + +private_release_key = import_rsa_privatekey_from_file("path/to/release_key") +Enter a password for the encrypted RSA key: + +private_timestamp_key = import_rsa_privatekey_from_file("path/to/timestamp_key") +Enter a password for the encrypted RSA key: + +# Load the signing keys of the remaining roles so that valid signatures are generated when +# repository.write() is called. +repository.targets.load_signing_key(private_targets_key) +repository.release.load_signing_key(private_release_key) +repository.timestamp.load_signing_key(private_timestamp_key) + +# Optionally set the expiration date of the timestamp role. By default, roles are set to expire +# as follows: root(1 year), targets(3 months), release(1 week), timestamp(1 day). +repository.timestamp.expiration = "2014-10-28 12:08:00" + +# Metadata files may also be compressed. Only "gz" is currently supported. +repository.targets.compressions = ["gz"] +repository.release.compressions = ["gz"] + +# Write all metadata to "path/to/repository/metadata.staged/". The common case is to crawl the +# filesystem for all delegated roles in "path/to/repository/metadata.staged/targets/". +repository.write() +``` + +### Targets + +#### Add Target Files +```Bash +# Create and save target files to the targets directory of the repository. +$ cd path/to/repository/targets/ +$ echo 'file1' > file1.txt +$ echo 'file2' > file2.txt +$ echo 'file3' > file3.txt +$ mkdir django; echo 'file4' > django/file4.txt +``` + +```python +from tuf.libtuf import * + +# Load the repository created in the previous section. This repository so far contains metadata for +# the top-level roles, but no targets. +repository = load_repository("path/to/repository/") + +# Get a list of file paths in a directory, even those in sub-directories. +# This must be relative to an existing directory in the repository, raise an exception. +list_of_targets = repository.get_filepaths_in_directory("path/to/repository/targets/", + recursive_walk=False, followlinks=True) + +# Add the list of target paths to the metadata of the Targets role. Any target file paths +# that may already exist are NOT replaced. add_targets() does not create or move target files. +repository.targets.add_targets(list_of_targets) + +# Individual target files may also be added. +repository.targets.add_target("path/to/repository/targets/file3.txt") + +# The private key of the updated targets metadata must be loaded before it can be signed and +# written (Note the load_repository() call above). +private_targets_key = import_rsa_privatekey_from_file("path/to/targets_key") +Enter a password for the encrypted RSA key: + +repository.targets.load_signing_key(private_targets_key) + +# Due to the load_repository(), we must also load the private keys of the other top-level roles +# to generate a valid set of metadata. +private_root_key = import_rsa_privatekey_from_file("path/to/root_key") +Enter a password for the encrypted RSA key: + +private_root_key2 = import_rsa_privatekey_from_file("path/to/root_key2") +Enter a password for the encrypted RSA key: + +private_release_key = import_rsa_privatekey_from_file("path/to/release_key") +Enter a password for the encrypted RSA key: + +private_timestamp_key = import_rsa_privatekey_from_file("path/to/timestamp_key") +Enter a password for the encrypted RSA key: + +repository.root.load_signing_key(private_root_key) +repository.root.load_signing_key(private_root_key2) +repository.release.load_signing_key(private_release_key) +repository.timestamp.load_signing_key(private_timestamp_key) + +# Generate new versions of all the top-level metadata. +repository.write() +``` + +#### Remove Target Files +```python +# Continuing from the previous section . . . + +# Remove a target file listed in the "targets" metadata. The target file is not actually deleted +# from the file system. +repository.targets.remove_target("path/to/repository/targets/file3.txt") + +# repository.write() creates any new metadata files, updates those that have changed, and any that +# need updating to make a new "release" (new release.txt and timestamp.txt). +repository.write() +``` + +### Delegations +```python +# Continuing from the previous section . . . + +# Generate a key for a new delegated role named "unclaimed". +generate_and_write_rsa_keypair("path/to/unclaimed_key", bits=2048, password="password") +public_unclaimed_key = import_rsa_publickey_from_file("path/to/unclaimed_key.pub") + +# Make a delegation from "targets" to "targets/unclaimed", initially containing zero targets. +# The delegated role’s full name is not expected. +# delegate(rolename, list_of_public_keys, list_of_file_paths, threshold, +# restricted_paths, path_hash_prefixes) +repository.targets.delegate("unclaimed", [public_unclaimed_key], []) + +# Load the private key of "targets/unclaimed" so that signatures are later added and valid +# metadata is created. +private_unclaimed_key = import_rsa_privatekey_from_file("path/to/unclaimed_key") +Enter a password for the encrypted RSA key: + +repository.targets.unclaimed.load_signing_key(private_unclaimed_key) + +# Update an attribute of the unclaimed role. +repository.targets('unclaimed').version = 2 + +# Delegations may also be nested. Create the delegated role "targets/unclaimed/django", +# where it initially contains zero targets and future targets are restricted to a +# particular directory. +repository.targets('unclaimed').delegate("django", [public_unclaimed_key], [], + restricted_paths=["path/to/repository/targets/django/"]) +repository.targets('unclaimed')('django').load_signing_key(private_unclaimed_key) +repository.targets('unclaimed')('django').add_target("path/to/repository/targets/django/file4.txt") +repository.targets('unclaimed')('django').compressions = ["gz"] + +# Write the metadata of "targets/unclaimed", "targets/unclaimed/django", root, targets, release, +# and timestamp. +repository.write() +``` + +#### Revoke Delegated Role +```python +# Continuing from the previous section . . . + +# Create a delegated role that will be revoked in the next step. +repository.targets('unclaimed').delegate("flask", [public_unclaimed_key], []) + +# Revoke "targets/unclaimed/flask" and write the metadata of all remaining roles. +repository.targets('unclaimed').revoke("flask") +repository.write() +``` + +```bash +# Copy the staged metadata directory changes to the live repository. +$ cp -r "path/to/repository/metadata.staged/" "path/to/repository/metadata/" +``` + +## Client Setup and Repository Trial + +### Using TUF Within an Example Client Updater +```python +from tuf.libtuf import * + +# The following function creates a directory structure that a client +# downloading new software using TUF (via tuf/client/updater.py) will expect. +# The root.txt metadata file must exist, and also the directories that hold the metadata files +# downloaded from a repository. Software updaters integrating with TUF may use this +# directory to store TUF updates saved on the client side. create_tuf_client_directory() +# moves metadata from "path/to/repository/metadata" to "path/to/client/". The repository +# in "path/to/repository/" is the repository created in the "Create TUF Repository" section. +create_tuf_client_directory("path/to/repository/", "path/to/client/") +``` + +#### Test TUF Locally +```Bash +# Run the local TUF repository server. +$ cd "path/to/repository/"; python -m SimpleHTTPServer 8001 + +# Retrieve targets from the TUF repository and save them to "path/to/client/". The +# basic_client.py module is available in "tuf/client/". +# In a different command-line prompt . . . +$ cd "path/to/client/" +$ ls +metadata/ + +$ basic_client.py --repo http://localhost:8001 +$ ls . targets/ targets/django/ +.: +metadata targets tuf.log + +targets/: +django file1.txt file2.txt + +targets/django/: +file4.txt +``` diff --git a/tuf/__init__.py b/tuf/__init__.py index 7fa5e4af..14a61f6c 100755 --- a/tuf/__init__.py +++ b/tuf/__init__.py @@ -18,7 +18,6 @@ The names chosen for TUF Exception classes should end in 'Error' except where there is a good reason not to, and provide that reason in those cases. - """ import urlparse @@ -117,6 +116,14 @@ class RepositoryError(Error): +class InsufficientKeysError(Error): + """Indicate that metadata role lacks a threshold of pubic or private keys.""" + pass + + + + + class ForbiddenTargetError(RepositoryError): """Indicate that a role signed for a target that it was not delegated to.""" pass @@ -165,7 +172,7 @@ class CryptoError(Error): class BadSignatureError(CryptoError): - """Indicate that some metadata file had a bad signature.""" + """Indicate that some metadata file has a bad signature.""" def __init__(self, metadata_role_name): self.metadata_role_name = metadata_role_name @@ -285,6 +292,13 @@ class InvalidNameError(Error): +class UnsignedMetadataError(Error): + """Indicate metadata object with insufficient threshold of signatures.""" + + + + + class NoWorkingMirrorError(Error): """An updater will throw this exception in case it could not download a metadata or target file. diff --git a/ed25519/__init__.py b/tuf/_vendor/__init__.py old mode 100755 new mode 100644 similarity index 100% rename from ed25519/__init__.py rename to tuf/_vendor/__init__.py diff --git a/tuf/_vendor/ed25519/.gitignore b/tuf/_vendor/ed25519/.gitignore new file mode 100644 index 00000000..0d20b648 --- /dev/null +++ b/tuf/_vendor/ed25519/.gitignore @@ -0,0 +1 @@ +*.pyc diff --git a/tuf/_vendor/ed25519/.travis.yml b/tuf/_vendor/ed25519/.travis.yml new file mode 100644 index 00000000..8518ca28 --- /dev/null +++ b/tuf/_vendor/ed25519/.travis.yml @@ -0,0 +1,28 @@ +language: python +python: 2.7 +env: + - TOXENV=py26 + - TOXENV=py27 + #- TOXENV=py32 + #- TOXENV=py33 + - TOXENV=pypy + +install: + # Add the PyPy repository + - "if [[ $TOXENV == 'pypy' ]]; then sudo add-apt-repository -y ppa:pypy/ppa; fi" + # Upgrade PyPy + - "if [[ $TOXENV == 'pypy' ]]; then sudo apt-get -y install pypy; fi" + # This is required because we need to get rid of the Travis installed PyPy + # or it'll take precedence over the PPA installed one. + - "if [[ $TOXENV == 'pypy' ]]; then sudo rm -rf /usr/local/pypy/bin; fi" + - pip install tox + +script: + - tox + +notifications: + irc: + channels: + - "irc.freenode.org#cryptography-dev" + use_notice: true + skip_join: true diff --git a/tuf/_vendor/ed25519/__init__.py b/tuf/_vendor/ed25519/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ed25519/checkparams.py b/tuf/_vendor/ed25519/checkparams.py similarity index 100% rename from ed25519/checkparams.py rename to tuf/_vendor/ed25519/checkparams.py diff --git a/tuf/_vendor/ed25519/ed25519.py b/tuf/_vendor/ed25519/ed25519.py new file mode 100644 index 00000000..b7e9ff04 --- /dev/null +++ b/tuf/_vendor/ed25519/ed25519.py @@ -0,0 +1,161 @@ +import hashlib + + +b = 256 +q = 2 ** 255 - 19 +l = 2 ** 252 + 27742317777372353535851937790883648493 + + +def H(m): + return hashlib.sha512(m).digest() + + +def pow2(x, p): + """== pow(x, 2**p, q)""" + while p > 0: + x = x * x % q + p -= 1 + return x + +def inv(z): + """$= z^{-1} \mod q$, for z != 0""" + # Adapted from curve25519_athlon.c in djb's Curve25519. + z2 = z * z % q # 2 + z9 = pow2(z2, 2) * z % q # 9 + z11 = z9 * z2 % q # 11 + z2_5_0 = (z11*z11)%q * z9 % q # 31 == 2^5 - 2^0 + z2_10_0 = pow2(z2_5_0, 5) * z2_5_0 % q # 2^10 - 2^0 + z2_20_0 = pow2(z2_10_0, 10) * z2_10_0 % q # ... + z2_40_0 = pow2(z2_20_0, 20) * z2_20_0 % q + z2_50_0 = pow2(z2_40_0, 10) * z2_10_0 % q + z2_100_0 = pow2(z2_50_0, 50) * z2_50_0 % q + z2_200_0 = pow2(z2_100_0, 100) * z2_100_0 % q + z2_250_0 = pow2(z2_200_0, 50) * z2_50_0 % q # 2^250 - 2^0 + return pow2(z2_250_0, 5) * z11 % q # 2^255 - 2^5 + 11 = q - 2 + + +d = -121665 * inv(121666) +I = pow(2, (q - 1) / 4, q) + + +def xrecover(y): + xx = (y * y - 1) * inv(d * y * y + 1) + x = pow(xx, (q + 3) / 8, q) + + if (x * x - xx) % q != 0: + x = (x * I) % q + + if x % 2 != 0: + x = q-x + + return x + + +By = 4 * inv(5) +Bx = xrecover(By) +B = (Bx % q, By % q) + + +def edwards(P, Q): + x1, y1 = P + x2, y2 = Q + x3 = (x1 * y2 + x2 * y1) * inv(1 + d * x1 * x2 * y1 * y2) + y3 = (y1 * y2 + x1 * x2) * inv(1 - d * x1 * x2 * y1 * y2) + + return (x3 % q, y3 % q) + + +def scalarmult(P, e): + if e == 0: + return (0, 1) + + Q = scalarmult(P, e / 2) + Q = edwards(Q, Q) + + if e & 1: + Q = edwards(Q, P) + + return Q + + +def encodeint(y): + bits = [(y >> i) & 1 for i in range(b)] + return ''.join([ + chr(sum([bits[i * 8 + j] << j for j in range(8)])) + for i in range(b/8) + ]) + + +def encodepoint(P): + x = P[0] + y = P[1] + bits = [(y >> i) & 1 for i in range(b - 1)] + [x & 1] + return ''.join([ + chr(sum([bits[i * 8 + j] << j for j in range(8)])) + for i in range(b/8) + ]) + + +def bit(h, i): + return (ord(h[i / 8]) >> (i % 8)) & 1 + + +def publickey(sk): + h = H(sk) + a = 2 ** (b - 2) + sum(2 ** i * bit(h, i) for i in range(3, b - 2)) + A = scalarmult(B, a) + return encodepoint(A) + + +def Hint(m): + h = H(m) + return sum(2 ** i * bit(h, i) for i in range(2 * b)) + + +def signature(m, sk, pk): + h = H(sk) + a = 2 ** (b - 2) + sum(2 ** i * bit(h, i) for i in range(3, b - 2)) + r = Hint(''.join([h[j] for j in range(b / 8, b / 4)]) + m) + R = scalarmult(B, r) + S = (r + Hint(encodepoint(R) + pk + m) * a) % l + return encodepoint(R) + encodeint(S) + + +def isoncurve(P): + x, y = P + return (-x * x + y * y - 1 - d * x * x * y * y) % q == 0 + + +def decodeint(s): + return sum(2 ** i * bit(s, i) for i in range(0, b)) + + +def decodepoint(s): + y = sum(2 ** i * bit(s, i) for i in range(0, b - 1)) + x = xrecover(y) + + if x & 1 != bit(s, b-1): + x = q-x + + P = (x, y) + + if not isoncurve(P): + raise Exception("decoding point that is not on curve") + + return P + + +def checkvalid(s, m, pk): + if len(s) != b / 4: + raise Exception("signature length is wrong") + + if len(pk) != b / 8: + raise Exception("public-key length is wrong") + + R = decodepoint(s[:b / 8]) + A = decodepoint(pk) + S = decodeint(s[b / 8:b / 4]) + h = Hint(encodepoint(R) + pk + m) + + if scalarmult(B, S) != edwards(R, scalarmult(A, h)): + raise Exception("signature does not pass verification") diff --git a/tuf/_vendor/ed25519/runtests.sh b/tuf/_vendor/ed25519/runtests.sh new file mode 100755 index 00000000..4dec6743 --- /dev/null +++ b/tuf/_vendor/ed25519/runtests.sh @@ -0,0 +1,8 @@ +#!/bin/sh +set -e + +python -u signfast.py < sign.input + +if [[ $TEST == 'slow' ]]; then + python -u sign.py < sign.input +fi diff --git a/tuf/_vendor/ed25519/science.py b/tuf/_vendor/ed25519/science.py new file mode 100644 index 00000000..18a2fcc2 --- /dev/null +++ b/tuf/_vendor/ed25519/science.py @@ -0,0 +1,32 @@ +import os +import timeit + +import ed25519 + + +seed = os.urandom(32) + +data = "The quick brown fox jumps over the lazy dog" +private_key = seed +public_key = ed25519.publickey(seed) +signature = ed25519.signature(data, private_key, public_key) + + +print('Time generate') +print(timeit.timeit("ed25519.publickey(seed)", + setup="from __main__ import ed25519, seed", + number=10, +)) + +print('\nTime create signature') +print(timeit.timeit("ed25519.signature(data, private_key, public_key)", + setup="from __main__ import ed25519, data, private_key, public_key", + number=10, +)) + + +print('\nTime verify signature') +print(timeit.timeit("ed25519.checkvalid(signature, data, public_key)", + setup="from __main__ import ed25519, signature, data, public_key", + number=10, +)) diff --git a/tuf/_vendor/ed25519/setup.py b/tuf/_vendor/ed25519/setup.py new file mode 100644 index 00000000..d65e3d2a --- /dev/null +++ b/tuf/_vendor/ed25519/setup.py @@ -0,0 +1,11 @@ +from setuptools import setup + + +setup( + name="ed25519", + version="1.0", + + py_modules="ed25519", + + zip_safe=False, +) diff --git a/ed25519/sign.input b/tuf/_vendor/ed25519/sign.input similarity index 100% rename from ed25519/sign.input rename to tuf/_vendor/ed25519/sign.input diff --git a/ed25519/sign.py b/tuf/_vendor/ed25519/sign.py similarity index 94% rename from ed25519/sign.py rename to tuf/_vendor/ed25519/sign.py index be099ad2..18eea684 100644 --- a/ed25519/sign.py +++ b/tuf/_vendor/ed25519/sign.py @@ -1,3 +1,5 @@ +from __future__ import print_function + import sys import binascii import ed25519 @@ -16,6 +18,7 @@ while 1: line = sys.stdin.readline() if not line: break + print(".", end="") x = line.split(':') sk = binascii.unhexlify(x[0][0:64]) pk = ed25519.publickey(sk) diff --git a/tuf/_vendor/ed25519/signfast.py b/tuf/_vendor/ed25519/signfast.py new file mode 100644 index 00000000..630c92ca --- /dev/null +++ b/tuf/_vendor/ed25519/signfast.py @@ -0,0 +1,48 @@ +from __future__ import print_function + +import sys +import binascii +import ed25519 + +# examples of inputs: see sign.input +# should produce no output: python sign.py < sign.input + +# warning: currently 37 seconds/line on a fast machine + +# fields on each input line: sk, pk, m, sm +# each field hex +# each field colon-terminated +# sk includes pk at end +# sm includes m at end + +MAX = 10 + +i = 0 +while 1: + if i >= MAX: + break + i += 1 + line = sys.stdin.readline() + if not line: break + print(".", end="") + x = line.split(':') + sk = binascii.unhexlify(x[0][0:64]) + pk = ed25519.publickey(sk) + m = binascii.unhexlify(x[2]) + s = ed25519.signature(m,sk,pk) + ed25519.checkvalid(s,m,pk) + forgedsuccess = 0 + try: + if len(m) == 0: + forgedm = "x" + else: + forgedmlen = len(m) + forgedm = ''.join([chr(ord(m[i])+(i==forgedmlen-1)) for i in range(forgedmlen)]) + ed25519.checkvalid(s,forgedm,pk) + forgedsuccess = 1 + except: + pass + assert not forgedsuccess + assert x[0] == binascii.hexlify(sk + pk) + assert x[1] == binascii.hexlify(pk) + assert x[3] == binascii.hexlify(s + m) diff --git a/tuf/_vendor/ed25519/tox.ini b/tuf/_vendor/ed25519/tox.ini new file mode 100644 index 00000000..1fdba166 --- /dev/null +++ b/tuf/_vendor/ed25519/tox.ini @@ -0,0 +1,5 @@ +[tox] +envlist = py26,py27,pypy,py32,py33 + +[testenv] +commands = ./runtests.sh diff --git a/tuf/client/README.md b/tuf/client/README.md new file mode 100644 index 00000000..3fbf93b6 --- /dev/null +++ b/tuf/client/README.md @@ -0,0 +1,172 @@ +#updater.py +**updater.py** is intended as the only TUF module that software update +systems need to utilize for a low-level integration. It provides a single +class representing an updater that includes methods to download, install, and +verify metadata or target files in a secure manner. Importing +**tuf.client.updater.py** and instantiating its main class is all that is +required by the client prior to a TUF update request. The importation and +instantiation steps allow TUF to load all of the required metadata files +and set the repository mirror information. + +The **tuf.libtuf** module can be used to create a TUF repository. See +[tuf/README](../README.md) for more information on creating TUF repositories. + +The **tuf.interposition** package can also assist in integrating TUF with a +software updater. See [tuf/interposition/README](../interposition/README.md) +for more information on interposing Python urllib calls with TUF. + + +## Overview of the Update Process +1. The software update system instructs TUF to check for updates. + +2. TUF downloads and verifies timestamp.txt. + +3. If timestamp.txt indicates that release.txt has changed, TUF downloads and +verifies release.txt. + +4. TUF determines which metadata files listed in release.txt differ from those +described in the last release.txt that TUF has seen. If root.txt has changed, +the update process starts over using the new root.txt. + +5. TUF provides the software update system with a list of available files +according to targets.txt. + +6. The software update system instructs TUF to download a specific target +file. + +7. TUF downloads and verifies the file and then makes the file available to +the software update system. + + +## Example Client +### Refresh TUF Metadata and Download Target Files +```Python +# The client first imports the 'updater.py' module, the only module the +# client is required to import. The client will utilize a single class +# from this module. +import tuf.client.updater + +# The only other module the client interacts with is 'tuf.conf'. The +# client accesses this module solely to set the repository directory. +# This directory will hold the files downloaded from a remote repository. +tuf.conf.repository_directory = 'path/to/local_repository' + +# Next, the client creates a dictionary object containing the repository +# mirrors. The client may download content from any one of these mirrors. +# In the example below, a single mirror named 'mirror1' is defined. The +# mirror is located at 'http://localhost:8001', and all of the metadata +# and targets files can be found in the 'metadata' and 'targets' directory, +# respectively. If the client wishes to only download target files from +# specific directories on the mirror, the 'confined_target_dirs' field +# should be set. In the example, the client has chosen '', which is +# interpreted as no confinement. In other words, the client can download +# targets from any directory or subdirectories. If the client had chosen +# 'targets1/', they would have been confined to the '/targets/targets1/' +# directory on the 'http://localhost:8001' mirror. +repository_mirrors = {'mirror1': {'url_prefix': 'http://localhost:8001', + 'metadata_path': 'metadata', + 'targets_path': 'targets', + 'confined_target_dirs': ['']}} + +# The updater may now be instantiated. The Updater class of 'updater.py' +# is called with two arguments. The first argument assigns a name to this +# particular updater and the second argument the repository mirrors defined +# above. +updater = tuf.client.updater.Updater('updater', repository_mirrors) + +# The client calls the refresh() method to ensure it has the latest +# copies of the top-level metadata files (i.e., Root, Targets, Release, +# Timestamp). +updater.refresh() + +# The target file information of all the repository targets is determined next. +# Since all_targets() downloads the target files of every role, all role +# metadata is updated. +targets = updater.all_targets() + +# Among these targets, determine the ones that have changed since the client's +# last refresh(). A target is considered updated if it does not exist in +# 'destination_directory' (current directory) or the target located there has +# changed. +destination_directory = '.' +updated_targets = updater.updated_targets(targets, destination_directory) + +# Lastly, attempt to download each target among those that have changed. +# The updated target files are saved locally to 'destination_directory'. +for target in updated_targets: + updater.download_target(target, destination_directory) + +# Remove any files from the destination directory that are no longer being +# tracked. For example, a target file from a previous release that has since +# been removed on the remote repository. +updater.remove_obsolete_targets(destination_directory) +``` + +### Download Target Files of a Role +```Python +# Example demonstrating an update that only downloads the targets of +# a specific role (i.e., 'targets/django'). + +# Refresh the metadata of the top-level roles (i.e., Root, Targets, Release, Timestamp). +updater.refresh() + +# Update the 'targets/django' role, and determine the target files that have changed. +# targets_of_role() refreshes the minimum metadata needed to download the target files +# of the specified role (e.g., R1->R4->R5, where R2 and R3 are excluded). +targets_of_django = updater.targets_of_role('targets/django') +updated_targets = updater.updated_targets(targets_of_django, destination_directory) + +for target in updated_targets: + updater.download_target(target, destination_directory) +``` + +### Download Specific Target File +```Python +# Example demonstrating an update that downloads a specific target. + +# Refresh the metadata of the top-level roles (i.e., Root, Targets, Release, Timestamp). +updater.refresh() + +# target() updates role metadata when required. +target = updater.target('LICENSE.txt') +updated_target = updater.updated_targets([target], destination_directory) + +for target in updated_target: + updater.download_target(target, destination_directory) +``` + +###A Simple Integration Example with basic_client.py +```Bash +# Assume a simple TUF repository has been setup with 'tuf.libtuf.py'. +$ basic_client.py --repo http://localhost:8001 + +# Metadata and target files are silently updated. An exception is only raised if an error, +# or attack, is detected. Inspect 'tuf.log' for the outcome of the update process. + +$ cat tuf.log +[2013-12-16 16:17:05,267 UTC] [tuf.download] [INFO][_download_file:726@download.py] +Downloading: http://localhost:8001/metadata/timestamp.txt + +[2013-12-16 16:17:05,269 UTC] [tuf.download] [WARNING][_check_content_length:589@download.py] +reported_length (545) < required_length (2048) + +[2013-12-16 16:17:05,269 UTC] [tuf.download] [WARNING][_check_downloaded_length:656@download.py] +Downloaded 545 bytes, but expected 2048 bytes. There is a difference of 1503 bytes! + +[2013-12-16 16:17:05,611 UTC] [tuf.download] [INFO][_download_file:726@download.py] +Downloading: http://localhost:8001/metadata/release.txt + +[2013-12-16 16:17:05,612 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] +The file\'s sha256 hash is correct: 782675fadd650eeb2926d33c401b5896caacf4fd6766498baf2bce2f3b739db4 + +[2013-12-16 16:17:05,951 UTC] [tuf.download] [INFO][_download_file:726@download.py] +Downloading: http://localhost:8001/metadata/targets.txt + +[2013-12-16 16:17:05,952 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] +The file\'s sha256 hash is correct: a5019c28a1595c43a14cad2b6252c4d1db472dd6412a9204181ad6d61b1dd69a + +[2013-12-16 16:17:06,299 UTC] [tuf.download] [INFO][_download_file:726@download.py] +Downloading: http://localhost:8001/targets/file1.txt + +[2013-12-16 16:17:06,303 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] +The file's sha256 hash is correct: ecdc5536f73bdae8816f0ea40726ef5e9b810d914493075903bb90623d97b1d8 diff --git a/tuf/client/basic_client.py b/tuf/client/basic_client.py index 1ba36dbc..86397e4a 100755 --- a/tuf/client/basic_client.py +++ b/tuf/client/basic_client.py @@ -211,7 +211,7 @@ def parse_options(): # the current directory. try: update_client(repository_mirror) - except (tuf.RepositoryError, tuf.ExpiredMetadataError), e: + except (tuf.NoWorkingMirrorError, tuf.RepositoryError), e: sys.stderr.write('Error: '+str(e)+'\n') sys.exit(1) diff --git a/tuf/client/updater.py b/tuf/client/updater.py index 3b875fb3..4416ebd6 100755 --- a/tuf/client/updater.py +++ b/tuf/client/updater.py @@ -97,7 +97,6 @@ # The updated target files are saved locally to 'destination_directory'. for target in updated_targets: updater.download_target(target, destination_directory) - """ import errno @@ -105,12 +104,15 @@ import os import shutil import time +import urllib +import random import tuf import tuf.conf import tuf.download import tuf.formats import tuf.hash +import tuf.keys import tuf.keydb import tuf.log import tuf.mirrors @@ -195,6 +197,12 @@ class Updater(object): served by the repository but have since been removed, can be deleted from disk by the client by calling this method. + Note: The methods listed above are public and intended for the software + updater integrating TUF with this module. All other methods that may begin + with a single leading underscore are non-public and only used internally. + updater.py is not subclassed in TUF, nor is it designed to be subclassed, + so double leading underscores is not used. + http://www.python.org/dev/peps/pep-0008/#method-names-and-instance-variables """ def __init__(self, updater_name, repository_mirrors): @@ -251,7 +259,6 @@ def __init__(self, updater_name, repository_mirrors): None. - """ # Do the arguments have the correct format? @@ -282,6 +289,11 @@ def __init__(self, updater_name, repository_mirrors): # Store the location of the client's metadata directory. self.metadata_directory = {} + + # Store the 'consistent_snapshots' of the Root role. This setting + # determines if metadata and target files downloaded from remote + # repositories include the digest. + self.consistent_snapshots = False # Ensure the repository metadata directory has been set. if tuf.conf.repository_directory is None: @@ -327,7 +339,6 @@ def __init__(self, updater_name, repository_mirrors): def __str__(self): """ The string representation of an Updater object. - """ return self.name @@ -370,7 +381,6 @@ def _load_metadata_from_file(self, metadata_set, metadata_role): None. - """ # Ensure we have a valid metadata set. @@ -395,12 +405,15 @@ def _load_metadata_from_file(self, metadata_set, metadata_role): # Save the metadata object to the metadata store. self.metadata[metadata_set][metadata_role] = metadata_object - - # We need to rebuild the key and role databases if - # metadata object is 'root' or target metadata. + + # If 'metadata_role' is 'root' or targets metadata, the key and role + # databases must be rebuilt. If 'root', ensure self.consistent_snaptshots + # is updated. if metadata_set == 'current': if metadata_role == 'root': self._rebuild_key_and_role_db() + self.consistent_snapshots = metadata_object['consistent_snapshots'] + elif metadata_object['_type'] == 'Targets': # TODO: Should we also remove the keys of the delegated roles? tuf.roledb.remove_delegated_roles(metadata_role) @@ -415,8 +428,8 @@ def _rebuild_key_and_role_db(self): Rebuild the key and role databases from the currently trusted 'root' metadata object extracted from 'root.txt'. This private - function is called when a new/updated 'root' metadata file is loaded. - This function will only store the role information for the top-level + method is called when a new/updated 'root' metadata file is loaded. + This method will only store the role information for the top-level roles (i.e., 'root', 'targets', 'release', 'timestamp'). @@ -435,7 +448,6 @@ def _rebuild_key_and_role_db(self): None. - """ # Clobbering this means all delegated metadata files are rendered outdated @@ -475,7 +487,6 @@ def _import_delegations(self, parent_role): None. - """ current_parent_metadata = self.metadata['current'][parent_role] @@ -492,13 +503,13 @@ def _import_delegations(self, parent_role): # Iterate through the keys of the delegated roles of 'parent_role' # and load them. for keyid, keyinfo in keys_info.items(): - if keyinfo['keytype'] == 'rsa': - rsa_key = tuf.rsa_key.create_from_metadata_format(keyinfo) + if keyinfo['keytype'] in ['rsa', 'ed25519']: + key = tuf.keys.format_metadata_to_key(keyinfo) # We specify the keyid to ensure that it's the correct keyid # for the key. try: - tuf.keydb.add_rsakey(rsa_key, keyid) + tuf.keydb.add_key(key, keyid) except tuf.KeyAlreadyExistsError: pass except (tuf.FormatError, tuf.Error), e: @@ -527,7 +538,7 @@ def _import_delegations(self, parent_role): - def refresh(self): + def refresh(self, unsafely_update_root_if_necessary=True): """ Update the latest copies of the metadata for the top-level roles. @@ -537,11 +548,13 @@ def refresh(self): The client would call refresh() prior to requesting target file information. Calling refresh() ensures target methods, like all_targets() and target(), refer to the latest available content. - The latest copies for delegated metadata are downloaded and updated - by the target methods. + The latest copies, according to the currently trusted top-level metadata, + of delegated metadata are downloaded and updated by the target methods. - None. + unsafely_update_root_if_necessary: + Boolean that indicates whether to unsafely update the Root metadata + if any of the top-level metadata cannot be downloaded successfully. tuf.NoWorkingMirrorError: @@ -556,17 +569,32 @@ def refresh(self): None. - """ + + # Do the arguments have the correct format? + # This check ensures the arguments have the appropriate + # number of objects and object types, and that all dict + # keys are properly named. + # Raise 'tuf.FormatError' if the check fail. + tuf.formats.BOOLEAN_SCHEMA.check_match(unsafely_update_root_if_necessary) # The timestamp role does not have signed metadata about it; otherwise we # would need an infinite regress of metadata. Therefore, we use some # default, sane metadata about it. DEFAULT_TIMESTAMP_FILEINFO = { - 'hashes':None, + 'hashes': {}, 'length': tuf.conf.DEFAULT_TIMESTAMP_REQUIRED_LENGTH } + # The Root role may be updated without knowing its hash if top-level + # metadata cannot be safely downloaded (e.g., keys may have been revoked, + # thus requiring a new Root file that includes the updated keys) and + # 'unsafely_update_root_if_necessary' is True. + DEFAULT_ROOT_FILEINFO = { + 'hashes': {}, + 'length': tuf.conf.DEFAULT_ROOT_REQUIRED_LENGTH + } + # Update the top-level metadata. The _update_metadata_if_changed() and # _update_metadata() calls below do NOT perform an update if there # is insufficient trusted signatures for the specified metadata. @@ -574,40 +602,54 @@ def refresh(self): # Use default but sane information for timestamp metadata, and do not # require strict checks on its required length. - self._update_metadata('timestamp', DEFAULT_TIMESTAMP_FILEINFO) + try: + self._update_metadata('timestamp', DEFAULT_TIMESTAMP_FILEINFO) + self._update_metadata_if_changed('release', + referenced_metadata='timestamp') + self._update_metadata_if_changed('root') + self._update_metadata_if_changed('targets') + + except tuf.NoWorkingMirrorError, e: + if unsafely_update_root_if_necessary: + message = 'Valid top-level metadata cannot be downloaded. Unsafely '+\ + 'update the Root metadata.' + logger.info(message) + + self._update_metadata('root', DEFAULT_ROOT_FILEINFO) + self.refresh(unsafely_update_root_if_necessary=False) + + else: + raise - self._update_metadata_if_changed('release', referenced_metadata='timestamp') - - self._update_metadata_if_changed('root') - - self._update_metadata_if_changed('targets') - - # Updated the top-level metadata (which all had valid signatures), however, - # have they expired? Raise 'tuf.ExpiredMetadataError' if any of the metadata - # has expired. - for metadata_role in ['timestamp', 'root', 'release', 'targets']: - self._ensure_not_expired(metadata_role) + else: + # Updated the top-level metadata (which all had valid signatures), + # however, have they expired? Raise 'tuf.ExpiredMetadataError' if any of + # the metadata has expired. + for metadata_role in ['timestamp', 'root', 'release', 'targets']: + self._ensure_not_expired(metadata_role) - def __check_hashes(self, file_object, trusted_hashes): + def _check_hashes(self, file_object, trusted_hashes): """ - A helper function that verifies multiple secure hashes of the downloaded - file. If any of these fail it raises an exception. This is to conform - with the TUF specs, which support clients with different hashing - algorithms. The 'hash.py' module is used to compute the hashes of the - 'file_object'. + A private helper method that verifies multiple secure hashes of the + downloaded file 'file_object'. If any of these fail it raises an + exception. This is to conform with the TUF spec, which support clients + with different hashing algorithms. The 'hash.py' module is used to compute + the hashes of 'file_object'. file_object: - A file-like object. + A 'tuf.util.TempFile' file-like object. 'file_object' ensures that a + read() without a size argument properly reads the entire file. trusted_hashes: A dictionary with hash-algorithm names as keys and hashes as dict values. - The hashes should be in the hexdigest format. + The hashes should be in the hexdigest format. Should be Conformant to + 'tuf.formats.HASHDICT_SCHEMA'. tuf.BadHashError, if the hashes don't match. @@ -617,15 +659,16 @@ def __check_hashes(self, file_object, trusted_hashes): None. - """ - # Verify each trusted hash of 'trusted_hashes'. Raise exception if - # any of the hashes are incorrect and return if all are correct. + # Verify each trusted hash of 'trusted_hashes'. If all are valid, simply + # return. for algorithm, trusted_hash in trusted_hashes.items(): digest_object = tuf.hash.digest(algorithm) digest_object.update(file_object.read()) computed_hash = digest_object.hexdigest() + + # Raise an exception if any of the hashes are incorrect. if trusted_hash != computed_hash: raise tuf.BadHashError(trusted_hash, computed_hash) else: @@ -635,102 +678,115 @@ def __check_hashes(self, file_object, trusted_hashes): - def __hard_check_compressed_file_length(self, file_object, - compressed_file_length): + def _hard_check_file_length(self, file_object, trusted_file_length): """ - A helper function that checks the expected compressed length of a - file-like object. The length of the file must be strictly equal to the - expected length. This is a deliberately redundant implementation designed - to complement tuf.download._check_downloaded_length(). + A private helper method that ensures the length of 'file_object' is + strictly equal to 'trusted_file_length'. This is a deliberately + redundant implementation designed to complement + tuf.download._check_downloaded_length(). file_object: - A file-like object. + A 'tuf.util.TempFile' file-like object. 'file_object' ensures that a + read() without a size argument properly reads the entire file. - compressed_file_length: - A nonnegative integer that is the expected compressed length of the - file. + trusted_file_length: + A non-negative integer that is the trusted length of the file. - tuf.DownloadLengthMismatchError, if the lengths don't match. + tuf.DownloadLengthMismatchError, if the lengths do not match. - None. + Reads the contents of 'file_object' and logs a message if 'file_object' + matches the trusted length. None. - """ - observed_length = file_object.get_compressed_length() - if observed_length != compressed_file_length: - raise tuf.DownloadLengthMismatchError(compressed_file_length, + # Read the entire contents of 'file_object', a 'tuf.util.TempFile' file-like + # object that ensures the entire file is read. + observed_length = len(file_object.read()) + + # Return and log a message if the length 'file_object' is equal to + # 'trusted_file_length', otherwise raise an exception. A hard check + # ensures that a downloaded file strictly matches a known, or trusted, + # file length. + if observed_length != trusted_file_length: + raise tuf.DownloadLengthMismatchError(trusted_file_length, observed_length) else: - logger.debug('file length ('+str(observed_length)+\ - ') == trusted length ('+str(compressed_file_length)+')') + logger.debug('Observed length ('+str(observed_length)+\ + ') == trusted length ('+str(trusted_file_length)+')') - def __soft_check_compressed_file_length(self, file_object, - compressed_file_length): + def _soft_check_file_length(self, file_object, trusted_file_length): """ - A helper function that checks the expected compressed length of a - file-like object. The length of the file must be less than or equal to - the expected length. This is a deliberately redundant implementation - designed to complement tuf.download._check_downloaded_length(). + A private helper method that checks the trusted file length of a + 'tuf.util.TempFile' file-like object. The length of the file must be less + than or equal to the expected length. This is a deliberately redundant + implementation designed to complement + tuf.download._check_downloaded_length(). file_object: - A file-like object. + A 'tuf.util.TempFile' file-like object. 'file_object' ensures that a + read() without a size argument properly reads the entire file. - compressed_file_length: - A nonnegative integer that is the expected compressed length of the - file. + trusted_file_length: + A non-negative integer that is the trusted length of the file. - tuf.DownloadLengthMismatchError, if the lengths don't match. + tuf.DownloadLengthMismatchError, if the lengths do not match. - None. + Reads the contents of 'file_object' and logs a message if 'file_object' + is less than or equal to the trusted length. None. - """ - observed_length = file_object.get_compressed_length() - if observed_length > compressed_file_length: - raise tuf.DownloadLengthMismatchError(compressed_file_length, + # Read the entire contents of 'file_object', a 'tuf.util.TempFile' file-like + # object that ensures the entire file is read. + observed_length = len(file_object.read()) + + # Return and log a message if 'file_object' is less than or equal to + # 'trusted_file_length', otherwise raise an exception. A soft check + # ensures that an upper bound restricts how large a file is downloaded. + if observed_length > trusted_file_length: + raise tuf.DownloadLengthMismatchError(trusted_file_length, observed_length) else: - logger.debug('file length ('+str(observed_length)+\ - ') <= trusted length ('+str(compressed_file_length)+')') + logger.debug('Observed length ('+str(observed_length)+\ + ') <= trusted length ('+str(trusted_file_length)+')') - def get_target_file(self, target_filepath, compressed_file_length, - uncompressed_file_hashes): + def _get_target_file(self, target_filepath, file_length, file_hashes): """ - Safely download a target file up to a certain length, and check its + Safely (i.e., the file length and hash are strictly equal to the + trusted) download a target file up to a certain length, and check its hashes thereafter. target_filepath: - The relative target filepath obtained from TUF targets metadata. + The target filepath (relative to the repository targets directory) + obtained from TUF targets metadata. - compressed_file_length: + file_length: The expected compressed length of the target file. If the file is not compressed, then it will simply be its uncompressed length. - uncompressed_file_hashes: + file_hashes: The expected hashes of the target file. @@ -744,37 +800,50 @@ def get_target_file(self, target_filepath, compressed_file_length, a temporary file and returned. - A tuf.util.TempFile file-like object containing the target. - + A 'tuf.util.TempFile' file-like object containing the target. """ - def verify_uncompressed_target_file(target_file_object): + # Define a callable function that is passed as an argument to _get_file() + # and called. The 'verify_target_file' function ensures the file length + # and hashes of 'target_filepath' are strictly equal to the trusted values. + def verify_target_file(target_file_object): + # Every target file must have its length and hashes inspected. - self.__hard_check_compressed_file_length(target_file_object, - compressed_file_length) - self.__check_hashes(target_file_object, uncompressed_file_hashes) + self._hard_check_file_length(target_file_object, file_length) + self._check_hashes(target_file_object, file_hashes) - return self.__get_file(target_filepath, verify_uncompressed_target_file, - 'target', compressed_file_length, - download_safely=True, compression=None) + # Target files, unlike metadata files, are not decompressed; the + # 'compression' argument to _get_file() is needed only for decompression of + # metadata. Target files may be compressed or uncompressed. + if self.consistent_snapshots: + target_digest = random.choice(file_hashes.values()) + dirname, basename = os.path.split(target_filepath) + target_filepath = os.path.join(dirname, target_digest+'.'+basename) + + return self._get_file(target_filepath, verify_target_file, + 'target', file_length, compression=None, + verify_compressed_file_function=None, + download_safely=True) - def __verify_uncompressed_metadata_file(self, metadata_file_object, - metadata_role): + def _verify_uncompressed_metadata_file(self, metadata_file_object, + metadata_role): """ - A private helper function to verify an uncompressed downloaded metadata + A private helper function to verify an uncompressed metadata file. metadata_file_object: - A tuf.util.TempFile instance containing the metadata file. + A 'tuf.util.TempFile' instance containing the metadata file. + 'metadata_file_object' ensures the entire is returned with read(). metadata_role: - The role name of the metadata. + The role name of the metadata (e.g., 'root', 'targets', + 'targets/linux/x86'). tuf.ForbiddenTargetError: @@ -797,11 +866,10 @@ def __verify_uncompressed_metadata_file(self, metadata_file_object, In case the metadata file does not have a valid signature. - None. + The contents of 'metadata_file_object' is read and loaded. None. - """ metadata = metadata_file_object.read() @@ -810,7 +878,8 @@ def __verify_uncompressed_metadata_file(self, metadata_file_object, except Exception, exception: raise tuf.InvalidMetadataJSONError(exception) else: - # Ensure the loaded 'metadata_signable' is properly formatted. + # Ensure the loaded 'metadata_signable' is properly formatted. Raise + # 'tuf.FormatError' if not. tuf.formats.check_signable_object_format(metadata_signable) # Is 'metadata_signable' newer than the currently installed @@ -840,8 +909,10 @@ def __verify_uncompressed_metadata_file(self, metadata_file_object, - def unsafely_get_metadata_file(self, metadata_role, metadata_filepath, - compressed_file_length): + def _unsafely_get_metadata_file(self, metadata_role, metadata_filepath, + uncompressed_fileinfo, + compression=None, compressed_fileinfo=None): + """ Unsafely download a metadata file up to a certain length. The actual file @@ -850,14 +921,24 @@ def unsafely_get_metadata_file(self, metadata_role, metadata_filepath, metadata_role: - The role name of the metadata. + The role name of the metadata (e.g., 'root', 'targets', + 'targets/linux/x86'). metadata_filepath: - The relative metadata filepath. + The metadata filepath (i.e., relative to the repository metadata + directory). - compressed_file_length: - The expected compressed length of the metadata file. If the file is not - compressed, then it will simply be its uncompressed length. + uncompressed_fileinfo: + The trusted file length and hashes of the uncompressed version of the + metadata file. Should be 'tuf.formats.FILEINFO_SCHEMA'. + + compression: + The name of the compression algorithm (e.g., 'gzip'), if the metadata + file is compressed. + + compressed_fileinfo: + The fileinfo of the metadata file, if it is compressed. Should be + 'tuf.formats.FILEINFO_SCHEMA'. tuf.NoWorkingMirrorError: @@ -870,28 +951,50 @@ def unsafely_get_metadata_file(self, metadata_role, metadata_filepath, in a temporary file and returned. - A tuf.util.TempFile file-like object containing the metadata. - + A 'tuf.util.TempFile' file-like object containing the metadata. """ + + # Store file length and hashes of the uncompressed version metadata. + # The uncompressed version is always verified. + uncompressed_file_length = uncompressed_fileinfo['length'] + uncompressed_file_hashes = uncompressed_fileinfo['hashes'] + download_file_length = uncompressed_file_length + + # Store the file length and hashes of the compressed version of the + # metadata, if compressions is set. + if compression is not None and compressed_fileinfo is not None: + compressed_file_length = compressed_fileinfo['length'] + compressed_file_hashes = compressed_fileinfo['hashes'] + download_file_length = compressed_file_length def unsafely_verify_uncompressed_metadata_file(metadata_file_object): - self.__soft_check_compressed_file_length(metadata_file_object, - compressed_file_length) - self.__verify_uncompressed_metadata_file(metadata_file_object, - metadata_role) + self._soft_check_file_length(metadata_file_object, + uncompressed_file_length) + self._check_hashes(metadata_file_object, uncompressed_file_hashes) + self._verify_uncompressed_metadata_file(metadata_file_object, + metadata_role) + + def unsafely_verify_compressed_metadata_file(metadata_file_object): + self._hard_check_file_length(metadata_file_object, + compressed_file_length) + self._check_hashes(metadata_file_object, compressed_file_hashes) - return self.__get_file(metadata_filepath, - unsafely_verify_uncompressed_metadata_file, 'meta', - compressed_file_length, download_safely=False, - compression=None) + if compression is None: + unsafely_verify_compressed_metadata_file = None + + return self._get_file(metadata_filepath, + unsafely_verify_uncompressed_metadata_file, 'meta', + download_file_length, compression, + unsafely_verify_compressed_metadata_file, + download_safely=False) - def safely_get_metadata_file(self, metadata_role, metadata_filepath, - compressed_file_length, - uncompressed_file_hashes, compression): + def _safely_get_metadata_file(self, metadata_role, metadata_filepath, + uncompressed_fileinfo, + compression=None, compressed_fileinfo=None): """ Safely download a metadata file up to a certain length, and check its @@ -899,20 +1002,24 @@ def safely_get_metadata_file(self, metadata_role, metadata_filepath, metadata_role: - The role name of the metadata. + The role name of the metadata (e.g., 'root', 'targets', + 'targets/linux/x86'). metadata_filepath: - The relative metadata filepath. - - compressed_file_length: - The expected compressed length of the metadata file. If the file is not - compressed, then it will simply be its uncompressed length. - - uncompressed_file_hashes: - The expected hashes of the metadata file. + The metadata filepath (i.e., relative to the repository metadata + directory). + + uncompressed_fileinfo: + The trusted file length and hashes of the uncompressed version of the + metadata file. Should be 'tuf.formats.FILEINFO_SCHEMA'. compression: - The name of the compression algorithm used to compress the metadata. + The name of the compression algorithm (e.g., 'gzip'), if the metadata + file is compressed. + + compressed_fileinfo: + The fileinfo of the metadata file, if it is compressed. Should be + 'tuf.formats.FILEINFO_SCHEMA'. tuf.NoWorkingMirrorError: @@ -925,21 +1032,42 @@ def safely_get_metadata_file(self, metadata_role, metadata_filepath, in a temporary file and returned. - A tuf.util.TempFile file-like object containing the metadata. - + A 'tuf.util.TempFile' file-like object containing the metadata. """ - + + # Store file length and hashes of the uncompressed version metadata. + # The uncompressed version is always verified. + uncompressed_file_length = uncompressed_fileinfo['length'] + uncompressed_file_hashes = uncompressed_fileinfo['hashes'] + download_file_length = uncompressed_file_length + + # Store the file length and hashes of the compressed version of the + # metadata, if compressions is set. + if compression and compressed_fileinfo: + compressed_file_length = compressed_fileinfo['length'] + compressed_file_hashes = compressed_fileinfo['hashes'] + download_file_length = compressed_file_length + def safely_verify_uncompressed_metadata_file(metadata_file_object): - self.__hard_check_compressed_file_length(metadata_file_object, - compressed_file_length) - self.__check_hashes(metadata_file_object, uncompressed_file_hashes) - self.__verify_uncompressed_metadata_file(metadata_file_object, + self._hard_check_file_length(metadata_file_object, + uncompressed_file_length) + self._check_hashes(metadata_file_object, uncompressed_file_hashes) + self._verify_uncompressed_metadata_file(metadata_file_object, metadata_role) - return self.__get_file(metadata_filepath, - safely_verify_uncompressed_metadata_file, 'meta', - compressed_file_length, download_safely=True, - compression=compression) + def safely_verify_compressed_metadata_file(metadata_file_object): + self._hard_check_file_length(metadata_file_object, + compressed_file_length) + self._check_hashes(metadata_file_object, compressed_file_hashes) + + if compression is None: + safely_verify_compressed_metadata_file = None + + return self._get_file(metadata_filepath, + safely_verify_uncompressed_metadata_file, 'meta', + download_file_length, compression, + safely_verify_compressed_metadata_file, + download_safely=True) @@ -948,8 +1076,9 @@ def safely_verify_uncompressed_metadata_file(metadata_file_object): # TODO: Instead of the more fragile 'download_safely' switch, unroll the # function into two separate ones: one for "safe" download, and the other one # for "unsafe" download? This should induce safer and more readable code. - def __get_file(self, filepath, verify_uncompressed_file, file_type, - compressed_file_length, download_safely, compression): + def _get_file(self, filepath, verify_file_function, file_type, + file_length, compression=None, + verify_compressed_file_function=None, download_safely=True): """ Try downloading, up to a certain length, a metadata or target file from a @@ -960,26 +1089,34 @@ def __get_file(self, filepath, verify_uncompressed_file, file_type, filepath: The relative metadata or target filepath. - verify_uncompressed_file: - A function which expects an uncompressed file-like object and which - will raise an exception in case the file is not valid for any reason. + verify_file_function: + A callable function that expects a 'tuf.util.TempFile' file-like object + and raises an exception if the file is invalid. Target files and + uncompressed versions of metadata may be verified with + 'verify_file_function'. file_type: Type of data needed for download, must correspond to one of the strings in the list ['meta', 'target']. 'meta' for metadata file type or - 'target' for target file type. It should correspond to NAME_SCHEMA - format. + 'target' for target file type. It should correspond to the + 'tuf.formats.NAME_SCHEMA' format. - compressed_file_length: - The expected compressed length of the target or metadata file. If the - file is not compressed, then it will simply be its uncompressed length. + file_length: + The expected length, or upper bound, of the target or metadata file to + be downloaded. + + compression: + The name of the compression algorithm (e.g., 'gzip'), if the metadata + file is compressed. + + verify_compressed_file_function: + If compression is specified, in the case of metadata files, this + callable function may be set to perform verification of the compressed + version of the metadata file. Decompressed metadata is also verified. download_safely: A boolean switch to toggle safe or unsafe download of the file. - compression: - The name of the compression algorithm used to compress the file. - tuf.NoWorkingMirrorError: The metadata could not be fetched. This is raised only when all known @@ -991,8 +1128,7 @@ def __get_file(self, filepath, verify_uncompressed_file, file_type, file and returned. - A tuf.util.TempFile file-like object containing the metadata or target. - + A 'tuf.util.TempFile' file-like object containing the metadata or target. """ file_mirrors = tuf.mirrors.get_list_of_mirrors(file_type, filepath, @@ -1005,18 +1141,23 @@ def __get_file(self, filepath, verify_uncompressed_file, file_type, try: if download_safely: file_object = tuf.download.safe_download(file_mirror, - compressed_file_length) + file_length) else: file_object = tuf.download.unsafe_download(file_mirror, - compressed_file_length) + file_length) - if compression: - logger.debug('Decompressing '+str(file_mirror)) + if compression is not None: + if verify_compressed_file_function is not None: + verify_compressed_file_function(file_object) + logger.info('Decompressing '+str(file_mirror)) file_object.decompress_temp_file_object(compression) else: - logger.debug('Not decompressing '+str(file_mirror)) - - verify_uncompressed_file(file_object) + logger.info('Not decompressing '+str(file_mirror)) + + # Verify 'file_object' according to the callable function. + # 'file_object' is also verified if decompressed above (i.e., the + # uncompressed version). + verify_file_function(file_object) except Exception, exception: # Remember the error from this mirror, and "reset" the target file. @@ -1037,11 +1178,12 @@ def __get_file(self, filepath, verify_uncompressed_file, file_type, - def _update_metadata(self, metadata_role, fileinfo, compression=None): + def _update_metadata(self, metadata_role, uncompressed_fileinfo, + compression=None, compressed_fileinfo=None): """ Download, verify, and 'install' the metadata belonging to 'metadata_role'. - Calling this function implies the metadata has been updated by the + Calling this method implies the metadata has been updated by the repository and thus needs to be re-downloaded. The current and previous metadata stores are updated if the newly downloaded metadata is successfully downloaded and verified. @@ -1051,27 +1193,26 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): The name of the metadata. This is a role name and should not end in '.txt'. Examples: 'root', 'targets', 'targets/linux/x86'. - fileinfo: - A dictionary containing length and hashes of the metadata file. + uncompressed_fileinfo: + A dictionary containing length and hashes of the uncompressed metadata + file. + Ex: {"hashes": {"sha256": "3a5a6ec1f353...dedce36e0"}, "length": 1340} - The length must be that of the compressed metadata file if it is - compressed, or uncompressed metadata file if it is uncompressed. - The hashes must be that of the uncompressed metadata file. - - STRICT_REQUIRED_LENGTH: - A Boolean indicator used to signal whether we should perform strict - checking of the required length in 'fileinfo'. True by default. True - by default. We explicitly set this to False when we know that we want - to turn this off for downloading the timestamp metadata, which has no - signed required_length. - + compression: A string designating the compression type of 'metadata_role'. The 'release' metadata file may be optionally downloaded and stored in compressed form. Currently, only metadata files compressed with 'gzip' are considered. Any other string is ignored. + compressed_fileinfo: + A dictionary containing length and hashes of the compressed metadata + file. + + Ex: {"hashes": {"sha256": "3a5a6ec1f353...dedce36e0"}, + "length": 1340} + tuf.NoWorkingMirrorError: The metadata could not be updated. This is not specific to a single @@ -1085,7 +1226,6 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): None. - """ # Construct the metadata filename as expected by the download/mirror modules. @@ -1097,11 +1237,6 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): if compression == 'gzip': metadata_filename = metadata_filename + '.gz' - # Extract file length and file hashes. They will be passed as arguments - # to 'download_file' function. - compressed_file_length = fileinfo['length'] - uncompressed_file_hashes = fileinfo['hashes'] - # Attempt a file download from each mirror until the file is downloaded and # verified. If the signature of the downloaded file is valid, proceed, # otherwise log a warning and try the next mirror. 'metadata_file_object' @@ -1121,17 +1256,36 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): # Note also that we presently support decompression of only "safe" # metadata, but this is easily extend to "unsafe" metadata as well as # "safe" targets. - + if metadata_role == 'timestamp': metadata_file_object = \ - self.unsafely_get_metadata_file(metadata_role, metadata_filename, - compressed_file_length) - else: + self._unsafely_get_metadata_file(metadata_role, metadata_filename, + uncompressed_fileinfo, + compression, compressed_fileinfo) + + elif metadata_role == 'root' and not len(uncompressed_fileinfo['hashes']): metadata_file_object = \ - self.safely_get_metadata_file(metadata_role, metadata_filename, - compressed_file_length, - uncompressed_file_hashes, - compression=compression) + self._unsafely_get_metadata_file(metadata_role, metadata_filename, + uncompressed_fileinfo, + compression, compressed_fileinfo) + + else: + remote_filename = metadata_filename + if self.consistent_snapshots: + if compression: + filename_digest = \ + random.choice(compressed_fileinfo['hashes'].values()) + + else: + filename_digest = \ + random.choice(uncompressed_fileinfo['hashes'].values()) + dirname, basename = os.path.split(remote_filename) + remote_filename = os.path.join(dirname, filename_digest+'.'+basename) + + metadata_file_object = \ + self._safely_get_metadata_file(metadata_role, remote_filename, + uncompressed_fileinfo, + compression, compressed_fileinfo) # The metadata has been verified. Move the metadata file into place. # First, move the 'current' metadata file to the 'previous' directory @@ -1144,6 +1298,7 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): previous_filepath = os.path.join(self.metadata_directory['previous'], metadata_filename) previous_filepath = os.path.abspath(previous_filepath) + if os.path.exists(current_filepath): # Previous metadata might not exist, say when delegations are added. tuf.util.ensure_parent_dir(previous_filepath) @@ -1160,6 +1315,7 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): current_uncompressed_filepath = \ os.path.abspath(current_uncompressed_filepath) metadata_file_object.move(current_uncompressed_filepath) + else: metadata_file_object.move(current_filepath) @@ -1169,11 +1325,20 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): updated_metadata_object = metadata_signable['signed'] current_metadata_object = self.metadata['current'].get(metadata_role) - # Finally, update the metadata and fileinfo stores. + # Finally, update the metadata and fileinfo stores, and rebuild the + # key and role info for the top-level roles if 'metadata_role' is root. + # Rebuilding the the key and role info is required if the newly-installed + # root metadata has revoked keys or updated any top-level role information. logger.debug('Updated '+repr(current_filepath)+'.') self.metadata['previous'][metadata_role] = current_metadata_object self.metadata['current'][metadata_role] = updated_metadata_object - self._update_fileinfo(metadata_filename) + self._update_fileinfo(metadata_filename) + + # Ensure the role and key information of the top-level roles is also updated + # according to the newly-installed Root metadata. + if metadata_role == 'root': + self._rebuild_key_and_role_db() + self.consistent_snapshots = updated_metadata_object['consistent_snapshots'] @@ -1184,13 +1349,13 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas Update the metadata for 'metadata_role' if it has changed. With the exception of the 'timestamp' role, all the top-level roles are updated - by this function. The 'timestamp' role is always downloaded from a mirror + by this method. The 'timestamp' role is always downloaded from a mirror without first checking if it has been updated; it is updated in refresh() - by calling _update_metadata('timestamp'). This function is also called for + by calling _update_metadata('timestamp'). This method is also called for delegated role metadata, which are referenced by 'release'. If the metadata needs to be updated but an update cannot be obtained, - this function will delete the file (with the exception of the root + this method will delete the file (with the exception of the root metadata, which never gets removed without a replacement). Due to the way in which metadata files are updated, it is expected that @@ -1199,7 +1364,7 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas root -> targets' order. For delegated metadata, the parent role is updated before the delegated role. Taking into account that 'referenced_metadata' is updated and verified before 'metadata_role', - this function determines if 'metadata_role' has changed by checking + this method determines if 'metadata_role' has changed by checking the 'meta' field of the newly updated 'referenced_metadata'. @@ -1213,13 +1378,13 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas is the referenced metadata for the 'root', and 'targets' roles. The 'timestamp' metadata is always downloaded regardless. In other words, it is updated by calling _update_metadata('timestamp') - and not by this function. The referenced metadata for 'release' + and not by this method. The referenced metadata for 'release' is 'timestamp'. See refresh(). tuf.NoWorkingMirrorError: - If 'metadata_role' could not be downloaded after determining - that it had changed. + If 'metadata_role' could not be downloaded after determining that it had + changed. tuf.RepositoryError: If the referenced metadata is missing. @@ -1234,7 +1399,6 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas None. - """ uncompressed_metadata_filename = metadata_role + '.txt' @@ -1245,6 +1409,7 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas message = 'Cannot update '+repr(metadata_role)+' because ' \ +referenced_metadata+' is missing.' raise tuf.RepositoryError(message) + # The referenced metadata has been loaded. Extract the new # fileinfo for 'metadata_role' from it. else: @@ -1259,18 +1424,25 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas # must begin with 'targets/'. The Release role lists all the Targets # metadata available on the repository, including any that may be in # compressed form. + # + # In addition to validating the fileinfo (i.e., file lengths and hashes) + # of the uncompressed metadata, the compressed version is also verified to + # match its respective fileinfo. Verifying the compressed fileinfo ensures + # untrusted data is not decompressed prior to verifying hashes, or + # decompressing a file that may be invalid or partially intact. compression = None + compressed_fileinfo = None # Extract the fileinfo of the uncompressed version of 'metadata_role'. uncompressed_fileinfo = self.metadata['current'][referenced_metadata] \ ['meta'] \ [uncompressed_metadata_filename] - # Check for availability of compressed versions of 'release.txt', - # 'targets.txt', and delegated Targets, which also start with 'targets'. + # Check for the availability of compressed versions of 'release.txt', + # 'targets.txt', and delegated Targets (that also start with 'targets'). # For 'targets.txt' and delegated metadata, 'referenced_metata' # should always be 'release'. 'release.txt' specifies all roles - # provided by a repository, including their file sizes and hashes. + # provided by a repository, including their file lengths and hashes. if metadata_role == 'release' or metadata_role.startswith('targets'): gzip_metadata_filename = uncompressed_metadata_filename + '.gz' if gzip_metadata_filename in self.metadata['current'] \ @@ -1278,33 +1450,28 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas compression = 'gzip' compressed_fileinfo = self.metadata['current'][referenced_metadata] \ ['meta'][gzip_metadata_filename] - # NOTE: When we download the compressed file, we care about its - # compressed length. However, we check the hash of the uncompressed - # file; therefore we use the hashes of the uncompressed file. - fileinfo = {'length': compressed_fileinfo['length'], - 'hashes': uncompressed_fileinfo['hashes']} + logger.debug('Compressed version of '+\ repr(uncompressed_metadata_filename)+' is available at '+\ repr(gzip_metadata_filename)+'.') else: logger.debug('Compressed version of '+\ repr(uncompressed_metadata_filename)+' not available.') - fileinfo = uncompressed_fileinfo - else: - fileinfo = uncompressed_fileinfo # Simply return if the file has not changed, according to the metadata # about the uncompressed file provided by the referenced metadata. if not self._fileinfo_has_changed(uncompressed_metadata_filename, uncompressed_fileinfo): + logger.info(repr(uncompressed_metadata_filename)+' up-to-date.') + return logger.debug('Metadata '+repr(uncompressed_metadata_filename)+\ ' has changed.') try: - self._update_metadata(metadata_role, fileinfo=fileinfo, - compression=compression) + self._update_metadata(metadata_role, uncompressed_fileinfo, compression, + compressed_fileinfo) except: # The current metadata we have is not current but we couldn't # get new metadata. We shouldn't use the old metadata anymore. @@ -1315,13 +1482,15 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas # We shouldn't need to, but we need to check the trust # implications of the current implementation. self._delete_metadata(metadata_role) - logger.error('Metadata for '+str(metadata_role)+' could not be updated') + logger.error('Metadata for '+repr(metadata_role)+' cannot be updated.') raise + else: # We need to remove delegated roles because the delegated roles # may not be trusted anymore. if metadata_role == 'targets' or metadata_role.startswith('targets/'): logger.debug('Removing delegated roles of '+repr(metadata_role)+'.') + # TODO: Should we also remove the keys of the delegated roles? tuf.roledb.remove_delegated_roles(metadata_role) self._import_delegations(metadata_role) @@ -1373,7 +1542,6 @@ def _ensure_all_targets_allowed(self, metadata_role, metadata_object): None. - """ # Return if 'metadata_role' is 'targets'. 'targets' is not @@ -1536,7 +1704,6 @@ def _fileinfo_has_changed(self, metadata_filename, new_fileinfo): Boolean. True if the fileinfo has changed, false otherwise. - """ # If there is no fileinfo currently stored for 'metadata_filename', @@ -1564,8 +1731,9 @@ def _fileinfo_has_changed(self, metadata_filename, new_fileinfo): for algorithm, hash_value in new_fileinfo['hashes'].items(): # We're only looking for a single match. This isn't a security # check, we just want to prevent unnecessary downloads. - if hash_value == current_fileinfo['hashes'][algorithm]: - return False + if algorithm in current_fileinfo['hashes']: + if hash_value == current_fileinfo['hashes'][algorithm]: + return False return True @@ -1595,7 +1763,6 @@ def _update_fileinfo(self, metadata_filename): None. - """ # In case we delayed loading the metadata and didn't do it in @@ -1640,7 +1807,6 @@ def _move_current_to_previous(self, metadata_role): None. - """ # Get the 'current' and 'previous' full file paths for 'metadata_role' @@ -1685,7 +1851,6 @@ def _delete_metadata(self, metadata_role): None. - """ # The root metadata role is never deleted without a replacement. @@ -1723,7 +1888,6 @@ def _ensure_not_expired(self, metadata_role): None. - """ # Construct the full metadata filename and the location of its @@ -1756,8 +1920,11 @@ def all_targets(self): Get a list of the target information for all the trusted targets on the repository. This list also includes all the targets of - delegated roles. The list conforms to 'tuf.formats.TARGETFILES_SCHEMA' + delegated roles. Targets of the list returned are ordered according + the trusted order of the delegated roles, where parent roles come before + children. The list conforms to 'tuf.formats.TARGETFILES_SCHEMA' and has the form: + [{'filepath': 'a/b/c.txt', 'fileinfo': {'length': 13323, 'hashes': {'sha256': dbfac345..}} @@ -1779,7 +1946,6 @@ def all_targets(self): A list of targets, conformant to 'tuf.formats.TARGETFILES_SCHEMA'. - """ # Load the most up-to-date targets of the 'targets' role and all @@ -1787,11 +1953,12 @@ def all_targets(self): self._refresh_targets_metadata(include_delegations=True) all_targets = [] + # Fetch the targets for the 'targets' role. all_targets = self._targets_of_role('targets', skip_refresh=True) - # Fetch the targets for the delegated roles. - for delegated_role in tuf.roledb.get_delegated_rolenames('targets'): + # Fetch the targets of the delegated roles. + for delegated_role in sorted(tuf.roledb.get_delegated_rolenames('targets')): all_targets = self._targets_of_role(delegated_role, all_targets, skip_refresh=True) @@ -1810,7 +1977,7 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals _update_metadata_if_changed('targets') call, not here. Delegated roles are not loaded when the repository is first initialized. They are loaded from disk, updated if they have changed, and stored to the 'self.metadata' - store by this function. This function is called by the target methods, + store by this method. This method is called by the target methods, like all_targets() and targets_of_role(). @@ -1834,7 +2001,6 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals None. - """ roles_to_update = [] @@ -1846,7 +2012,10 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals if metadata_path == rolename + '.txt': roles_to_update.append(metadata_path[:-len('.txt')]) elif include_delegations and metadata_path.startswith(role_prefix): - roles_to_update.append(metadata_path[:-len('.txt')]) + # Add delegated roles. Skip roles names containing compression + # extensions. + if metadata_path.endswith('.txt'): + roles_to_update.append(metadata_path[:-len('.txt')]) # Remove the 'targets' role because it gets updated when the targets.txt # file is updated in _update_metadata_if_changed('targets'). @@ -1885,35 +2054,83 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals def refresh_targets_metadata_chain(self, rolename): """ - Proof-of-concept. + + Refresh the minimum targets metadata of 'rolename'. If 'rolename' is + 'targets/claimed/3.3/django', refresh the metadata of the following roles: + + targets.txt + targets/claimed.txt + targets/claimed/3.3.txt + Note that 'targets/claimed/3.3/django.txt' is not refreshed here. + + The metadata of the 'targets' role is updated in refresh() by the + _update_metadata_if_changed('targets') call, not here. Delegated roles + are not loaded when the repository is first initialized; they can be + loaded from disk, updated if they have changed, and stored to the + 'self.metadata' store by this method. This method may be called + before targets_of_role('rolename') so that the most up-to-date metadata is + available to verify the target files of 'rolename', including the metadata + of 'rolename'. + + + rolename: + This is a full delegated rolename and should not end in '.txt'. + Example: 'targets/linux/x86'. + + + tuf.FormatError: + If any of the arguments are improperly formatted. + + tuf.RepositoryError: + If the metadata of any of the parent roles of 'rolename' is missing + from the 'release.txt' metadata file. + + + The metadata of the parent roles of 'rolename' are loaded from disk and + updated if they have changed. Metadata is removed from the role database + if it has expired. + + + A list of the roles that have been updated, loaded, and are valid. """ + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fail. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + # List of parent roles to update. parent_roles = [] + # Separate each rolename (i.e., each rolename should exclude parent and + # child rolenames). 'rolename' should be the full rolename, as in + # 'targets/linux/x86'. parts = rolename.split('/') # Append the first role to the list. parent_roles.append(parts[0]) - # The 'roles_added' string contains the roles already added. If 'a' and 'a/b' - # have been added to 'parent_roles', 'roles_added' would contain 'a/b' + # The 'roles_added' string contains the roles (full rolename) already added. + # If 'a' and 'a/b' have been added to 'parent_roles', 'roles_added' would + # contain 'a/b'. roles_added = parts[0] # Add each subsequent role to the previous string (with a '/' separator). - # This only goes to -1 because we only want to return the parents (so we + # This only goes to -1 because we only want to store the parents (so we # ignore the last element). for next_role in parts[1:-1]: parent_roles.append(roles_added+'/'+next_role) roles_added = roles_added+'/'+next_role - message = 'Minimum metadata to download to set chain of trust: '+\ + message = 'Minimum metadata to download and set the chain of trust: '+\ repr(parent_roles)+'.' logger.info(message) - # See if this role provides metadata. All the available roles - # on the repository are specified in the 'release.txt' metadata. + # Check if 'release.txt' provides metadata for each of the roles in + # 'parent_roles'. All the available roles on the repository are specified + # in the 'release.txt' metadata. targets_metadata_allowed = self.metadata['current']['release']['meta'].keys() for parent_role in parent_roles: parent_role = parent_role + '.txt' @@ -1921,7 +2138,7 @@ def refresh_targets_metadata_chain(self, rolename): if parent_role not in targets_metadata_allowed: message = '"release.txt" does not provide all the parent roles'+\ 'of '+repr(rolename)+'.' - raise tuf.Repository(message) + raise tuf.RepositoryError(message) # Remove the 'targets' role because it gets updated when the targets.txt # file is updated in _update_metadata_if_changed('targets'). @@ -1940,8 +2157,9 @@ def refresh_targets_metadata_chain(self, rolename): parent_roles.sort() logger.debug('Roles to update: '+repr(parent_roles)+'.') - # Iterate over 'roles_to_update', load its metadata - # file, and update it if it has changed. + # Iterate 'parent_roles', load each role's metadata file from disk, and + # update it if it has changed. + refreshed_chain = [] for rolename in parent_roles: self._load_metadata_from_file('previous', rolename) self._load_metadata_from_file('current', rolename) @@ -1951,9 +2169,11 @@ def refresh_targets_metadata_chain(self, rolename): # Remove the role if it has expired. try: self._ensure_not_expired(rolename) + refreshed_chain.append(rolename) except tuf.ExpiredMetadataError: tuf.roledb.remove_role(rolename) + return refreshed_chain @@ -1965,6 +2185,7 @@ def _targets_of_role(self, rolename, targets=None, skip_refresh=False): Return the target information for all the targets of 'rolename'. The returned information is a list conformant to 'tuf.formats.TARGETFILES_SCHEMA' and has the form: + [{'filepath': 'a/b/c.txt', 'fileinfo': {'length': 13323, 'hashes': {'sha256': dbfac345..}} @@ -1993,7 +2214,6 @@ def _targets_of_role(self, rolename, targets=None, skip_refresh=False): A list of dict objects containing the target information of all the targets of 'rolename'. Conformant to 'tuf.formats.TARGETFILES_SCHEMA'. - """ if targets is None: @@ -2011,7 +2231,7 @@ def _targets_of_role(self, rolename, targets=None, skip_refresh=False): # Do we have metadata for 'rolename'? if rolename not in self.metadata['current']: - message = 'No metadata for '+rolename+'. Unable to determine targets.' + message = 'No metadata for '+repr(rolename)+'. Unable to determine targets.' logger.debug(message) return targets @@ -2035,13 +2255,15 @@ def targets_of_role(self, rolename='targets'): Return a list of trusted targets directly specified by 'rolename'. The returned information is a list conformant to tuf.formats.TARGETFILES_SCHEMA and has the form: + [{'filepath': 'a/b/c.txt', 'fileinfo': {'length': 13323, 'hashes': {'sha256': dbfac345..}} ...] - - This may be a very slow operation if there is a large number of - delegations and many metadata files aren't already downloaded. + + The metadata of 'rolename' is updated if out of date, including the + metadata of its parent roles (i.e., the minimum roles needed to set the + chain of trust). rolename: @@ -2063,15 +2285,15 @@ def targets_of_role(self, rolename='targets'): A list of targets, conformant to 'tuf.formats.TARGETFILES_SCHEMA'. - """ # Does 'rolename' have the correct format? # Raise 'tuf.FormatError' if there is a mismatch. tuf.formats.RELPATH_SCHEMA.check_match(rolename) + self.refresh_targets_metadata_chain(rolename) self._refresh_targets_metadata(rolename) - + return self._targets_of_role(rolename, skip_refresh=True) @@ -2081,7 +2303,8 @@ def targets_of_role(self, rolename='targets'): def target(self, target_filepath): """ - Return the target file information for 'target_filepath'. + Return the target file information of 'target_filepath' and update + its corresponding metadata, if necessary. target_filepath: @@ -2104,12 +2327,15 @@ def target(self, target_filepath): The target information for 'target_filepath', conformant to 'tuf.formats.TARGETFILE_SCHEMA'. - """ # Does 'target_filepath' have the correct format? # Raise 'tuf.FormatError' if there is a mismatch. tuf.formats.RELPATH_SCHEMA.check_match(target_filepath) + + # 'target_filepath' might contain URL encoding escapes. + # http://docs.python.org/2/library/urllib.html#urllib.unquote + target_filepath = urllib.unquote(target_filepath) # Get target by looking at roles in order of priority tags. target = self._preorder_depth_first_walk(target_filepath) @@ -2152,7 +2378,6 @@ def _preorder_depth_first_walk(self, target_filepath): The target information for 'target_filepath', conformant to 'tuf.formats.TARGETFILE_SCHEMA'. - """ target = None @@ -2160,10 +2385,10 @@ def _preorder_depth_first_walk(self, target_filepath): role_names = ['targets'] # Ensure the client has the most up-to-date version of 'targets.txt'. - # Raise 'tuf.NoWorkingMirrorError' if the changed metadata cannot be successfully - # downloaded and 'tuf.RepositoryError' if the referenced metadata is - # missing. Target methods such as this one are called after the top-level - # metadata have been refreshed (i.e., updater.refresh()). + # Raise 'tuf.NoWorkingMirrorError' if the changed metadata cannot be + # successfully downloaded and 'tuf.RepositoryError' if the referenced + # metadata is missing. Target methods such as this one are called after the + # top-level metadata have been refreshed (i.e., updater.refresh()). self._update_metadata_if_changed('targets') # Preorder depth-first traversal of the tree of target delegations. @@ -2234,7 +2459,6 @@ def _get_target_from_targets_role(self, role_name, targets, target_filepath): The target information for 'target_filepath', conformant to 'tuf.formats.TARGETFILE_SCHEMA'. - """ target = None @@ -2266,7 +2490,7 @@ def _visit_child_role(self, child_role, target_filepath): Ensure that we explore only delegated roles trusted with the target. We assume conservation of delegated paths in the complete tree of delegations. Note that the call to _ensure_all_targets_allowed in - __verify_uncompressed_metadata_file should already ensure that all + _verify_uncompressed_metadata_file should already ensure that all targets metadata is valid; i.e. that the targets signed by a delegatee is a proper subset of the targets delegated to it by the delegator. Nevertheless, we check it again here for performance and safety reasons. @@ -2295,7 +2519,6 @@ def _visit_child_role(self, child_role, target_filepath): 'target_filepath', then we return the role name of 'child_role'. Otherwise, we return None. - """ child_role_name = child_role['name'] @@ -2367,7 +2590,6 @@ def _get_target_hash(self, target_filepath, hash_function='sha256'): The hash of 'target_filepath'. - """ # Calculate the hash of the filepath to determine which bin to find the @@ -2417,7 +2639,6 @@ def remove_obsolete_targets(self, destination_directory): None. - """ # Does 'destination_directory' have the correct format? @@ -2459,6 +2680,7 @@ def updated_targets(self, targets, destination_directory): The returned information is a list conformant to 'tuf.formats.TARGETFILES_SCHEMA' and has the form: + [{'filepath': 'a/b/c.txt', 'fileinfo': {'length': 13323, 'hashes': {'sha256': dbfac345..}} @@ -2466,7 +2688,8 @@ def updated_targets(self, targets, destination_directory): targets: - A list of target files. + A list of target files. Targets that come earlier in the list are + chosen over duplicates that may occur later. destination_directory: The directory containing the target files. @@ -2480,7 +2703,6 @@ def updated_targets(self, targets, destination_directory): A list of targets, conformant to 'tuf.formats.TARGETFILES_SCHEMA'. - """ # Do the arguments have the correct format? @@ -2488,13 +2710,20 @@ def updated_targets(self, targets, destination_directory): tuf.formats.TARGETFILES_SCHEMA.check_match(targets) tuf.formats.PATH_SCHEMA.check_match(destination_directory) + # Keep track of the target objects and filepaths of updated targets. + # Return 'updated_targets' and use 'updated_targetpaths' to avoid + # duplicates. updated_targets = [] + updated_targetpaths = [] for target in targets: # Get the target's filepath located in 'destination_directory'. # We will compare targets against this file. target_filepath = os.path.join(destination_directory, target['filepath']) + if target_filepath in updated_targetpaths: + continue + # Try one of the algorithm/digest combos for a mismatch. We break # as soon as we find a mismatch. for algorithm, digest in target['fileinfo']['hashes'].items(): @@ -2502,13 +2731,17 @@ def updated_targets(self, targets, destination_directory): try: digest_object = tuf.hash.digest_filename(target_filepath, algorithm=algorithm) + # This exception would occur if the target does not exist locally. except IOError: updated_targets.append(target) + updated_targetpaths.append(target_filepath) break + # The file does exist locally, check if its hash differs. if digest_object.hexdigest() != digest: updated_targets.append(target) + updated_targetpaths.append(target_filepath) break return updated_targets @@ -2545,7 +2778,6 @@ def download_target(self, target, destination_directory): None. - """ # Do the arguments have the correct format? @@ -2563,8 +2795,8 @@ def download_target(self, target, destination_directory): # get_target_file checks every mirror and returns the first target # that passes verification. - target_file_object = self.get_target_file(target_filepath, trusted_length, - trusted_hashes) + target_file_object = self._get_target_file(target_filepath, trusted_length, + trusted_hashes) # We acquired a target file object from a mirror. Move the file into # place (i.e., locally to 'destination_directory'). @@ -2581,8 +2813,3 @@ def download_target(self, target, destination_directory): logger.warn(str(target_dirpath)+' does not exist.') target_file_object.move(destination) - - - - - diff --git a/tuf/conf.py b/tuf/conf.py index 542bc81b..5f3ce452 100755 --- a/tuf/conf.py +++ b/tuf/conf.py @@ -12,11 +12,12 @@ See LICENSE for licensing information. - A central location for TUF configuration settings. - + A central location for TUF configuration settings. Example options include + setting the destination of temporary files and downloaded content, the maximum + length of downloaded metadata (unknown file attributes), download behavior, + and cryptography libraries clients wish to use. """ - # Set a directory that should be used for all temporary files. If this # is None, then the system default will be used. The system default # will also be used if a directory path set here is invalid or @@ -41,6 +42,12 @@ # default but sane upper bound for the number of bytes required to download it. DEFAULT_TIMESTAMP_REQUIRED_LENGTH = 16384 #bytes +# The Root role may be updated without knowing its hash if top-level metadata +# cannot be safely downloaded (e.g., keys may have been revoked, thus requiring +# a new Root file that includes the updated keys). Set a default upper bound +# for the maximum total bytes that may be downloaded for Root metadata. +DEFAULT_ROOT_REQUIRED_LENGTH = 512000 #bytes + # Set a timeout value in seconds (float) for non-blocking socket operations. SOCKET_TIMEOUT = 1 #seconds @@ -62,6 +69,26 @@ # computational restrictions. A strong user password is still important. # Modifying the number of iterations will result in a new derived key+PBDKF2 # combination if the key is loaded and re-saved, overriding any previous -# iteration setting used by the old '.key'. +# iteration setting used in the old '' key file. # https://en.wikipedia.org/wiki/PBKDF2 PBKDF2_ITERATIONS = 100000 + +# The user client may set the specific cryptography library used by The Update +# Framework updater, or the software updater integrating TUF. +# Supported RSA cryptography libraries: ['pycrypto'] +RSA_CRYPTO_LIBRARY = 'pycrypto' + +# Supported ed25519 cryptography libraries: ['pynacl', 'ed25519'] +ED25519_CRYPTO_LIBRARY = 'ed25519' + +# General purpose cryptography. Algorithms and functions that fall under general +# purpose include AES, PBKDF2, cryptographically strong random number +# generators, and cryptographic hash functions. The majority of the general +# cryptography is needed by the repository and developer tools. +# RSA_CRYPTO_LIBRARY and ED25519_CRYPTO_LIBRARY are needed on the client side +# of the software updater. +GENERAL_CRYPTO_LIBRARY = 'pycrypto' + +# The algorithm in HASH_ALGORITHMS are chosen by the repository tool to generate +# the digests listed in metadata. +REPOSITORY_HASH_ALGORITHMS = ['sha224', 'sha256'] diff --git a/tuf/download.py b/tuf/download.py index 4c3d0006..cd8bf2ac 100755 --- a/tuf/download.py +++ b/tuf/download.py @@ -19,7 +19,6 @@ file-like object that will automatically destroys itself once closed. Note that the file-like object, 'tuf.util.TempFile', is returned by the '_download_file()' function. - """ # Induce "true division" (http://www.python.org/dev/peps/pep-0238/). @@ -106,7 +105,6 @@ def __start_clock(self): None. - """ # We must have reset the clock before this. @@ -139,7 +137,6 @@ def __stop_clock_and_check_speed(self, data_length): None. - """ # We use (platform-specific) wall time, so it will be imprecise sometimes. @@ -201,7 +198,6 @@ def read(self, size): Received data up to 'size' bytes. - """ # We should never try to specify a negative size. @@ -426,7 +422,6 @@ def _open_connection(url): File-like object. - """ # urllib2.Request produces a Request object that allows for a finer control @@ -478,7 +473,6 @@ def _download_fixed_amount_of_data(connection, temp_file, required_length): total_downloaded: The total number of bytes we have downloaded for the desired file and which should be equal to 'required_length'. - """ # Keep track of total bytes downloaded. @@ -538,7 +532,6 @@ def _get_content_length(connection): reported_length: The total number of bytes reported by server. If the process fails, we return None; otherwise we would return a nonnegative integer. - """ try: @@ -559,7 +552,7 @@ def _get_content_length(connection): -def _check_content_length(reported_length, required_length): +def _check_content_length(reported_length, required_length, strict_length=True): """ A helper function that checks whether the length reported by server is @@ -572,6 +565,10 @@ def _check_content_length(reported_length, required_length): required_length: The total number of bytes obtained from (possibly default) metadata. + strict_length: + Boolean that indicates whether the required length of the file is an + exact match, or an upper limit (e.g., downloading a Timestamp file). + No known side effects. @@ -580,21 +577,33 @@ def _check_content_length(reported_length, required_length): None. - """ + logger.debug('The server reported a length of '+repr(reported_length)+' bytes.') + comparison_result = None + try: if reported_length < required_length: - logger.warn('reported_length ('+str(reported_length)+ - ') < required_length ('+str(required_length)+')') + comparison_result = 'less than' + elif reported_length > required_length: - logger.warn('reported_length ('+str(reported_length)+ - ') > required_length ('+str(required_length)+')') + comparison_result = 'greater than' + else: - logger.debug('reported_length ('+str(reported_length)+ - ') == required_length ('+str(required_length)+')') + comparison_result = 'equal to' + except: - logger.exception('Could not check reported and required lengths!') + logger.exception('Could not check reported and required lengths.') + + if strict_length: + message = 'The reported length is '+comparison_result+' the required '+\ + 'length of '+repr(required_length)+' bytes.' + logger.debug(message) + + else: + message = 'The reported length is '+comparison_result+' the upper limit '+\ + 'of '+repr(required_length)+' bytes.' + logger.debug(message) @@ -612,8 +621,11 @@ def _check_downloaded_length(total_downloaded, required_length, The total number of bytes supposedly downloaded for the file in question. required_length: - The total number of bytes expected of the file as seen from its (possibly - default) metadata. + The total number of bytes expected of the file as seen from its metadata. + The Timestamp role is always downloaded without a known file length, and + the Root role when the client cannot download any of the required + top-level roles. In both cases, 'required_length' is actually an upper + limit on the length of the downloaded file. STRICT_REQUIRED_LENGTH: A Boolean indicator used to signal whether we should perform strict @@ -630,30 +642,33 @@ def _check_downloaded_length(total_downloaded, required_length, None. - """ if total_downloaded == required_length: - logger.debug('total_downloaded == required_length == '+ - str(required_length)) + logger.info('Downloaded '+str(total_downloaded)+' bytes out of the '+\ + 'expected '+str(required_length)+ ' bytes.') else: difference_in_bytes = abs(total_downloaded-required_length) - message = 'Downloaded '+str(total_downloaded)+' bytes, but expected '+\ - str(required_length)+' bytes. There is a difference of '+\ - str(difference_in_bytes)+' bytes!' # What we downloaded is not equal to the required length, but did we ask # for strict checking of required length? - if STRICT_REQUIRED_LENGTH: + if STRICT_REQUIRED_LENGTH: + message = 'Downloaded '+str(total_downloaded)+' bytes, but expected '+\ + str(required_length)+' bytes. There is a difference of '+\ + str(difference_in_bytes)+' bytes.' + # This must be due to a programming error, and must never happen! logger.error(message) raise tuf.DownloadLengthMismatchError(required_length, total_downloaded) + else: + message = 'Downloaded '+str(total_downloaded)+' bytes out of an upper '+\ + 'limit of '+str(required_length)+' bytes.' # We specifically disabled strict checking of required length, but we # will log a warning anyway. This is useful when we wish to download the - # timestamp metadata, for which we have no signed metadata; so, we must - # guess a reasonable required_length for it. - logger.warn(message) + # Timestamp or Root metadata, for which we have no signed metadata; so, + # we must guess a reasonable required_length for it. + logger.info(message) @@ -711,7 +726,6 @@ def _download_file(url, required_length, STRICT_REQUIRED_LENGTH=True): A 'tuf.util.TempFile' file-like object which points to the contents of 'url'. - """ # Do all of the arguments have the appropriate format? @@ -748,7 +762,8 @@ def _download_file(url, required_length, STRICT_REQUIRED_LENGTH=True): reported_length = _get_content_length(connection) # Then, we check whether the required length matches the reported length. - _check_content_length(reported_length, required_length) + _check_content_length(reported_length, required_length, + STRICT_REQUIRED_LENGTH) # Download the contents of the URL, up to the required length, to a # temporary file, and get the total number of downloaded bytes. @@ -773,8 +788,3 @@ def _download_file(url, required_length, STRICT_REQUIRED_LENGTH=True): # Restore previously saved values or functions. httplib.HTTPConnection.response_class = previous_http_response_class socket.setdefaulttimeout(previous_socket_timeout) - - - - - diff --git a/tuf/ed25519_key.py b/tuf/ed25519_key.py deleted file mode 100755 index f2583f76..00000000 --- a/tuf/ed25519_key.py +++ /dev/null @@ -1,624 +0,0 @@ -""" - - ed25519_key.py - - - Vladimir Diaz - - - September 24, 2013. - - - See LICENSE for licensing information. - - - The goal of this module is to support ed25519 signatures. ed25519 is an - elliptic-curve public key signature scheme, its main strength being small - signatures (64 bytes) and small public keys (32 bytes). - http://ed25519.cr.yp.to/ - - 'tuf/ed25519_key.py' calls 'ed25519/ed25519.py', which is the pure Python - implementation of ed25519 provided by the author: - http://ed25519.cr.yp.to/software.html - Optionally, ed25519 cryptographic operations may be executed by PyNaCl, which - provides Python bindings to the NaCl library and is much faster than the pure - python implementation. PyNaCl relies on the C library, libsodium. - - https://github.com/dstufft/pynacl - https://github.com/jedisct1/libsodium - http://nacl.cr.yp.to/ - - The ed25519-related functions included here are generate(), create_signature() - and verify_signature(). The 'ed25519' and PyNaCl (i.e., 'nacl') modules used - by ed25519_key.py generate the actual ed25519 keys and the functions listed - above can be viewed as an easy-to-use public interface. Additional functions - contained here include format_keyval_to_metadata() and - format_metadata_to_key(). These last two functions produce or use - ed25519 keys compatible with the key structures listed in TUF Metadata files. - The generate() function returns a dictionary containing all the information - needed of ed25519 keys, such as public/private keys and a keyID identifier. - create_signature() and verify_signature() are supplemental functions used for - generating ed25519 signatures and verifying them. - - Key IDs are used as identifiers for keys (e.g., RSA key). They are the - hexadecimal representation of the hash of key object (specifically, the key - object containing only the public key). Review 'ed25519_key.py' and the - '_get_keyid()' function to see precisely how keyids are generated. One may - get the keyid of a key object by simply accessing the dictionary's 'keyid' - key (i.e., ed25519_key_dict['keyid']). - """ - -# Help with Python 3 compatability, where the print statement is a function, an -# implicit relative import is invalid, and the '/' operator performs true -# division. Example: print 'hello world' raises a 'SyntaxError' exception. -from __future__ import print_function -from __future__ import absolute_import -from __future__ import division - -# Required for hexadecimal conversions. Signatures and public/private keys are -# hexlified. -import binascii - -# Generate OS-specific randomness (os.urandom) suitable for cryptographic use. -# http://docs.python.org/2/library/os.html#miscellaneous-functions -import os - -import tuf - -# Import the python implementation of the ed25519 algorithm that is provided by -# the author. Note: This implementation is very slow and does not include -# protection against side-channel attacks according to the author. Verifying -# signatures can take approximately 9 seconds on an intel core 2 duo @ -# 2.2 ghz x 2). Optionally, the PyNaCl module may be used to speed up ed25519 -# cryptographic operations. -# http://ed25519.cr.yp.to/software.html -# Try to import PyNaCl. The functions found in this module provide the option -# of using PyNaCl over the slower implementation of ed25519. -try: - import nacl.signing - import nacl.encoding -except (ImportError, IOError): - message = 'The PyNacl library and/or its dependencies cannot be imported.' - raise tuf.UnsupportedLibraryError(message) - -# The pure Python implementation of ed25519. -import ed25519.ed25519 - -# Digest objects needed to generate hashes. -import tuf.hash - -# Perform object format-checking. -import tuf.formats - -# The default hash algorithm to use when generating KeyIDs. -_KEY_ID_HASH_ALGORITHM = 'sha256' - -# Supported ed25519 signing methods. 'ed25519-python' is the pure Python -# implementation signing method. 'ed25519-pynacl' (i.e., 'nacl' module) is the -# (libsodium+Python bindings) implementation signing method. -_SUPPORTED_ED25519_SIGNING_METHODS = ['ed25519-python', 'ed25519-pynacl'] - - -def generate(use_pynacl=False): - """ - - Generate an ed25519 seed key ('sk') and public key ('pk'). - In addition, a keyid used as an identifier for ed25519 keys is generated. - The object returned conforms to 'tuf.formats.ED25519KEY_SCHEMA' and has the - form: - {'keytype': 'ed25519', - 'keyid': keyid, - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - The public and private keys are strings. An ed25519 seed key is a random - 32-byte value and public key 32 bytes, although both are hexlified to 64 - bytes. - - >>> ed25519_key = generate() - >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key) - True - >>> len(ed25519_key['keyval']['public']) - 64 - >>> len(ed25519_key['keyval']['private']) - 64 - >>> ed25519_key_pynacl = generate(use_pynacl=True) - >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key_pynacl) - True - >>> len(ed25519_key_pynacl['keyval']['public']) - 64 - >>> len(ed25519_key_pynacl['keyval']['private']) - 64 - - - use_pynacl: - True, if the ed25519 keys should be generated with PyNaCl. False, if the - keys should be generated with the pure Python implementation of ed25519 - (much slower). - - - NotImplementedError, if a randomness source is not found. - - - The ed25519 keys are generated by first creating a random 32-byte value - 'sk' with os.urandom() and then calling ed25519's ed25519.25519.publickey(sk) - or PyNaCl's nacl.signing.SigningKey(). - - - A dictionary containing the ed25519 keys and other identifying information. - Conforms to 'tuf.formats.ED25519KEY_SCHEMA'. - """ - - # Begin building the ed25519 key dictionary. - ed25519_key_dict = {} - keytype = 'ed25519' - - # Generate ed25519's seed key by calling os.urandom(). The random bytes - # returned should be suitable for cryptographic use and is OS-specific. - # Raise 'NotImplementedError' if a randomness source is not found. - # ed25519 seed keys are fixed at 32 bytes (256-bit keys). - # http://blog.mozilla.org/warner/2011/11/29/ed25519-keys/ - seed = os.urandom(32) - public = None - - if use_pynacl: - # Generate the public key. PyNaCl (i.e., 'nacl' module) performs - # the actual key generation. - nacl_key = nacl.signing.SigningKey(seed) - public = str(nacl_key.verify_key) - - # Use the pure Python implementation of ed25519. - else: - public = ed25519.ed25519.publickey(seed) - - # Generate the keyid for the ed25519 key dict. 'key_value' corresponds to the - # 'keyval' entry of the 'ED25519KEY_SCHEMA' dictionary. The seed (private) - # key information is not included in the generation of the 'keyid' identifier. - key_value = {'public': binascii.hexlify(public), - 'private': ''} - keyid = _get_keyid(key_value) - - # Build the 'ed25519_key_dict' dictionary. Update 'key_value' with the - # ed25519 seed key prior to adding 'key_value' to 'ed25519_key_dict'. - key_value['private'] = binascii.hexlify(seed) - - ed25519_key_dict['keytype'] = keytype - ed25519_key_dict['keyid'] = keyid - ed25519_key_dict['keyval'] = key_value - - return ed25519_key_dict - - - - - -def format_keyval_to_metadata(key_value, private=False): - """ - - Return a dictionary conformant to 'tuf.formats.KEY_SCHEMA'. - If 'private' is True, include the private key. The dictionary - returned has the form: - {'keytype': 'ed25519', - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - or if 'private' is False: - - {'keytype': 'ed25519', - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': ''}} - - The private and public keys are 32 bytes, although hexlified to 64 bytes. - - ed25519 keys are stored in Metadata files (e.g., root.txt) in the format - returned by this function. - - >>> ed25519_key = generate() - >>> key_val = ed25519_key['keyval'] - >>> ed25519_metadata = format_keyval_to_metadata(key_val, private=True) - >>> tuf.formats.KEY_SCHEMA.matches(ed25519_metadata) - True - - - key_value: - A dictionary containing a seed and public ed25519 key. - 'key_value' is of the form: - - {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'} - - conformat to 'tuf.formats.KEYVAL_SCHEMA'. - - private: - Indicates if the private key should be included in the - returned dictionary. - - - tuf.FormatError, if 'key_value' does not conform to - 'tuf.formats.KEYVAL_SCHEMA'. - - - None. - - - A 'KEY_SCHEMA' dictionary. - """ - - # Does 'key_value' have the correct format? - # This check will ensure 'key_value' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.KEYVAL_SCHEMA.check_match(key_value) - - if private is True and len(key_value['private']): - return {'keytype': 'ed25519', 'keyval': key_value} - else: - public_key_value = {'public': key_value['public'], 'private': ''} - return {'keytype': 'ed25519', 'keyval': public_key_value} - - - - - -def format_metadata_to_key(key_metadata): - """ - - Construct an ed25519 key dictionary (i.e., tuf.formats.ED25519KEY_SCHEMA) - from 'key_metadata'. The dict returned by this function has the exact - format as the dict returned by generate(). It is of the form: - - {'keytype': 'ed25519', - 'keyid': keyid, - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - The public and private keys are 32-byte strings, although hexlified to 64 - bytes. - - ed25519 key dictionaries in 'ED25519KEY_SCHEMA' format should be used by - modules storing a collection of keys, such as a keydb keystore. - ed25519 keys as stored in metadata files use a different format, so this - function should be called if an ed25519 key is extracted from one of these - metadata files and needs converting. Generate() creates an entirely - new key and returns it in the format appropriate for 'keydb.py' and - 'keystore.py'. - - >>> ed25519_key = generate() - >>> key_val = ed25519_key['keyval'] - >>> ed25519_metadata = format_keyval_to_metadata(key_val, private=True) - >>> ed25519_key_2 = format_metadata_to_key(ed25519_metadata) - >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key_2) - True - >>> ed25519_key == ed25519_key_2 - True - - - key_metadata: - The ed25519 key dictionary as stored in Metadata files, conforming to - 'tuf.formats.KEY_SCHEMA'. It has the form: - - {'keytype': 'ed25519', - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - - tuf.FormatError, if 'key_metadata' does not conform to - 'tuf.formats.KEY_SCHEMA'. - - - None. - - - A dictionary containing the ed25519 keys and other identifying information. - """ - - # Does 'key_metadata' have the correct format? - # This check will ensure 'key_metadata' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.KEY_SCHEMA.check_match(key_metadata) - - # Construct the dictionary to be returned. - ed25519_key_dict = {} - keytype = 'ed25519' - key_value = key_metadata['keyval'] - - # Convert 'key_value' to 'tuf.formats.KEY_SCHEMA' and generate its hash - # The hash is in hexdigest form. _get_keyid() ensures the private key - # information is not included. - keyid = _get_keyid(key_value) - - # We now have all the required key values. Build 'ed25519_key_dict'. - ed25519_key_dict['keytype'] = keytype - ed25519_key_dict['keyid'] = keyid - ed25519_key_dict['keyval'] = key_value - - return ed25519_key_dict - - - - - -def _get_keyid(key_value): - """Return the keyid for 'key_value'.""" - - # 'keyid' will be generated from an object conformant to 'KEY_SCHEMA', - # which is the format Metadata files (e.g., root.txt) store keys. - # 'format_keyval_to_metadata()' returns the object needed by _get_keyid(). - ed25519_key_meta = format_keyval_to_metadata(key_value, private=False) - - # Convert the ed25519 key to JSON Canonical format suitable for adding - # to digest objects. - ed25519_key_update_data = tuf.formats.encode_canonical(ed25519_key_meta) - - # Create a digest object and call update(), using the JSON - # canonical format of 'ed25519_key_meta' as the update data. - digest_object = tuf.hash.digest(_KEY_ID_HASH_ALGORITHM) - digest_object.update(ed25519_key_update_data) - - # 'keyid' becomes the hexadecimal representation of the hash. - keyid = digest_object.hexdigest() - - return keyid - - - - - -def create_signature(ed25519_key_dict, data, use_pynacl=False): - """ - - Return a signature dictionary of the form: - {'keyid': 'a0469d9491e3c0b42dd41fe3455359dbacb3306b6e8fb59...', - 'method': 'ed25519-python', - 'sig': '4b3829671b2c6b90034518a918d2447c722474c878c2431dd...'} - - Note: 'method' may also be 'ed25519-pynacl', if the signature was created - by the 'nacl' module. - - The signing process will use the public and seed key - ed25519_key_dict['keyval']['private'], - ed25519_key_dict['keyval']['public'] - - and 'data' to generate the signature. - - >>> ed25519_key_dict = generate() - >>> data = 'The quick brown fox jumps over the lazy dog.' - >>> signature = create_signature(ed25519_key_dict, data) - >>> tuf.formats.SIGNATURE_SCHEMA.matches(signature) - True - >>> len(signature['sig']) - 128 - >>> signature_pynacl = create_signature(ed25519_key_dict, data, True) - >>> tuf.formats.SIGNATURE_SCHEMA.matches(signature_pynacl) - True - >>> len(signature_pynacl['sig']) - 128 - - - ed25519_key_dict: - A dictionary containing the ed25519 keys and other identifying information. - 'ed25519_key_dict' has the form: - - {'keytype': 'ed25519', - 'keyid': keyid, - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - The public and private keys are 32-byte strings, although hexlified to 64 - bytes. - - data: - Data object used by create_signature() to generate the signature. - - use_pynacl: - True, if the ed25519 signature should be generated with PyNaCl. False, - if the signature should be generated with the pure Python implementation - of ed25519 (much slower). - - - TypeError, if a private key is not defined for 'ed25519_key_dict'. - - tuf.FormatError, if an incorrect format is found for 'ed25519_key_dict'. - - tuf.CryptoError, if a signature cannot be created. - - - ed25519.ed25519.signature() or nacl.signing.SigningKey.sign() called to - generate the actual signature. - - - A signature dictionary conformat to 'tuf.format.SIGNATURE_SCHEMA'. - ed25519 signatures are 64 bytes, however, the hexlified signature - (128 bytes) is stored in the dictionary returned. - """ - - # Does 'ed25519_key_dict' have the correct format? - # This check will ensure 'ed25519_key_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.ED25519KEY_SCHEMA.check_match(ed25519_key_dict) - - # Signing the 'data' object requires a seed and public key. - # 'ed25519.ed25519.py' generates the actual 64-byte signature in pure Python. - # nacl.signing.SigningKey.sign() generates the signature if 'use_pynacl' - # is True. - signature = {} - private_key = ed25519_key_dict['keyval']['private'] - public_key = ed25519_key_dict['keyval']['public'] - private_key = binascii.unhexlify(private_key) - public_key = binascii.unhexlify(public_key) - - keyid = ed25519_key_dict['keyid'] - method = None - sig = None - - # Verify the signature, but only if the private key has been set. The private - # key is a NULL string if unset. Although it may be clearer to explicit check - # that 'private_key' is not '', we can/should check for a value and not - # compare identities with the 'is' keyword. - if len(private_key): - if use_pynacl: - method = 'ed25519-pynacl' - try: - nacl_key = nacl.signing.SigningKey(private_key) - nacl_sig = nacl_key.sign(data) - sig = nacl_sig.signature - except (ValueError, nacl.signing.CryptoError): - message = 'An "ed25519-pynacl" signature could not be created.' - raise tuf.CryptoError(message) - - # Generate an "ed25519-python" (i.e., pure python implementation) signature. - else: - # ed25519.ed25519.signature() requires both the seed and public keys. - # It calculates the SHA512 of the seed key, which is 32 bytes. - method = 'ed25519-python' - try: - sig = ed25519.ed25519.signature(data, private_key, public_key) - except Exception, e: - message = 'An "ed25519-python" signature could not be generated.' - raise tuf.CryptoError(message) - - # Raise an exception since the private key is not defined. - else: - message = 'The required private key is not defined for "ed25519_key_dict".' - raise TypeError(message) - - # Build the signature dictionary to be returned. - # The hexadecimal representation of 'sig' is stored in the signature. - signature['keyid'] = keyid - signature['method'] = method - signature['sig'] = binascii.hexlify(sig) - - return signature - - - - - -def verify_signature(ed25519_key_dict, signature, data, use_pynacl=False): - """ - - Determine whether the seed key belonging to 'ed25519_key_dict' produced - 'signature'. verify_signature() will use the public key found in - 'ed25519_key_dict', the 'method' and 'sig' objects contained in 'signature', - and 'data' to complete the verification. Type-checking performed on both - 'ed25519_key_dict' and 'signature'. - - >>> ed25519_key_dict = generate() - >>> data = 'The quick brown fox jumps over the lazy dog.' - >>> signature = create_signature(ed25519_key_dict, data) - >>> verify_signature(ed25519_key_dict, signature, data) - True - >>> verify_signature(ed25519_key_dict, signature, data, True) - True - >>> bad_data = 'The sly brown fox jumps over the lazy dog.' - >>> bad_signature = create_signature(ed25519_key_dict, bad_data) - >>> verify_signature(ed25519_key_dict, bad_signature, data, True) - False - - - ed25519_key_dict: - A dictionary containing the ed25519 keys and other identifying - information. 'ed25519_key_dict' has the form: - - {'keytype': 'ed25519', - 'keyid': 'a0469d9491e3c0b42dd41fe3455359dbacb3306b6e8fb59...', - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - The public and private keys are 32-byte strings, although hexlified to - 64 bytes. - - signature: - The signature dictionary produced by tuf.ed25519_key.create_signature(). - 'signature' has the form: - - {'keyid': 'a0469d9491e3c0b42dd41fe3455359dbacb3306b6e8fb59...', - 'method': 'ed25519-python', - 'sig': '4b3829671b2c6b90034518a918d2447c722474c878c2431dd...'} - - Conformant to 'tuf.formats.SIGNATURE_SCHEMA'. - - data: - Data object used by tuf.ed25519_key.create_signature() to generate - 'signature'. 'data' is needed here to verify the signature. - - use_pynacl: - True, if the ed25519 signature should be verified with PyNaCl. False, - if the signature should be verified with the pure Python implementation - of ed25519 (much slower). - - - tuf.UnknownMethodError. Raised if the signing method used by - 'signature' is not one supported by tuf.ed25519_key.create_signature(). - - tuf.FormatError. Raised if either 'ed25519_key_dict' - or 'signature' do not match their respective tuf.formats schema. - 'ed25519_key_dict' must conform to 'tuf.formats.ED25519KEY_SCHEMA'. - 'signature' must conform to 'tuf.formats.SIGNATURE_SCHEMA'. - - - ed25519.ed25519.checkvalid() called to do the actual verification. - nacl.signing.VerifyKey.verify() called if 'use_pynacl' is True. - - - Boolean. True if the signature is valid, False otherwise. - """ - - # Does 'ed25519_key_dict' have the correct format? - # This check will ensure 'ed25519_key_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.ED25519KEY_SCHEMA.check_match(ed25519_key_dict) - - # Does 'signature' have the correct format? - tuf.formats.SIGNATURE_SCHEMA.check_match(signature) - - # Using the public key belonging to 'ed25519_key_dict' - # (i.e., ed25519_key_dict['keyval']['public']), verify whether 'signature' - # was produced by ed25519_key_dict's corresponding seed key - # ed25519_key_dict['keyval']['private']. Before returning the Boolean result, - # ensure 'ed25519-python' or 'ed25519-pynacl' was used as the signing method. - method = signature['method'] - sig = signature['sig'] - sig = binascii.unhexlify(sig) - public = ed25519_key_dict['keyval']['public'] - public = binascii.unhexlify(public) - valid_signature = False - - if method in _SUPPORTED_ED25519_SIGNING_METHODS: - if use_pynacl: - try: - nacl_verify_key = nacl.signing.VerifyKey(public) - nacl_message = nacl_verify_key.verify(data, sig) - if nacl_message == data: - valid_signature = True - except nacl.signing.BadSignatureError: - pass - - # Verify signature with 'ed25519-python' (i.e., pure Python implementation). - else: - try: - ed25519.ed25519.checkvalid(sig, data, public) - valid_signature = True - - # The pure Python implementation raises 'Exception' if 'signature' is - # invalid. - except Exception, e: - pass - else: - message = 'Unsupported ed25519 signing method: '+repr(method)+'.\n'+ \ - 'Supported methods: '+repr(_SUPPORTED_ED25519_SIGNING_METHODS)+'.' - raise tuf.UnknownMethodError(message) - - return valid_signature - - - -if __name__ == '__main__': - # The interactive sessions of the documentation strings can - # be tested by running 'ed25519_key.py' as a standalone module. - # python -B ed25519_key.py - import doctest - doctest.testmod() diff --git a/tuf/ed25519_keys.py b/tuf/ed25519_keys.py new file mode 100755 index 00000000..5a8303ec --- /dev/null +++ b/tuf/ed25519_keys.py @@ -0,0 +1,411 @@ +""" + + ed25519_keys.py + + + Vladimir Diaz + + + September 24, 2013. + + + See LICENSE for licensing information. + + + The goal of this module is to support ed25519 signatures. ed25519 is an + elliptic-curve public key signature scheme, its main strength being small + signatures (64 bytes) and small public keys (32 bytes). + http://ed25519.cr.yp.to/ + + 'tuf/ed25519_keys.py' calls 'tuf._vendor.ed25519/ed25519.py', which is the + pure Python implementation of ed25519 optimized for a faster runtime. + The Python reference implementation is concise, but very slow (verifying + signatures takes ~9 seconds on an Intel core 2 duo @ 2.2 ghz x 2). The + optimized version can verify signatures in ~2 seconds. + + http://ed25519.cr.yp.to/software.html + https://github.com/pyca/ed25519 + + Optionally, ed25519 cryptographic operations may be executed by PyNaCl, which + is a Python binding to the NaCl library and is faster than the pure python + implementation. Verifying signatures can take approximately 0.0009 seconds. + PyNaCl relies on the libsodium C library. + + https://github.com/pyca/pynacl + https://github.com/jedisct1/libsodium + http://nacl.cr.yp.to/ + + The ed25519-related functions included here are generate(), create_signature() + and verify_signature(). The 'ed25519' and PyNaCl (i.e., 'nacl') modules used + by ed25519_keys.py generate the actual ed25519 keys and the functions listed + above can be viewed as an easy-to-use public interface. + """ + +# Help with Python 3 compatibility, where the print statement is a function, an +# implicit relative import is invalid, and the '/' operator performs true +# division. Example: print 'hello world' raises a 'SyntaxError' exception. +from __future__ import print_function +from __future__ import absolute_import +from __future__ import division + +# 'binascii' required for hexadecimal conversions. Signatures and +# public/private keys are hexlified. +import binascii + +# 'os' required to generate OS-specific randomness (os.urandom) suitable for +# cryptographic use. +# http://docs.python.org/2/library/os.html#miscellaneous-functions +import os + +# Import the python implementation of the ed25519 algorithm provided by pyca, +# which is an optimized version of the one provided by ed25519's authors. +# Note: The pure Python version do not include protection against side-channel +# attacks. Verifying signatures can take approximately 2 seconds on a intel +# core 2 duo @ 2.2 ghz x 2). Optionally, the PyNaCl module may be used to +# speed up ed25519 cryptographic operations. +# http://ed25519.cr.yp.to/software.html +# https://github.com/pyca/ed25519 +# https://github.com/pyca/pynacl +# +# PyNaCl's 'cffi' dependency may thrown an 'IOError' exception when +# importing 'nacl.signing'. +try: + import nacl.signing + import nacl.encoding +except (ImportError, IOError): + pass + +# The optimized pure Python implementation of ed25519 provided by TUF. If +# PyNaCl cannot be imported and an attempt to use is made in this module, a +# 'tuf.UnsupportedLibraryError' exception is raised. +import tuf._vendor.ed25519.ed25519 + +import tuf + +# Digest objects needed to generate hashes. +import tuf.hash + +# Perform object format-checking. +import tuf.formats + +# Supported ed25519 signing method: 'ed25519'. The pure Python +# implementation (i.e., 'tuf._vendor.ed25519.ed25519') and PyNaCl +# (i.e., 'nacl', libsodium+Python bindgs) modules are currently supported in +# the creationg of 'ed25519' signatures. Previously, a distinction was made +# between signatures made by the pure Python implementation and PyNaCl. +_SUPPORTED_ED25519_SIGNING_METHODS = ['ed25519',] + + +def generate_public_and_private(use_pynacl=False): + """ + + Generate a pair of ed25519 public and private keys. + The public and private keys returned conform to + 'tuf.formats.ED25519PULIC_SCHEMA' and 'tuf.formats.ED25519SEED_SCHEMA', + respectively, and have the form: + + '\xa2F\x99\xe0\x86\x80%\xc8\xee\x11\xb95T\xd9\...' + + An ed25519 seed key is a random 32-byte string. Public keys are also 32 + bytes. + + >>> public, private = generate_public_and_private(use_pynacl=False) + >>> tuf.formats.ED25519PUBLIC_SCHEMA.matches(public) + True + >>> tuf.formats.ED25519SEED_SCHEMA.matches(private) + True + >>> public, private = generate_public_and_private(use_pynacl=True) + >>> tuf.formats.ED25519PUBLIC_SCHEMA.matches(public) + True + >>> tuf.formats.ED25519SEED_SCHEMA.matches(private) + True + + + use_pynacl: + True, if the ed25519 keys should be generated with PyNaCl. False, if the + keys should be generated with the pure Python implementation of ed25519 + (slower). + + + tuf.FormatError, if 'use_pynacl' is not a Boolean. + + tuf.UnsupportedLibraryError, if the PyNaCl ('nacl') module is unavailable + and 'use_pynacl' is True. + + NotImplementedError, if a randomness source is not found by 'os.urandom'. + + + The ed25519 keys are generated by first creating a random 32-byte seed + with os.urandom() and then calling ed25519's + ed25519.25519.publickey(seed) or PyNaCl's nacl.signing.SigningKey(). + + + A (public, private) tuple that conform to 'tuf.formats.ED25519PUBLIC_SCHEMA' + and 'tuf.formats.ED25519SEED_SCHEMA', respectively. + """ + + # Does 'use_pynacl' have the correct format? + # This check will ensure 'use_pynacl' conforms to 'tuf.formats.BOOLEAN_SCHEMA'. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.BOOLEAN_SCHEMA.check_match(use_pynacl) + + # Generate ed25519's seed key by calling os.urandom(). The random bytes + # returned should be suitable for cryptographic use and is OS-specific. + # Raise 'NotImplementedError' if a randomness source is not found. + # ed25519 seed keys are fixed at 32 bytes (256-bit keys). + # http://blog.mozilla.org/warner/2011/11/29/ed25519-keys/ + seed = os.urandom(32) + public = None + + if use_pynacl: + # Generate the public key. PyNaCl (i.e., 'nacl' module) performs + # the actual key generation. + try: + nacl_key = nacl.signing.SigningKey(seed) + public = str(nacl_key.verify_key) + except NameError: + message = 'The PyNaCl library and/or its dependencies unavailable.' + raise tuf.UnsupportedLibraryError(message) + + # Use the pure Python implementation of ed25519. + else: + public = tuf._vendor.ed25519.ed25519.publickey(seed) + + return public, seed + + + + + +def create_signature(public_key, private_key, data, use_pynacl=False): + """ + + Return a (signature, method) tuple, where the method is 'ed25519' and + generated by either the pure python implemenation, or by PyNaCl + (i.e., 'nacl'). The signature returns conforms to + 'tuf.formats.ED25519SIGNATURE_SCHEMA', and has the form: + + '\xae\xd7\x9f\xaf\x95{bP\x9e\xa8YO Z\x86\x9d...' + + A signature is a 64-byte string. + + >>> public, private = generate_public_and_private(use_pynacl=False) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature, method = \ + create_signature(public, private, data, use_pynacl=False) + >>> tuf.formats.ED25519SIGNATURE_SCHEMA.matches(signature) + True + >>> method == 'ed25519' + True + >>> signature, method = \ + create_signature(public, private, data, use_pynacl=True) + >>> tuf.formats.ED25519SIGNATURE_SCHEMA.matches(signature) + True + >>> method == 'ed25519' + True + + + public: + The ed25519 public key, which is a 32-byte string. + + private: + The ed25519 private key, which is a 32-byte string. + + data: + Data object used by create_signature() to generate the signature. + + use_pynacl: + True, if the ed25519 signature should be generated with PyNaCl. False, + if the signature should be generated with the pure Python implementation + of ed25519 (much slower). + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if a signature cannot be created. + + + tuf._vendor.ed25519.ed25519.signature() or nacl.signing.SigningKey.sign() + called to generate the actual signature. + + + A signature dictionary conformat to 'tuf.format.SIGNATURE_SCHEMA'. + ed25519 signatures are 64 bytes, however, the hexlified signature is + stored in the dictionary returned. + """ + + # Does 'public_key' have the correct format? + # This check will ensure 'public_key' conforms to + # 'tuf.formats.ED25519PUBLIC_SCHEMA', which must have length 32 bytes. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ED25519PUBLIC_SCHEMA.check_match(public_key) + + # Is 'private_key' properly formatted? + tuf.formats.ED25519SEED_SCHEMA.check_match(private_key) + + # Is 'use_pynacl' properly formatted? + tuf.formats.BOOLEAN_SCHEMA.check_match(use_pynacl) + + # Signing the 'data' object requires a seed and public key. + # 'tuf._vendor.ed25519.ed25519.py' generates the actual 64-byte signature in + # pure Python. nacl.signing.SigningKey.sign() generates the signature if + # 'use_pynacl' is True. + public = public_key + private = private_key + + method = None + signature = None + + # The private and public keys have been validated above by 'tuf.formats' and + # should be 32-byte strings. + if use_pynacl: + method = 'ed25519' + try: + nacl_key = nacl.signing.SigningKey(private) + nacl_sig = nacl_key.sign(data) + signature = nacl_sig.signature + + except NameError: + message = 'The PyNaCl library and/or its dependencies unavailable.' + raise tuf.UnsupportedLibraryError(message) + + except (ValueError, nacl.signing.CryptoError): + message = 'An "ed25519" signature could not be created with PyNaCl.' + raise tuf.CryptoError(message) + + # Generate an "ed25519" signature with the pure python implementation. + else: + # tuf._vendor.ed25519.ed25519.signature() requires both the seed and + # public keys. It calculates the SHA512 of the seed key, which is 32 bytes. + method = 'ed25519' + try: + signature = tuf._vendor.ed25519.ed25519.signature(data, private, public) + + # 'Exception' raised by ed25519.py for any exception that may occur. + except Exception, e: + message = 'An "ed25519" signature could not be generated in pure Python.' + raise tuf.CryptoError(message) + + return signature, method + + + + + +def verify_signature(public_key, method, signature, data, use_pynacl=False): + """ + + Determine whether the private key corresponding to 'public_key' produced + 'signature'. verify_signature() will use the public key, the 'method' and + 'sig', and 'data' arguments to complete the verification. + + >>> public, private = generate_public_and_private(use_pynacl=False) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature, method = \ + create_signature(public, private, data, use_pynacl=False) + >>> verify_signature(public, method, signature, data, use_pynacl=False) + True + >>> verify_signature(public, method, signature, data, use_pynacl=True) + True + >>> bad_data = 'The sly brown fox jumps over the lazy dog' + >>> bad_signature, method = \ + create_signature(public, private, bad_data, use_pynacl=False) + >>> verify_signature(public, method, bad_signature, data, use_pynacl=False) + False + + + public_key: + The public key is a 32-byte string. + + method: + 'ed25519' signature method generated by either the pure python + implementation (i.e., 'tuf._vendor.ed25519.ed25519.py') or PyNacl + (i.e., 'nacl'). + + signature: + The signature is a 64-byte string. + + data: + Data object used by tuf.ed25519_keys.create_signature() to generate + 'signature'. 'data' is needed here to verify the signature. + + use_pynacl: + True, if the ed25519 signature should be verified by PyNaCl. False, + if the signature should be verified with the pure Python implementation + of ed25519 (slower). + + + tuf.UnknownMethodError. Raised if the signing method used by + 'signature' is not one supported by tuf.ed25519_keys.create_signature(). + + tuf.FormatError. Raised if the arguments are improperly formatted. + + + tuf._vendor.ed25519.ed25519.checkvalid() called to do the actual + verification. nacl.signing.VerifyKey.verify() called if 'use_pynacl' is + True. + + + Boolean. True if the signature is valid, False otherwise. + """ + + # Does 'public_key' have the correct format? + # This check will ensure 'public_key' conforms to + # 'tuf.formats.ED25519PUBLIC_SCHEMA', which must have length 32 bytes. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ED25519PUBLIC_SCHEMA.check_match(public_key) + + # Is 'method' properly formatted? + tuf.formats.NAME_SCHEMA.check_match(method) + + # Is 'signature' properly formatted? + tuf.formats.ED25519SIGNATURE_SCHEMA.check_match(signature) + + # Is 'use_pynacl' properly formatted? + tuf.formats.BOOLEAN_SCHEMA.check_match(use_pynacl) + + # Verify 'signature'. Before returning the Boolean result, + # ensure 'ed25519' was used as the signing method. + # Raise 'tuf.UnsupportedLibraryError' if 'use_pynacl' is True but 'nacl' is + # unavailable. + public = public_key + valid_signature = False + + if method in _SUPPORTED_ED25519_SIGNING_METHODS: + if use_pynacl: + try: + nacl_verify_key = nacl.signing.VerifyKey(public) + nacl_message = nacl_verify_key.verify(data, signature) + if nacl_message == data: + valid_signature = True + except NameError: + message = 'The PyNaCl library and/or its dependencies unavailable.' + raise tuf.UnsupportedLibraryError(message) + except nacl.signing.BadSignatureError: + pass + + # Verify 'ed25519' signature with pure Python implementation. + else: + try: + tuf._vendor.ed25519.ed25519.checkvalid(signature, data, public) + valid_signature = True + + # The pure Python implementation raises 'Exception' if 'signature' is + # invalid. + except Exception, e: + pass + else: + message = 'Unsupported ed25519 signing method: '+repr(method)+'.\n'+ \ + 'Supported methods: '+repr(_SUPPORTED_ED25519_SIGNING_METHODS)+'.' + raise tuf.UnknownMethodError(message) + + return valid_signature + + + +if __name__ == '__main__': + # The interactive sessions of the documentation strings can + # be tested by running 'ed25519_keys.py' as a standalone module. + # python -B ed25519_keys.py + import doctest + doctest.testmod() diff --git a/tuf/evp.py b/tuf/evp.py new file mode 100755 index 00000000..60895bbd --- /dev/null +++ b/tuf/evp.py @@ -0,0 +1,425 @@ +""" + + evp.py + + + Vladimir Diaz + + + October 2013. + + + See LICENSE for licensing information. + + + The goal of this module is to support public-key cryptography using + the RSA algorithm. The RSA-related functions provided include + generate(), create_signature(), and verify_signature(). The 'evpy' package + used by 'rsa_key.py' generates the actual RSA keys and the functions listed + above can be viewed as an easy-to-use public interface. Additional functions + contained here include create_in_metadata_format() and + create_from_metadata_format(). These last two functions produce or use RSA + keys compatible with the key structures listed in TUF Metadata files. + The generate() function returns a dictionary containing all the information + needed of RSA keys, such as public and private keys, keyIDs, and an iden- + fier. create_signature() and verify_signature() are supplemental functions + used for generating RSA signatures and verifying them. + + Key IDs are used as identifiers for keys (e.g., RSA key). They are the + hexadecimal representation of the hash of key object (specifically, the key + object containing only the public key). See 'rsa_key.py' and the + '_get_keyid()' function to see precisely how keyids are generated. One may + get the keyid of a key object by simply accessing the dictionary's 'keyid' + key (i.e., rsakey['keyid']). + +""" + + +# Required for hexadecimal conversions. +import binascii + +# Needed to generate, sign, and verify RSA keys. +import evpy.signature +import evpy.envelope + +# Digest objects needed to generate hashes. +import tuf.hash + +# Perform object format-checking. +import tuf.formats + + +_KEY_ID_HASH_ALGORITHM = 'sha256' + +# Recommended RSA key sizes: http://www.rsa.com/rsalabs/node.asp?id=2004 +# According to the document above, revised May 6, 2003, RSA keys of +# size 3072 provide security through 2031 and beyond. +_DEFAULT_RSA_KEY_BITS = 3072 + + +def generate(bits=_DEFAULT_RSA_KEY_BITS): + """ + + Generate public and private RSA keys, with modulus length 'bits'. + In addition, a keyid used as an identifier for RSA keys is generated. + The object returned conforms to tuf.formats.RSAKEY_SCHEMA and as the form: + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are in PEM format and stored as strings. + + + bits: + The key size, or key length, of the RSA key. + + + tuf.CryptoError, if an exception occurs after calling evpy.envelope.keygen(). + + tuf.FormatError, if 'bits' does not contain the correct format. + + + The RSA keys are generated by calling evpy.envelope.keygen(). + + + A dictionary containing the RSA keys and other identifying information. + + """ + + + # Does 'bits' have the correct format? + # This check will ensure 'bits' conforms to 'tuf.formats.RSAKEYBITS_SCHEMA'. + # 'bits' must be an integer object, with a minimum value of 2048. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) + + # Begin building the RSA key dictionary. + rsakey_dict = {} + keytype = 'rsa' + + # Generate the public and private keys. 'public_key' and 'private_key' + # will both be strings containing RSA keys in PEM format. + # The evpy.envelope module performs the actual key generation. The + # evpy.envelope.keygen() function returns a (public, private) tuple. + + try: + public_key, private_key = evpy.envelope.keygen(bits, pem=True) + except (evpy.envelope.EnvelopeError, evpy.envelope.KeygenError, MemoryError), e: + raise tuf.CryptoError(e) + + # Generate the keyid for the RSA key. 'key_value' corresponds to the + # 'keyval' entry of the RSAKEY_SCHEMA dictionary. + key_value = {'public': public_key, + 'private': ''} + + keyid = _get_keyid(key_value) + + # Build the 'rsakey_dict' dictionary. + # Update 'key_value' with the RSA private key prior to adding + # 'key_value' to 'rsakey_dict'. + key_value['private'] = private_key + + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def create_in_metadata_format(key_value, private=False): + """ + + Return a dictionary conformant to tuf.formats.KEY_SCHEMA. + If 'private' is True, include the private key. The dictionary + returned has the form: + {'keytype': 'rsa', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + or + + {'keytype': 'rsa', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': ''}} if 'private' is False. + + The private and public keys are in PEM format. + + RSA keys are stored in Metadata files (e.g., root.txt) in the format + returned by this function. + + + key_value: + A dictionary containing a private and public RSA key. + 'key_value' is of the form: + + {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}}, + conformat to tuf.formats.KEYVAL_SCHEMA. + + private: + Indicates if the private key should be included in the + returned dictionary. + + + tuf.FormatError, if 'key_value' does not conform to + tuf.formats.KEYVAL_SCHEMA. + + + None. + + + An KEY_SCHEMA dictionary. + + """ + + + # Does 'key_value' have the correct format? + # This check will ensure 'key_value' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.KEYVAL_SCHEMA.check_match(key_value) + + if private and key_value['private']: + return {'keytype': 'rsa', 'keyval': key_value} + else: + public_key_value = {'public': key_value['public'], 'private': ''} + return {'keytype': 'rsa', 'keyval': public_key_value} + + + + + +def create_from_metadata_format(key_metadata): + """ + + Construct an RSA key dictionary (i.e., tuf.formats.RSAKEY_SCHEMA) + from 'key_metadata'. The dict returned by this function has the exact + format as the dict returned by generate(). It is of the form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are in PEM format and stored as strings. + + RSA key dictionaries in RSAKEY_SCHEMA format should be used by + modules storing a collection of keys, such as a keydb and keystore. + RSA keys as stored in metadata files use a different format, so this + function should be called if an RSA key is extracted from one of these + metadata files and needs converting. Generate() creates an entirely + new key and returns it in the format appropriate for keydb and keystore. + + + key_metadata: + The RSA key dictionary as stored in Metadata files, conforming to + tuf.formats.KEY_SCHEMA. + + + tuf.FormatError, if 'key_metadata' does not conform to + tuf.formats.KEY_SCHEMA. + + + None. + + + A dictionary containing the RSA keys and other identifying information. + + """ + + + # Does 'key_metadata' have the correct format? + # This check will ensure 'key_metadata' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.KEY_SCHEMA.check_match(key_metadata) + + # Construct the dictionary to be returned. + rsakey_dict = {} + keytype = 'rsa' + key_value = key_metadata['keyval'] + + keyid = _get_keyid(key_value) + + # We now have all the required key values. + # Build 'rsakey_dict'. + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def _get_keyid(key_value): + """Return the keyid for 'key_value'.""" + + # 'keyid' will be generated from an object conformant to KEY_SCHEMA, + # which is the format Metadata files (e.g., root.txt) store keys. + # 'create_in_metadata_format()' returns the object needed by _get_keyid(). + rsakey_meta = create_in_metadata_format(key_value, private=False) + + # Convert the RSA key to JSON Canonical format suitable for adding + # to digest objects. + rsakey_update_data = tuf.formats.encode_canonical(rsakey_meta) + + # Create a digest object and call update(), using the JSON + # canonical format of 'rskey_meta' as the update data. + digest_object = tuf.hash.digest(_KEY_ID_HASH_ALGORITHM) + digest_object.update(rsakey_update_data) + + # 'keyid' becomes the hexadecimal representation of the hash. + keyid = digest_object.hexdigest() + + return keyid + + + + + +def create_signature(rsakey_dict, data): + """ + + Return a signature dictionary of the form: + {'keyid': keyid, 'method': 'evp', 'sig': sig}. + + The signing process will use the private key + rsakey_dict['keyval']['private'] and 'data' to generate the signature. + + + rsakey_dict: + A dictionary containing the RSA keys and other identifying information. + 'rsakey_dict' has the form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are in PEM format and stored as strings. + + data: + Data object used by create_signature() to generate the signature. + + + TypeError, if a private key is not defined for 'rsakey_dict'. + + tuf.FormatError, if an incorrect format is found for the + 'rsakey_dict' object. + + + evpy.signature.sign() called to perform the actual signing. + + + A signature dictionary conformat to tuf.format.SIGNATURE_SCHEMA. + + """ + + + # Does 'rsakey_dict' have the correct format? + # This check will ensure 'rsakey_dict' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) + + # Signing the 'data' object requires a private key. + # The 'evp' (i.e., evpy) signing method is the only method + # currently supported. + signature = {} + private_key = rsakey_dict['keyval']['private'] + keyid = rsakey_dict['keyid'] + method = 'evp' + + if private_key: + sig = evpy.signature.sign(data, key=private_key) + else: + raise TypeError('The required private key is not defined for rsakey_dict.') + + # Build the signature dictionary to be returned. + # The hexadecimal representation of 'sig' is stored in the signature. + signature['keyid'] = keyid + signature['method'] = method + signature['sig'] = binascii.hexlify(sig) + + return signature + + + + + +def verify_signature(rsakey_dict, signature, data): + """ + + Determine whether the private key belonging to 'rsakey_dict' produced + 'signature'. verify_signature() will use the public key found in + 'rsakey_dict', the 'method' and 'sig' objects contained in 'signature', + and 'data' to complete the verification. Type-checking performed on both + 'rsakey_dict' and 'signature'. + + + rsakey_dict: + A dictionary containing the RSA keys and other identifying information. + 'rsakey_dict' has the form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are in PEM format and stored as strings. + + signature: + The signature dictionary produced by tuf.rsa_key.create_signature(). + 'signature' has the form: + {'keyid': keyid, 'method': 'method', 'sig': sig}. Conformant to + tuf.formats.SIGNATURE_SCHEMA. + + data: + Data object used by tuf.rsa_key.create_signature() to generate + 'signature'. 'data' is needed here to verify the signature. + + + tuf.UnknownMethodError. Raised if the signing method used by + 'signature' is not one supported by tuf.rsa_key.create_signature(). + + tuf.FormatError. Raised if either 'rsakey_dict' + or 'signature' do not match their respective tuf.formats schema. + 'rsakey_dict' must conform to tuf.formats.RSAKEY_SCHEMA. + 'signature' must conform to tuf.formats.SIGNATURE_SCHEMA. + + + evpy.signature_verify() called to do the actual verification. + + + Boolean. + + """ + + + # Does 'rsakey_dict' have the correct format? + # This check will ensure 'rsakey_dict' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) + + # Does 'signature' have the correct format? + tuf.formats.SIGNATURE_SCHEMA.check_match(signature) + + # Using the public key belonging to 'rsakey_dict' + # (i.e., rsakey_dict['keyval']['public']), verify whether 'signature' + # was produced by rsakey_dict's corresponding private key + # rsakey_dict['keyval']['private']. Before returning the Boolean result, + # ensure 'evp' was used as the signing method. + + method = signature['method'] + sig = signature['sig'] + public_key = rsakey_dict['keyval']['public'] + + if method != 'evp': + raise tuf.UnknownMethodError(method) + return evpy.signature.verify(data, binascii.unhexlify(sig), key=public_key) diff --git a/tuf/formats.py b/tuf/formats.py index 85fc37c2..0a6f29b3 100755 --- a/tuf/formats.py +++ b/tuf/formats.py @@ -57,10 +57,8 @@ Example: signable_object = make_signable(unsigned_object) - """ - import binascii import calendar import re @@ -76,15 +74,19 @@ # easily backwards compatible with clients that are already deployed. # A date in 'YYYY-MM-DD HH:MM:SS UTC' format. +# TODO: Support timestamps according to the ISO 8601 standard. TIME_SCHEMA = SCHEMA.RegularExpression(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} UTC') +# A date in 'YYYY-MM-DD HH:MM:SS UTC' format. +DATETIME_SCHEMA = SCHEMA.RegularExpression(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}') + # A hexadecimal value in '23432df87ab..' format. HASH_SCHEMA = SCHEMA.RegularExpression(r'[a-fA-F0-9]+') # A dict in {'sha256': '23432df87ab..', 'sha512': '34324abc34df..', ...} format. HASHDICT_SCHEMA = SCHEMA.DictOf( - key_schema=SCHEMA.AnyString(), - value_schema=HASH_SCHEMA) + key_schema = SCHEMA.AnyString(), + value_schema = HASH_SCHEMA) # A hexadecimal value in '23432df87ab..' format. HEX_SCHEMA = SCHEMA.RegularExpression(r'[a-fA-F0-9]+') @@ -93,7 +95,7 @@ KEYID_SCHEMA = HASH_SCHEMA KEYIDS_SCHEMA = SCHEMA.ListOf(KEYID_SCHEMA) -# The method used for a generated signature (e.g., 'evp'). +# The method used for a generated signature (e.g., 'RSASSA-PSS'). SIG_METHOD_SCHEMA = SCHEMA.AnyString() # A relative file path (e.g., 'metadata/root/'). @@ -109,14 +111,14 @@ # A dictionary holding version information. VERSION_SCHEMA = SCHEMA.Object( - object_name='version', - major=SCHEMA.Integer(lo=0), - minor=SCHEMA.Integer(lo=0), - fix=SCHEMA.Integer(lo=0)) + object_name = 'VERSION_SCHEMA', + major = SCHEMA.Integer(lo=0), + minor = SCHEMA.Integer(lo=0), + fix = SCHEMA.Integer(lo=0)) # An integer representing the numbered version of a metadata file. # Must be 1, or greater. -METADATAVERSION_SCHEMA = SCHEMA.Integer(lo=1) +METADATAVERSION_SCHEMA = SCHEMA.Integer(lo=0) # An integer representing length. Must be 0, or greater. LENGTH_SCHEMA = SCHEMA.Integer(lo=0) @@ -127,9 +129,20 @@ # A string representing a named object. NAME_SCHEMA = SCHEMA.AnyString() +NAMES_SCHEMA = SCHEMA.ListOf(NAME_SCHEMA) + +# Supported hash algorithms. +HASHALGORITHMS_SCHEMA = SCHEMA.ListOf(SCHEMA.OneOf( + [SCHEMA.String('md5'), SCHEMA.String('sha1'), + SCHEMA.String('sha224'), SCHEMA.String('sha256'), + SCHEMA.String('sha384'), SCHEMA.String('sha512')])) + +# The contents of an encrypted TUF key. Encrypted TUF keys are saved to files +# in this format. +ENCRYPTEDKEY_SCHEMA = SCHEMA.AnyString() # A value that is either True or False, on or off, etc. -TOGGLE_SCHEMA = SCHEMA.Boolean() +BOOLEAN_SCHEMA = SCHEMA.Boolean() # A role's threshold value (i.e., the minimum number # of signatures required to sign a metadata file). @@ -142,6 +155,12 @@ # The minimum number of bits for an RSA key. Must be 2048 bits and greater. RSAKEYBITS_SCHEMA = SCHEMA.Integer(lo=2048) +# The number of bins used to delegate to hashed roles. +NUMBINS_SCHEMA = SCHEMA.Integer(lo=16) + +# A PyCrypto signature. +PYCRYPTOSIGNATURE_SCHEMA = SCHEMA.AnyString() + # An RSA key in PEM format. PEMRSA_SCHEMA = SCHEMA.AnyString() @@ -155,50 +174,76 @@ # key identifier ('rsa', 233df889cb). For RSA keys, the key value is a pair of # public and private keys in PEM Format stored as strings. KEYVAL_SCHEMA = SCHEMA.Object( - object_name='keyval', - public=SCHEMA.AnyString(), - private=SCHEMA.AnyString()) + object_name = 'KEYVAL_SCHEMA', + public = SCHEMA.AnyString(), + private = SCHEMA.AnyString()) -# A generic key. All TUF keys should be saved to metadata files in this format. +# Supported TUF key types. +KEYTYPE_SCHEMA = SCHEMA.OneOf( + [SCHEMA.String('rsa'), SCHEMA.String('ed25519')]) + +# A generic TUF key. All TUF keys should be saved to metadata files in this +# format. KEY_SCHEMA = SCHEMA.Object( - object_name='key', - keytype=SCHEMA.AnyString(), - keyval=KEYVAL_SCHEMA) + object_name = 'KEY_SCHEMA', + keytype = SCHEMA.AnyString(), + keyval = KEYVAL_SCHEMA) -# An RSA key. +# A TUF key object. This schema simplifies validation of keys that may be +# one of the supported key types. +# Supported key types: 'rsa', 'ed25519'. +ANYKEY_SCHEMA = SCHEMA.Object( + object_name = 'ANYKEY_SCHEMA', + keytype = KEYTYPE_SCHEMA, + keyid = KEYID_SCHEMA, + keyval = KEYVAL_SCHEMA) + +# A list of TUF key objects. +ANYKEYLIST_SCHEMA = SCHEMA.ListOf(ANYKEY_SCHEMA) + +# An RSA TUF key. RSAKEY_SCHEMA = SCHEMA.Object( - object_name='rsakey', - keytype=SCHEMA.String('rsa'), - keyid=KEYID_SCHEMA, - keyval=KEYVAL_SCHEMA) + object_name = 'RSAKEY_SCHEMA', + keytype = SCHEMA.String('rsa'), + keyid = KEYID_SCHEMA, + keyval = KEYVAL_SCHEMA) -# An ed25519 key. +# An ED25519 raw public key, which must be 32 bytes. +ED25519PUBLIC_SCHEMA = SCHEMA.LengthString(32) + +# An ED25519 raw seed key, which must be 32 bytes. +ED25519SEED_SCHEMA = SCHEMA.LengthString(32) + +# An ED25519 raw signature, which must be 64 bytes. +ED25519SIGNATURE_SCHEMA = SCHEMA.LengthString(64) + +# An ed25519 TUF key. ED25519KEY_SCHEMA = SCHEMA.Object( - object_name='ed25519key', - keytype=SCHEMA.String('ed25519'), - keyid=KEYID_SCHEMA, - keyval=KEYVAL_SCHEMA) + object_name = 'ED25519KEY_SCHEMA', + keytype = SCHEMA.String('ed25519'), + keyid = KEYID_SCHEMA, + keyval = KEYVAL_SCHEMA) # Info that describes both metadata and target files. # This schema allows the storage of multiple hashes for the same file # (e.g., sha256 and sha512 may be computed for the same file and stored). FILEINFO_SCHEMA = SCHEMA.Object( - object_name='fileinfo', - length=LENGTH_SCHEMA, - hashes=HASHDICT_SCHEMA, - custom=SCHEMA.Optional(SCHEMA.Object())) + object_name = 'FILEINFO_SCHEMA', + length = LENGTH_SCHEMA, + hashes = HASHDICT_SCHEMA, + custom = SCHEMA.Optional(SCHEMA.Object())) # A dict holding the information for a particular file. The keys hold the # relative file path and the values the relevant file information. FILEDICT_SCHEMA = SCHEMA.DictOf( - key_schema=RELPATH_SCHEMA, - value_schema=FILEINFO_SCHEMA) + key_schema = RELPATH_SCHEMA, + value_schema = FILEINFO_SCHEMA) # A dict holding a target file. TARGETFILE_SCHEMA = SCHEMA.Object( - object_name='targetfile', - filepath=RELPATH_SCHEMA, - fileinfo=FILEINFO_SCHEMA) + object_name = 'TARGETFILE_SCHEMA', + filepath = RELPATH_SCHEMA, + fileinfo = FILEINFO_SCHEMA) TARGETFILES_SCHEMA = SCHEMA.ListOf(TARGETFILE_SCHEMA) # A single signature of an object. Indicates the signature, the id of the @@ -210,10 +255,13 @@ # one can imagine that maybe a key wants to sign multiple times with different # signature methods. SIGNATURE_SCHEMA = SCHEMA.Object( - object_name='signature', - keyid=KEYID_SCHEMA, - method=SIG_METHOD_SCHEMA, - sig=HEX_SCHEMA) + object_name = 'SIGNATURE_SCHEMA', + keyid = KEYID_SCHEMA, + method = SIG_METHOD_SCHEMA, + sig = HEX_SCHEMA) + +# List of SIGNATURE_SCHEMA. +SIGNATURES_SCHEMA = SCHEMA.ListOf(SIGNATURE_SCHEMA) # A schema holding the result of checking the signatures of a particular # 'SIGNABLE_SCHEMA' role. @@ -221,31 +269,31 @@ # valid? This SCHEMA holds this information. See 'sig.py' for # more information. SIGNATURESTATUS_SCHEMA = SCHEMA.Object( - object_name='signaturestatus', - threshold=SCHEMA.Integer(), - good_sigs=SCHEMA.ListOf(KEYID_SCHEMA), - bad_sigs=SCHEMA.ListOf(KEYID_SCHEMA), - unknown_sigs=SCHEMA.ListOf(KEYID_SCHEMA), - untrusted_sigs=SCHEMA.ListOf(KEYID_SCHEMA), - unknown_method_sigs=SCHEMA.ListOf(KEYID_SCHEMA)) + object_name = 'SIGNATURESTATUS_SCHEMA', + threshold = SCHEMA.Integer(), + good_sigs = KEYIDS_SCHEMA, + bad_sigs = KEYIDS_SCHEMA, + unknown_sigs = KEYIDS_SCHEMA, + untrusted_sigs = KEYIDS_SCHEMA, + unknown_method_sigs = KEYIDS_SCHEMA) # A signable object. Holds the signing role and its associated signatures. SIGNABLE_SCHEMA = SCHEMA.Object( - object_name='signable', - signed=SCHEMA.Any(), - signatures=SCHEMA.ListOf(SIGNATURE_SCHEMA)) + object_name = 'SIGNABLE_SCHEMA', + signed = SCHEMA.Any(), + signatures = SCHEMA.ListOf(SIGNATURE_SCHEMA)) # A dict where the dict keys hold a keyid and the dict values a key object. KEYDICT_SCHEMA = SCHEMA.DictOf( - key_schema=KEYID_SCHEMA, - value_schema=KEY_SCHEMA) + key_schema = KEYID_SCHEMA, + value_schema = KEY_SCHEMA) # The format used by the key database to store keys. The dict keys hold a key # identifier and the dict values any object. The key database should store # key objects in the values (e.g., 'RSAKEY_SCHEMA', 'DSAKEY_SCHEMA'). KEYDB_SCHEMA = SCHEMA.DictOf( - key_schema=KEYID_SCHEMA, - value_schema=SCHEMA.Any()) + key_schema = KEYID_SCHEMA, + value_schema = SCHEMA.Any()) # The format of the resulting "scp config dict" after extraction from the # push configuration file (i.e., push.cfg). In the case of a config file @@ -255,18 +303,18 @@ # 'remote_directory' entries. See 'tuf/pushtools/pushtoolslib.py' and # 'tuf/pushtools/push.py'. SCPCONFIG_SCHEMA = SCHEMA.Object( - object_name='scp_config', - general=SCHEMA.Object( - object_name='[general]', - transfer_module=SCHEMA.String('scp'), - metadata_path=PATH_SCHEMA, - targets_directory=PATH_SCHEMA), + object_name = 'SCPCONFIG_SCHEMA', + general = SCHEMA.Object( + object_name = '[general]', + transfer_module = SCHEMA.String('scp'), + metadata_path = PATH_SCHEMA, + targets_directory = PATH_SCHEMA), scp=SCHEMA.Object( - object_name='[scp]', - host=URL_SCHEMA, - user=NAME_SCHEMA, - identity_file=PATH_SCHEMA, - remote_directory=PATH_SCHEMA)) + object_name = '[scp]', + host = URL_SCHEMA, + user = NAME_SCHEMA, + identity_file = PATH_SCHEMA, + remote_directory = PATH_SCHEMA)) # The format of the resulting "receive config dict" after extraction from the # receive configuration file (i.e., receive.cfg). The receive config file @@ -275,101 +323,136 @@ # 'backup_directory' entries. # see 'tuf/pushtools/pushtoolslib.py' and 'tuf/pushtools/receive/receive.py' RECEIVECONFIG_SCHEMA = SCHEMA.Object( - object_name='receive_config', - general=SCHEMA.Object( - object_name='[general]', - pushroots=SCHEMA.ListOf(PATH_SCHEMA), - repository_directory=PATH_SCHEMA, - metadata_directory=PATH_SCHEMA, - targets_directory=PATH_SCHEMA, - backup_directory=PATH_SCHEMA)) + object_name = 'RECEIVECONFIG_SCHEMA', general=SCHEMA.Object( + object_name = '[general]', + pushroots = SCHEMA.ListOf(PATH_SCHEMA), + repository_directory = PATH_SCHEMA, + metadata_directory = PATH_SCHEMA, + targets_directory = PATH_SCHEMA, + backup_directory = PATH_SCHEMA)) # A path hash prefix is a hexadecimal string. PATH_HASH_PREFIX_SCHEMA = HEX_SCHEMA + # A list of path hash prefixes. PATH_HASH_PREFIXES_SCHEMA = SCHEMA.ListOf(PATH_HASH_PREFIX_SCHEMA) # Role object in {'keyids': [keydids..], 'name': 'ABC', 'threshold': 1, -# 'paths':[filepaths..]} # format. +# 'paths':[filepaths..]} format. ROLE_SCHEMA = SCHEMA.Object( - object_name='role', - keyids=SCHEMA.ListOf(KEYID_SCHEMA), - name=SCHEMA.Optional(ROLENAME_SCHEMA), - threshold=THRESHOLD_SCHEMA, - paths=SCHEMA.Optional(RELPATHS_SCHEMA), - path_hash_prefixes=SCHEMA.Optional(PATH_HASH_PREFIXES_SCHEMA)) + object_name = 'ROLE_SCHEMA', + name = SCHEMA.Optional(ROLENAME_SCHEMA), + keyids = KEYIDS_SCHEMA, + threshold = THRESHOLD_SCHEMA, + paths = SCHEMA.Optional(RELPATHS_SCHEMA), + path_hash_prefixes = SCHEMA.Optional(PATH_HASH_PREFIXES_SCHEMA)) # A dict of roles where the dict keys are role names and the dict values holding # the role data/information. ROLEDICT_SCHEMA = SCHEMA.DictOf( - key_schema=ROLENAME_SCHEMA, - value_schema=ROLE_SCHEMA) + key_schema = ROLENAME_SCHEMA, + value_schema = ROLE_SCHEMA) # Like ROLEDICT_SCHEMA, except that ROLE_SCHEMA instances are stored in order. ROLELIST_SCHEMA = SCHEMA.ListOf(ROLE_SCHEMA) -# The root: indicates root keys and top-level roles. +# The delegated roles of a Targets role (a parent). +DELEGATIONS_SCHEMA = SCHEMA.Object( + keys = KEYDICT_SCHEMA, + roles = ROLELIST_SCHEMA) + +# The number of seconds before metadata expires. The minimum is 86400 seconds +# (= 1 day). This schema is used for the initial expiration date. Repository +# maintainers may later modify this value (TIME_SCHEMA). +EXPIRATION_SCHEMA = SCHEMA.Integer(lo=86400) + +# Supported compression extension (e.g., 'gz'). +COMPRESSION_SCHEMA = SCHEMA.OneOf([SCHEMA.String(''), SCHEMA.String('gz')]) + +# List of supported compression extensions. +COMPRESSIONS_SCHEMA = SCHEMA.ListOf( + SCHEMA.OneOf([SCHEMA.String(''), SCHEMA.String('gz')])) + +# tuf.roledb +ROLEDB_SCHEMA = SCHEMA.Object( + object_name = 'ROLEDB_SCHEMA', + keyids = KEYIDS_SCHEMA, + signing_keyids = SCHEMA.Optional(KEYIDS_SCHEMA), + threshold = THRESHOLD_SCHEMA, + version = SCHEMA.Optional(METADATAVERSION_SCHEMA), + expires = SCHEMA.Optional(SCHEMA.OneOf([EXPIRATION_SCHEMA, TIME_SCHEMA])), + signatures = SCHEMA.Optional(SIGNATURES_SCHEMA), + compressions = SCHEMA.Optional(COMPRESSIONS_SCHEMA), + paths = SCHEMA.Optional(RELPATHS_SCHEMA), + path_hash_prefixes = SCHEMA.Optional(PATH_HASH_PREFIXES_SCHEMA), + delegations = SCHEMA.Optional(DELEGATIONS_SCHEMA), + partial_loaded = SCHEMA.Optional(BOOLEAN_SCHEMA)) + +# Root role: indicates root keys and top-level roles. ROOT_SCHEMA = SCHEMA.Object( - object_name='root', - _type=SCHEMA.String('Root'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - keys=KEYDICT_SCHEMA, - roles=ROLEDICT_SCHEMA) + object_name = 'ROOT_SCHEMA', + _type = SCHEMA.String('Root'), + version = METADATAVERSION_SCHEMA, + consistent_snapshots = BOOLEAN_SCHEMA, + expires = TIME_SCHEMA, + keys = KEYDICT_SCHEMA, + roles = ROLEDICT_SCHEMA) -# Targets. Indicates targets and delegates target paths to other roles. +# Targets role: Indicates targets and delegates target paths to other roles. TARGETS_SCHEMA = SCHEMA.Object( - object_name='targets', - _type=SCHEMA.String('Targets'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - targets=FILEDICT_SCHEMA, - delegations=SCHEMA.Optional(SCHEMA.Object( - keys=KEYDICT_SCHEMA, - roles=ROLELIST_SCHEMA))) + object_name = 'TARGETS_SCHEMA', + _type = SCHEMA.String('Targets'), + version = METADATAVERSION_SCHEMA, + expires = TIME_SCHEMA, + targets = FILEDICT_SCHEMA, + delegations = SCHEMA.Optional(DELEGATIONS_SCHEMA)) -# A Release: indicates the latest versions of all metadata (except timestamp). +# Release role: indicates the latest versions of all metadata (except timestamp). RELEASE_SCHEMA = SCHEMA.Object( - object_name='release', - _type=SCHEMA.String('Release'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - meta=FILEDICT_SCHEMA) + object_name = 'RELEASE_SCHEMA', + _type = SCHEMA.String('Release'), + version = METADATAVERSION_SCHEMA, + expires = TIME_SCHEMA, + meta = FILEDICT_SCHEMA) -# A Timestamp: indicates the latest version of the release file. +# Timestamp role: indicates the latest version of the release file. TIMESTAMP_SCHEMA = SCHEMA.Object( - object_name='timestamp', - _type=SCHEMA.String('Timestamp'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - meta=FILEDICT_SCHEMA) + object_name = 'TIMESTAMP_SCHEMA', + _type = SCHEMA.String('Timestamp'), + version = METADATAVERSION_SCHEMA, + expires = TIME_SCHEMA, + meta = FILEDICT_SCHEMA) # A schema containing information a repository mirror may require, # such as a url, the path of the directory metadata files, etc. MIRROR_SCHEMA = SCHEMA.Object( - object_name='mirror', - url_prefix=URL_SCHEMA, - metadata_path=RELPATH_SCHEMA, - targets_path=RELPATH_SCHEMA, - confined_target_dirs=RELPATHS_SCHEMA, - custom=SCHEMA.Optional(SCHEMA.Object())) + object_name = 'MIRROR_SCHEMA', + url_prefix = URL_SCHEMA, + metadata_path = RELPATH_SCHEMA, + targets_path = RELPATH_SCHEMA, + confined_target_dirs = RELPATHS_SCHEMA, + custom = SCHEMA.Optional(SCHEMA.Object())) # A dictionary of mirrors where the dict keys hold the mirror's name and # and the dict values the mirror's data (i.e., 'MIRROR_SCHEMA'). # The repository class of 'updater.py' accepts dictionaries # of this type provided by the TUF client. MIRRORDICT_SCHEMA = SCHEMA.DictOf( - key_schema=SCHEMA.AnyString(), - value_schema=MIRROR_SCHEMA) + key_schema = SCHEMA.AnyString(), + value_schema = MIRROR_SCHEMA) # A Mirrorlist: indicates all the live mirrors, and what documents they # serve. MIRRORLIST_SCHEMA = SCHEMA.Object( - object_name='mirrorlist', - _type=SCHEMA.String('Mirrors'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - mirrors=SCHEMA.ListOf(MIRROR_SCHEMA)) + object_name = 'MIRRORLIST_SCHEMA', + _type = SCHEMA.String('Mirrors'), + version = METADATAVERSION_SCHEMA, + expires = TIME_SCHEMA, + mirrors = SCHEMA.ListOf(MIRROR_SCHEMA)) + +# Any of the role schemas (e.g., TIMESTAMP_SCHEMA, RELEASE_SCHEMA, etc.) +ANYROLE_SCHEMA = SCHEMA.OneOf([ROOT_SCHEMA, TARGETS_SCHEMA, RELEASE_SCHEMA, + TIMESTAMP_SCHEMA, MIRROR_SCHEMA]) @@ -383,7 +466,6 @@ class MetaFile(object): and ReleaseFile all inherit from MetaFile. The __eq__, __ne__, perform 'equal' and 'not equal' comparisons between Metadata File objects. - """ info = None @@ -401,7 +483,6 @@ def __getattr__(self, name): Allow all metafile objects to have their interesting attributes referred to directly without the info dict. The info dict is just to be able to do the __eq__ comparison generically. - """ if name in self.info: @@ -450,12 +531,13 @@ def make_metadata(version, expiration_date, filedict): class RootFile(MetaFile): - def __init__(self, version, expires, keys, roles): + def __init__(self, version, expires, keys, roles, consistent_snapshots): self.info = {} self.info['version'] = version self.info['expires'] = expires self.info['keys'] = keys self.info['roles'] = roles + self.info['consistent_snapshots'] = consistent_snapshots @staticmethod @@ -468,21 +550,20 @@ def from_metadata(object): expires = parse_time(object['expires']) keys = object['keys'] roles = object['roles'] + consistent_snapshots = object['consistent_snapshots'] - return RootFile(version, expires, keys, roles) + return RootFile(version, expires, keys, roles, consistent_snapshots) @staticmethod - def make_metadata(version, expiration_seconds, keydict, roledict): - # Is 'expiration_seconds' properly formatted? - # Raise 'tuf.FormatError' if not. - LENGTH_SCHEMA.check_match(expiration_seconds) - + def make_metadata(version, expiration_date, keydict, roledict, + consistent_snapshots): result = {'_type' : 'Root'} result['version'] = version - result['expires'] = format_time(time.time() + expiration_seconds) + result['expires'] = expiration_date result['keys'] = keydict result['roles'] = roledict + result['consistent_snapshots'] = consistent_snapshots # Is 'result' a Root metadata file? # Raise 'tuf.FormatError' if not. @@ -645,7 +726,6 @@ def format_time(timestamp): A string in 'YYYY-MM-DD HH:MM:SS UTC' format. - """ try: @@ -677,7 +757,6 @@ def parse_time(string): A timestamp (e.g., 499137660). - """ # Is 'string' properly formatted? @@ -715,7 +794,6 @@ def format_base64(data): A base64-encoded string. - """ try: @@ -746,7 +824,6 @@ def parse_base64(base64_string): A byte string representing the parsed based64 encoding of 'base64_string'. - """ if not isinstance(base64_string, basestring): @@ -791,7 +868,6 @@ def make_signable(object): A dict in 'SIGNABLE_SCHEMA' format. - """ if not isinstance(object, dict) or 'signed' not in object: @@ -832,7 +908,6 @@ def make_fileinfo(length, hashes, custom=None): A dictionary conformant to 'FILEINFO_SCHEMA', representing the file information of a metadata or target file. - """ fileinfo = {'length' : length, 'hashes' : hashes} @@ -889,7 +964,6 @@ def make_role_metadata(keyids, threshold, name=None, paths=None, A properly formatted role meta dict, conforming to 'ROLE_SCHEMA'. - """ role_meta = {} @@ -950,7 +1024,6 @@ def get_role_class(expected_rolename): The class corresponding to 'expected_rolename'. E.g., 'Release' as an argument to this function causes 'ReleaseFile' to be returned. - """ # Does 'expected_rolename' have the correct type? @@ -993,7 +1066,6 @@ def expected_meta_rolename(meta_rolename): A string (e.g., 'Root', 'Targets'). - """ # Does 'meta_rolename' have the correct type? @@ -1033,7 +1105,6 @@ def check_signable_object_format(object): A string representing the signing role (e.g., 'root', 'targets'). The role string is returned with characters all lower case. - """ # Does 'object' have the correct type? @@ -1077,7 +1148,6 @@ def _canonical_string_encoder(string): A string with the canonical-encoded 'string' embedded. - """ string = '"%s"' % re.sub(r'(["\\])', r'\\\1', string) @@ -1182,7 +1252,6 @@ def encode_canonical(object, output_function=None): A string representing the 'object' encoded in canonical JSON form. - """ result = None diff --git a/tuf/interposition/__init__.py b/tuf/interposition/__init__.py index 00c53e7c..4360016e 100644 --- a/tuf/interposition/__init__.py +++ b/tuf/interposition/__init__.py @@ -285,6 +285,24 @@ def configure(filename="tuf.interposition.json", +def refresh(configurations): + """Refresh the top-level metadata for previously read configurations.""" + + # Get the updater and refresh its top-level metadata. In the majority of + # integrations, a software updater integrating TUF with interposition will + # usually only require an initial refresh() (i.e., when configure() is + # called). A series of target file requests may then occur, which are all + # referenced by the latest top-level metadata updated by configure(). + # Although interposition was designed to remain transparent, for software + # updaters that require an explicit refresh of top-level metadata, this + # method is provided. + for configuration in configurations.itervalues(): + __updater_controller.refresh(configuration) + + + + + def deconfigure(configurations): """Remove TUF interposition for previously read configurations.""" diff --git a/tuf/interposition/configuration.py b/tuf/interposition/configuration.py index 295ccac6..e165a073 100644 --- a/tuf/interposition/configuration.py +++ b/tuf/interposition/configuration.py @@ -45,7 +45,7 @@ def __init__(self, hostname, port, repository_directory, repository_mirrors, def __repr__(self): - MESSAGE = "Configuration(netloc={network_location})" + MESSAGE = "network location: {network_location}" return MESSAGE.format(network_location=self.network_location) diff --git a/tuf/interposition/updater.py b/tuf/interposition/updater.py index 8c065479..23447194 100644 --- a/tuf/interposition/updater.py +++ b/tuf/interposition/updater.py @@ -50,6 +50,18 @@ def __init__(self, configuration): self.switch_context() self.updater = tuf.client.updater.Updater(self.configuration.hostname, self.configuration.repository_mirrors) + + # Update the client's top-level metadata. The download_target() method does + # not automatically refresh top-level prior to retrieving target files and + # their associated Targets metadata, so update the top-level + # metadata here. + Logger.info('Refreshing top-level metadata for interposed '+repr(configuration)) + self.updater.refresh() + + + def refresh(self): + """Refresh top-level metadata""" + self.updater.refresh() def cleanup(self): @@ -66,11 +78,14 @@ def download_target(self, target_filepath): # download file into a temporary directory shared over runtime destination_directory = self.tempdir filename = os.path.join(destination_directory, target_filepath) - - self.switch_context() # switch TUF context - self.updater.refresh() # update TUF client repository metadata - - # then, update target at filepath + + # Switch TUF context. + self.switch_context() + + # Locate the fileinfo of 'target_filepath'. updater.target() searches + # Targets metadata in order of trust, according to the currently trusted + # release. To prevent consecutive target file requests from referring to + # different releases, top-level metadata is not automatically refreshed. targets = [self.updater.target(target_filepath)] # TODO: targets are always updated if destination directory is new, right? @@ -254,15 +269,37 @@ def __check_configuration_on_add(self, configuration): def add(self, configuration): """Add an Updater based on the given Configuration.""" - UPDATER_ADDED_MESSAGE = "Updater added for {configuration}." - repository_mirror_hostnames = self.__check_configuration_on_add(configuration) - + # If all is well, build and store an Updater, and remember hostnames. + Logger.info('Adding updater for interposed '+repr(configuration)) self.__updaters[configuration.hostname] = Updater(configuration) self.__repository_mirror_hostnames.update(repository_mirror_hostnames) + + + + def refresh(self, configuration): + """Refresh the top-level metadata of the given Configuration.""" + + assert isinstance(configuration, Configuration) + + repository_mirror_hostnames = configuration.get_repository_mirror_hostnames() + + assert configuration.hostname in self.__updaters + assert repository_mirror_hostnames.issubset(self.__repository_mirror_hostnames) + + # Get the updater and refresh its top-level metadata. In the majority of + # integrations, a software updater integrating TUF with interposition will + # usually only require an initial refresh() (i.e., when configure() is + # called). A series of target file requests may then occur, which are all + # referenced by the latest top-level metadata updated by configure(). + # Although interposition was designed to remain transparent, for software + # updaters that require an explicit refresh of top-level metadata, this + # method is provided. + Logger.info('Refreshing top-level metadata for '+ repr(configuration)) + updater = self.__updaters.get(configuration.hostname) + updater.refresh() - Logger.info(UPDATER_ADDED_MESSAGE.format(configuration=configuration)) def get(self, url): @@ -273,7 +310,7 @@ def get(self, url): GENERIC_WARNING_MESSAGE = "No updater or interposition for url={url}" DIFFERENT_NETLOC_MESSAGE = "We have an updater for netloc={netloc1} but not for netlocs={netloc2}" - HOSTNAME_FOUND_MESSAGE = "Found updater for hostname={hostname}" + HOSTNAME_FOUND_MESSAGE = "Found updater for interposed network location: {netloc}" HOSTNAME_NOT_FOUND_MESSAGE = "No updater for hostname={hostname}" updater = None @@ -298,7 +335,7 @@ def get(self, url): # Ensure that the updater is meant for this (hostname, port). if updater.configuration.network_location in network_locations: - Logger.info(HOSTNAME_FOUND_MESSAGE.format(hostname=hostname)) + Logger.info(HOSTNAME_FOUND_MESSAGE.format(netloc=network_location)) # Raises an exception in case we do not recognize how to # transform this URL for TUF. In that case, there will be no # updater for this URL. @@ -324,7 +361,7 @@ def get(self, url): def remove(self, configuration): """Remove an Updater matching the given Configuration.""" - UPDATER_REMOVED_MESSAGE = "Updater removed for {configuration}." + UPDATER_REMOVED_MESSAGE = "Updater removed for interposed {configuration}." assert isinstance(configuration, Configuration) diff --git a/tuf/keydb.py b/tuf/keydb.py index 601993b3..60cb4d89 100755 --- a/tuf/keydb.py +++ b/tuf/keydb.py @@ -25,15 +25,17 @@ and the '_get_keyid()' function to learn precisely how keyids are generated. One may get the keyid of a key object by simply accessing the dictionary's 'keyid' key (i.e., rsakey['keyid']). - """ - import logging +import copy import tuf import tuf.formats -import tuf.rsa_key +import tuf.keys + +# List of strings representing the key types supported by TUF. +_SUPPORTED_KEY_TYPES = ['rsa', 'ed25519'] # See 'log.py' to learn how logging is handled in TUF. logger = logging.getLogger('tuf.keydb') @@ -62,13 +64,12 @@ def create_keydb_from_root_metadata(root_metadata): A function to add the key to the database is called. In the case of RSA - keys, this function is add_rsakey(). + keys, this function is add_key(). The old keydb key database is replaced. None. - """ # Does 'root_metadata' have the correct format? @@ -84,13 +85,13 @@ def create_keydb_from_root_metadata(root_metadata): # them to 'RSAKEY_SCHEMA' if their type is 'rsa', and then # adding them the database. Duplicates are avoided. for keyid, key_metadata in root_metadata['keys'].items(): - if key_metadata['keytype'] == 'rsa': + if key_metadata['keytype'] in _SUPPORTED_KEY_TYPES: # 'key_metadata' is stored in 'KEY_SCHEMA' format. Call # create_from_metadata_format() to get the key in 'RSAKEY_SCHEMA' - # format, which is the format expected by 'add_rsakey()'. - rsakey_dict = tuf.rsa_key.create_from_metadata_format(key_metadata) + # format, which is the format expected by 'add_key()'. + key_dict = tuf.keys.format_metadata_to_key(key_metadata) try: - add_rsakey(rsakey_dict, keyid) + add_key(key_dict, keyid) # 'tuf.Error' raised if keyid does not match the keyid for 'rsakey_dict'. except tuf.Error, e: logger.error(e) @@ -105,7 +106,7 @@ def create_keydb_from_root_metadata(root_metadata): -def add_rsakey(rsakey_dict, keyid=None): +def add_key(key_dict, keyid=None): """ Add 'rsakey_dict' to the key database while avoiding duplicates. @@ -113,8 +114,8 @@ def add_rsakey(rsakey_dict, keyid=None): and raise an exception if it is not. - rsakey_dict: - A dictionary conformant to 'tuf.formats.RSAKEY_SCHEMA'. + key_dict: + A dictionary conformant to 'tuf.formats.ANYKEY_SCHEMA'. It has the form: {'keytype': 'rsa', 'keyid': keyid, @@ -138,15 +139,13 @@ def add_rsakey(rsakey_dict, keyid=None): None. - """ - # Does 'rsakey_dict' have the correct format? # This check will ensure 'rsakey_dict' has the appropriate number of objects # and object types, and that all dict keys are properly named. # Raise 'tuf.FormatError if the check fails. - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) + tuf.formats.ANYKEY_SCHEMA.check_match(key_dict) # Does 'keyid' have the correct format? if keyid is not None: @@ -154,16 +153,16 @@ def add_rsakey(rsakey_dict, keyid=None): tuf.formats.KEYID_SCHEMA.check_match(keyid) # Check if the keyid found in 'rsakey_dict' matches 'keyid'. - if keyid != rsakey_dict['keyid']: - raise tuf.Error('Incorrect keyid '+rsakey_dict['keyid']+' expected '+keyid) + if keyid != key_dict['keyid']: + raise tuf.Error('Incorrect keyid '+key_dict['keyid']+' expected '+keyid) # Check if the keyid belonging to 'rsakey_dict' is not already # available in the key database before returning. - keyid = rsakey_dict['keyid'] + keyid = key_dict['keyid'] if keyid in _keydb_dict: raise tuf.KeyAlreadyExistsError('Key: '+keyid) - _keydb_dict[keyid] = rsakey_dict + _keydb_dict[keyid] = copy.deepcopy(key_dict) @@ -190,7 +189,6 @@ def get_key(keyid): The key matching 'keyid'. In the case of RSA keys, a dictionary conformant to 'tuf.formats.RSAKEY_SCHEMA' is returned. - """ # Does 'keyid' have the correct format? @@ -201,7 +199,7 @@ def get_key(keyid): # Return the key belonging to 'keyid', if found in the key database. try: - return _keydb_dict[keyid] + return copy.deepcopy(_keydb_dict[keyid]) except KeyError: raise tuf.UnknownKeyError('Key: '+keyid) @@ -229,7 +227,6 @@ def remove_key(keyid): None. - """ # Does 'keyid' have the correct format? @@ -264,7 +261,6 @@ def clear_keydb(): None. - """ _keydb_dict.clear() diff --git a/tuf/keys.py b/tuf/keys.py new file mode 100755 index 00000000..fdbbc5ba --- /dev/null +++ b/tuf/keys.py @@ -0,0 +1,1244 @@ +""" + + keys.py + + + Vladimir Diaz + + + October 4, 2013. + + + See LICENSE for licensing information. + + + The goal of this module is to centralize cryptographic key routines and their + supported operations (e.g., creating and verifying signatures). This module + is designed to support multiple public-key algorithms, such as RSA and + ED25519, and multiple cryptography libraries. Which cryptography library to + use is determined by the default, or user modified, values set in + 'tuf.conf.py' + + The (RSA and ED25519)-related functions provided include generate_rsa_key(), + generate_ed25519_key(), create_signature(), and verify_signature(). + The cryptography libraries called by 'tuf.keys.py' generate the actual TUF + keys and the functions listed above can be viewed as the easy-to-use public + interface. + + Additional functions contained here include format_keyval_to_metadata() and + format_metadata_to_key(). These last two functions produce or use TUF keys + compatible with the key structures listed in TUF Metadata files. The key + generation functions return a dictionary containing all the information needed + of TUF keys, such as public & private keys, and a keyID. create_signature() + and verify_signature() are supplemental functions needed for generating + signatures and verifying them. + + https://en.wikipedia.org/wiki/RSA_(algorithm) + http://ed25519.cr.yp.to/ + + Key IDs are used as identifiers for keys (e.g., RSA key). They are the + hexadecimal representation of the hash of key object (specifically, the key + object containing only the public key). Review 'keys.py' and the + '_get_keyid()' function to see precisely how keyids are generated. One may + get the keyid of a key object by simply accessing the dictionary's 'keyid' + key (i.e., rsakey['keyid']). + """ + +# Required for hexadecimal conversions. Signatures and public/private keys are +# hexlified. +import binascii + +# 'pycrypto' is the only currently supported library for the creation of RSA +# keys. +# https://github.com/dlitz/pycrypto +_SUPPORTED_RSA_CRYPTO_LIBRARIES = ['pycrypto'] + +# The currently supported libraries for the creation of ed25519 keys and +# signatures. The 'pynacl' library should be installed and used over the slower +# python implementation of ed25519. The python implementation will be used +# if 'pynacl' is unavailable. +_SUPPORTED_ED25519_CRYPTO_LIBRARIES = ['ed25519', 'pynacl'] + +# 'pycrypto' is the only currently supported library for general-purpose +# cryptography, with plans to support pyca/cryptography. +# https://github.com/dlitz/pycrypto +# https://github.com/pyca/cryptography +_SUPPORTED_GENERAL_CRYPTO_LIBRARIES = ['pycrypto'] + +# Track which libraries are imported and thus available. An optimized version +# of the ed25519 python implementation is provided by TUF and avaialable by +# default. https://github.com/pyca/ed25519 +_available_crypto_libraries = ['ed25519'] + +# Import the PyCrypto library so that RSA keys are supported. +try: + import Crypto + import tuf.pycrypto_keys + _available_crypto_libraries.append('pycrypto') +except ImportError: + pass + +# Import the PyNaCl library, if available. It is recommended this library be +# used over the pure python implementation of ed25519, due to its speedier +# routines and side-channel protections available in the libsodium library. +try: + import nacl + import nacl.signing + _available_crypto_libraries.append('pynacl') +except (ImportError, IOError): + pass + +# The optimized version of the ed25519 library provided by default is imported +# regardless of the availability of PyNaCl. +import tuf.ed25519_keys + +# Import the TUF package and TUF-defined exceptions in __init__.py. +import tuf + +# Import the cryptography library settings. +import tuf.conf + +# Digest objects needed to generate hashes. +import tuf.hash + +# Perform format checks of argument objects. +import tuf.formats + +# The hash algorithm to use in the generation of keyids. +_KEY_ID_HASH_ALGORITHM = 'sha256' + +# Recommended RSA key sizes: +# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 +# According to the document above, revised May 6, 2003, RSA keys of +# size 3072 provide security through 2031 and beyond. +_DEFAULT_RSA_KEY_BITS = 3072 + +# The crypto libraries to use in 'keys.py', set by default or by the user. +# The following cryptography libraries are currently supported: +# ['pycrypto', 'pynacl', 'ed25519'] +_RSA_CRYPTO_LIBRARY = tuf.conf.RSA_CRYPTO_LIBRARY +_ED25519_CRYPTO_LIBRARY = tuf.conf.ED25519_CRYPTO_LIBRARY +_GENERAL_CRYPTO_LIBRARY = tuf.conf.GENERAL_CRYPTO_LIBRARY + + +def generate_rsa_key(bits=_DEFAULT_RSA_KEY_BITS): + """ + + Generate public and private RSA keys, with modulus length 'bits'. In + addition, a keyid identifier for the RSA key is generated. The object + returned conforms to 'tuf.formats.RSAKEY_SCHEMA' and has the + form: + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are strings in PEM format. + + Although the PyCrypto crytography library called sets a 1024-bit minimum + key size, generate() enforces a minimum key size of 2048 bits. If 'bits' is + unspecified, a 3072-bit RSA key is generated, which is the key size + recommended by TUF. + + >>> rsa_key = generate_rsa_key(bits=2048) + >>> tuf.formats.RSAKEY_SCHEMA.matches(rsa_key) + True + >>> public = rsa_key['keyval']['public'] + >>> private = rsa_key['keyval']['private'] + >>> tuf.formats.PEMRSA_SCHEMA.matches(public) + True + >>> tuf.formats.PEMRSA_SCHEMA.matches(private) + True + + + bits: + The key size, or key length, of the RSA key. 'bits' must be 2048, or + greater, and a multiple of 256. + + + tuf.FormatError, if 'bits' is improperly or invalid (i.e., not an integer + and not at least 2048). + + tuf.UnsupportedLibraryError, if any of the cryptography libraries specified + in 'tuf.conf.py' are unsupported or unavailable. + + ValueError, if an exception occurs after calling the RSA key generation + routine. 'bits' must be a multiple of 256. The 'ValueError' exception is + raised by the key generation function of the cryptography library called. + + + The RSA keys are generated by calling PyCrypto's + Crypto.PublicKey.RSA.generate(). + + + A dictionary containing the RSA keys and other identifying information. + Conforms to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'bits' have the correct format? + # This check will ensure 'bits' conforms to 'tuf.formats.RSAKEYBITS_SCHEMA'. + # 'bits' must be an integer object, with a minimum value of 2048. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY' and 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Begin building the RSA key dictionary. + rsakey_dict = {} + keytype = 'rsa' + public = None + private = None + + # Generate the public and private RSA keys. The PyCrypto module performs + # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 + # or not a multiple of 256, although a 2048-bit minimum is enforced by + # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + public, private = tuf.pycrypto_keys.generate_rsa_public_and_private(bits) + else: + message = 'Invalid crypto library: '+repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + # Generate the keyid of the RSA key. 'key_value' corresponds to the + # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key + # information is not included in the generation of the 'keyid' identifier. + key_value = {'public': public, + 'private': ''} + keyid = _get_keyid(keytype, key_value) + + # Build the 'rsakey_dict' dictionary. Update 'key_value' with the RSA + # private key prior to adding 'key_value' to 'rsakey_dict'. + key_value['private'] = private + + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def generate_ed25519_key(): + """ + + Generate public and private ED25519 keys, both of length 32-bytes, although + they are hexlified to 64 bytes. + In addition, a keyid identifier generated for the returned ED25519 object. + The object returned conforms to 'tuf.formats.ED25519KEY_SCHEMA' and has the + form: + {'keytype': 'ed25519', + 'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'keyval': {'public': '9ccf3f02b17f82febf5dd3bab878b767d8408...', + 'private': 'ab310eae0e229a0eceee3947b6e0205dfab3...'}} + + The public and private keys are strings in PEM format and stored in the + 'keyval' field of the returned dictionary. + + >>> ed25519_key = generate_ed25519_key() + >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key) + True + >>> len(ed25519_key['keyval']['public']) + 64 + >>> len(ed25519_key['keyval']['private']) + 64 + + + None. + + + tuf.UnsupportedLibraryError, if an unsupported or unavailable library is + detected. + + + The ED25519 keys are generated by calling either the optimized pure Python + implementation of ed25519, or the ed25519 routines provided by 'pynacl'. + + + A dictionary containing the ED25519 keys and other identifying information. + Conforms to 'tuf.formats.ED25519KEY_SCHEMA'. + """ + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified + # in 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY' and 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Begin building the ED25519 key dictionary. + ed25519_key = {} + keytype = 'ed25519' + public = None + private = None + + # Generate the public and private ED25519 keys. Use the 'pynacl' library + # if available, otherwise fall back to optimized pure python implementation + # provided by pyca and available in TUF. + if 'pynacl' in _available_crypto_libraries: + public, private = \ + tuf.ed25519_keys.generate_public_and_private(use_pynacl=True) + else: + public, private = \ + tuf.ed25519_keys.generate_public_and_private(use_pynacl=False) + + # Generate the keyid of the ED25519 key. 'key_value' corresponds to the + # 'keyval' entry of the 'ED25519KEY_SCHEMA' dictionary. The private key + # information is not included in the generation of the 'keyid' identifier. + key_value = {'public': binascii.hexlify(public), + 'private': ''} + keyid = _get_keyid(keytype, key_value) + + # Build the 'ed25519_key' dictionary. Update 'key_value' with the ED25519 + # private key prior to adding 'key_value' to 'ed25519_key'. + key_value['private'] = binascii.hexlify(private) + + ed25519_key['keytype'] = keytype + ed25519_key['keyid'] = keyid + ed25519_key['keyval'] = key_value + + return ed25519_key + + + + + +def format_keyval_to_metadata(keytype, key_value, private=False): + """ + + Return a dictionary conformant to 'tuf.formats.KEY_SCHEMA'. + If 'private' is True, include the private key. The dictionary + returned has the form: + {'keytype': keytype, + 'keyval': {'public': '...', + 'private': '...'}} + + or if 'private' is False: + + {'keytype': keytype, + 'keyval': {'public': '...', + 'private': ''}} + + TUF keys are stored in Metadata files (e.g., root.txt) in the format + returned by this function. + + >>> ed25519_key = generate_ed25519_key() + >>> key_val = ed25519_key['keyval'] + >>> keytype = ed25519_key['keytype'] + >>> ed25519_metadata = \ + format_keyval_to_metadata(keytype, key_val, private=True) + >>> tuf.formats.KEY_SCHEMA.matches(ed25519_metadata) + True + + + key_type: + The 'rsa' or 'ed25519' strings. + + key_value: + A dictionary containing a private and public keys. + 'key_value' is of the form: + + {'public': '...', + 'private': '...'}}, + + conformant to 'tuf.formats.KEYVAL_SCHEMA'. + + private: + Indicates if the private key should be included in the dictionary + returned. + + + tuf.FormatError, if 'key_value' does not conform to + 'tuf.formats.KEYVAL_SCHEMA'. + + + None. + + + A 'tuf.formats.KEY_SCHEMA' dictionary. + """ + + # Does 'keytype' have the correct format? + # This check will ensure 'keytype' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.KEYTYPE_SCHEMA.check_match(keytype) + + # Does 'key_value' have the correct format? + tuf.formats.KEYVAL_SCHEMA.check_match(key_value) + + if private is True and key_value['private']: + return {'keytype': keytype, 'keyval': key_value} + else: + public_key_value = {'public': key_value['public'], 'private': ''} + return {'keytype': keytype, 'keyval': public_key_value} + + + + + +def format_metadata_to_key(key_metadata): + """ + + Construct a TUF key dictionary (e.g., tuf.formats.RSAKEY_SCHEMA) + according to the keytype of 'key_metadata'. The dict returned by this + function has the exact format as the dict returned by one of the key + generations functions, like generate_ed25519_key(). The dict returned + has the form: + + {'keytype': keytype, + 'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'keyval': {'public': '...', + 'private': '...'}} + + For example, RSA key dictionaries in RSAKEY_SCHEMA format should be used by + modules storing a collection of keys, such as with keydb.py. RSA keys as + stored in metadata files use a different format, so this function should be + called if an RSA key is extracted from one of these metadata files and need + converting. The key generation functions create an entirely new key and + return it in the format appropriate for 'keydb.py'. + + >>> ed25519_key = generate_ed25519_key() + >>> key_val = ed25519_key['keyval'] + >>> keytype = ed25519_key['keytype'] + >>> ed25519_metadata = \ + format_keyval_to_metadata(keytype, key_val, private=True) + >>> ed25519_key_2 = format_metadata_to_key(ed25519_metadata) + >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key_2) + True + >>> ed25519_key == ed25519_key_2 + True + + + key_metadata: + The TUF key dictionary as stored in Metadata files, conforming to + 'tuf.formats.KEY_SCHEMA'. It has the form: + + {'keytype': '...', + 'keyval': {'public': '...', + 'private': '...'}} + + + tuf.FormatError, if 'key_metadata' does not conform to + 'tuf.formats.KEY_SCHEMA'. + + + None. + + + In the case of an RSA key, a dictionary conformant to + 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'key_metadata' have the correct format? + # This check will ensure 'key_metadata' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.KEY_SCHEMA.check_match(key_metadata) + + # Construct the dictionary to be returned. + key_dict = {} + keytype = key_metadata['keytype'] + key_value = key_metadata['keyval'] + + # Convert 'key_value' to 'tuf.formats.KEY_SCHEMA' and generate its hash + # The hash is in hexdigest form. + keyid = _get_keyid(keytype, key_value) + + # All the required key values gathered. Build 'key_dict'. + key_dict['keytype'] = keytype + key_dict['keyid'] = keyid + key_dict['keyval'] = key_value + + return key_dict + + + + + +def _get_keyid(keytype, key_value): + """Return the keyid for 'key_value'.""" + + # 'keyid' will be generated from an object conformant to KEY_SCHEMA, + # which is the format Metadata files (e.g., root.txt) store keys. + # 'format_keyval_to_metadata()' returns the object needed by _get_keyid(). + key_meta = format_keyval_to_metadata(keytype, key_value, private=False) + + # Convert the TUF key to JSON Canonical format, suitable for adding + # to digest objects. + key_update_data = tuf.formats.encode_canonical(key_meta) + + # Create a digest object and call update(), using the JSON + # canonical format of 'rskey_meta' as the update data. + digest_object = tuf.hash.digest(_KEY_ID_HASH_ALGORITHM) + digest_object.update(key_update_data) + + # 'keyid' becomes the hexadecimal representation of the hash. + keyid = digest_object.hexdigest() + + return keyid + + + + + +def _check_crypto_libraries(): + """ Ensure all the crypto libraries specified in tuf.conf are available. """ + + # The checks below all raise 'tuf.UnsupportedLibraryError' if the RSA and + # ED25519 crypto libraries specified in 'tuf.conf.py' are not supported or + # unavailable. The appropriate error message is added to the exception. + # The funcions of this module that depend on user-installed crypto libraries + # should call this private function to ensure the called routine does not fail + # with unpredictable exceptions in the event of a missing library. + # The supported and available lists checked are populated when 'tuf.keys.py' + # is imported. + if _RSA_CRYPTO_LIBRARY not in _SUPPORTED_RSA_CRYPTO_LIBRARIES: + message = 'The '+repr(_RSA_CRYPTO_LIBRARY)+' crypto library specified'+ \ + ' in "tuf.conf.RSA_CRYPTO_LIBRARY" is not supported.\n'+ \ + 'Supported crypto libraries: '+repr(_SUPPORTED_RSA_CRYPTO_LIBRARIES)+'.' + raise tuf.UnsupportedLibraryError(message) + + if _ED25519_CRYPTO_LIBRARY not in _SUPPORTED_ED25519_CRYPTO_LIBRARIES: + message = 'The '+repr(_ED25519_CRYPTO_LIBRARY)+' crypto library specified'+\ + ' in "tuf.conf.ED25519_CRYPTO_LIBRARY" is not supported.\n'+ \ + 'Supported crypto libraries: '+repr(_SUPPORTED_ED25519_CRYPTO_LIBRARIES)+'.' + raise tuf.UnsupportedLibraryError(message) + + if _GENERAL_CRYPTO_LIBRARY not in _SUPPORTED_GENERAL_CRYPTO_LIBRARIES: + message = 'The '+repr(_GENERAL_CRYPTO_LIBRARY)+' crypto library specified'+\ + ' in "tuf.conf.GENERAL_CRYPTO_LIBRARY" is not supported.\n'+ \ + 'Supported crypto libraries: '+repr(_SUPPORTED_GENERAL_CRYPTO_LIBRARIES)+'.' + raise tuf.UnsupportedLibraryError(message) + + if _RSA_CRYPTO_LIBRARY not in _available_crypto_libraries: + message = 'The '+repr(_RSA_CRYPTO_LIBRARY)+' crypto library specified'+ \ + ' in "tuf.conf.RSA_CRYPTO_LIBRARY" could not be imported.' + raise tuf.UnsupportedLibraryError(message) + + if _ED25519_CRYPTO_LIBRARY not in _available_crypto_libraries: + message = 'The '+repr(_ED25519_CRYPTO_LIBRARY)+' crypto library specified'+\ + ' in "tuf.conf.ED25519_CRYPTO_LIBRARY" could not be imported.' + raise tuf.UnsupportedLibraryError(message) + + if _GENERAL_CRYPTO_LIBRARY not in _available_crypto_libraries: + message = 'The '+repr(_GENERAL_CRYPTO_LIBRARY)+' crypto library specified'+\ + ' in "tuf.conf.GENERAL_CRYPTO_LIBRARY" could not be imported.' + raise tuf.UnsupportedLibraryError(message) + + + + + +def create_signature(key_dict, data): + """ + + Return a signature dictionary of the form: + {'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'method': '...', + 'sig': '...'}. + + The signing process will use the private key in + key_dict['keyval']['private'] and 'data' to generate the signature. + + The following signature methods are supported: + + 'RSASSA-PSS' + RFC3447 - RSASSA-PSS + http://www.ietf.org/rfc/rfc3447. + + 'ed25519' + ed25519 - high-speed high security signatures + http://ed25519.cr.yp.to/ + + Which signature to generate is determined by the key type of 'key_dict' + and the available cryptography library specified in 'tuf.conf'. + + >>> ed25519_key = generate_ed25519_key() + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature = create_signature(ed25519_key, data) + >>> tuf.formats.SIGNATURE_SCHEMA.matches(signature) + True + >>> len(signature['sig']) + 128 + >>> rsa_key = generate_rsa_key(2048) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature = create_signature(rsa_key, data) + >>> tuf.formats.SIGNATURE_SCHEMA.matches(signature) + True + + + key_dict: + A dictionary containing the TUF keys. An example RSA key dict has the + form: + + {'keytype': 'rsa', + 'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are strings in PEM format. + + data: + Data object used by create_signature() to generate the signature. + + + tuf.FormatError, if 'key_dict' is improperly formatted. + + tuf.UnsupportedLibraryError, if an unsupported or unavailable library is + detected. + + TypeError, if 'key_dict' contains an invalid keytype. + + + The cryptography library specified in 'tuf.conf' called to perform the + actual signing routine. + + + A signature dictionary conformat to 'tuf.format.SIGNATURE_SCHEMA'. + """ + + # Does 'key_dict' have the correct format? + # This check will ensure 'key_dict' has the appropriate number of objects + # and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + # The key type of 'key_dict' must be either 'rsa' or 'ed25519'. + tuf.formats.ANYKEY_SCHEMA.check_match(key_dict) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified + # in 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY' and 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Signing the 'data' object requires a private key. + # The 'RSASSA-PSS' (i.e., PyCrypto module) and 'ed25519' (i.e., PyNaCl and the + # optimized pure Python implementation of ed25519) are the only signing + # methods currently supported. + signature = {} + keytype = key_dict['keytype'] + public = key_dict['keyval']['public'] + private = key_dict['keyval']['private'] + keyid = key_dict['keyid'] + method = None + sig = None + + # Convert 'data' to canonical JSON format so that repeatable signatures are + # generated across different platforms and Python key dictionaries. The + # resulting 'data' is a string encoded in UTF-8 and compatible with the input + # expected by the cryptography functions called below. + data = tuf.formats.encode_canonical(data) + + # Call the appropriate cryptography libraries for the supported key types, + # otherwise raise an exception. + if keytype == 'rsa': + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + sig, method = tuf.pycrypto_keys.create_rsa_signature(private, data) + else: + message = 'Unsupported "tuf.conf.RSA_CRYPTO_LIBRARY": '+\ + repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + elif keytype == 'ed25519': + public = binascii.unhexlify(public) + private = binascii.unhexlify(private) + if _ED25519_CRYPTO_LIBRARY == 'pynacl' \ + and 'pynacl' in _available_crypto_libraries: + sig, method = tuf.ed25519_keys.create_signature(public, private, + data, use_pynacl=True) + + # Fall back to using the optimized pure python implementation of ed25519. + else: + sig, method = tuf.ed25519_keys.create_signature(public, private, + data, use_pynacl=False) + else: + raise TypeError('Invalid key type.') + + # Build the signature dictionary to be returned. + # The hexadecimal representation of 'sig' is stored in the signature. + signature['keyid'] = keyid + signature['method'] = method + signature['sig'] = binascii.hexlify(sig) + + return signature + + + + + +def verify_signature(key_dict, signature, data): + """ + + Determine whether the private key belonging to 'key_dict' produced + 'signature'. verify_signature() will use the public key found in + 'key_dict', the 'method' and 'sig' objects contained in 'signature', + and 'data' to complete the verification. + + >>> ed25519_key = generate_ed25519_key() + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature = create_signature(ed25519_key, data) + >>> verify_signature(ed25519_key, signature, data) + True + >>> verify_signature(ed25519_key, signature, 'bad_data') + False + >>> rsa_key = generate_rsa_key() + >>> signature = create_signature(rsa_key, data) + >>> verify_signature(rsa_key, signature, data) + True + >>> verify_signature(rsa_key, signature, 'bad_data') + False + + + key_dict: + A dictionary containing the TUF keys and other identifying information. + If 'key_dict' is an RSA key, it has the form: + + {'keytype': 'rsa', + 'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are strings in PEM format. + + signature: + The signature dictionary produced by one of the key generation functions. + 'signature' has the form: + + {'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'method': 'method', + 'sig': sig}. + + Conformant to 'tuf.formats.SIGNATURE_SCHEMA'. + + data: + Data object used by tuf.rsa_key.create_signature() to generate + 'signature'. 'data' is needed here to verify the signature. + + + tuf.FormatError, raised if either 'key_dict' or 'signature' are improperly + formatted. + + tuf.UnsupportedLibraryError, if an unsupported or unavailable library is + detected. + + tuf.UnknownMethodError. Raised if the signing method used by + 'signature' is not one supported. + + + The cryptography library specified in 'tuf.conf' called to do the actual + verification. + + + Boolean. True if the signature is valid, False otherwise. + """ + + # Does 'key_dict' have the correct format? + # This check will ensure 'key_dict' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ANYKEY_SCHEMA.check_match(key_dict) + + # Does 'signature' have the correct format? + tuf.formats.SIGNATURE_SCHEMA.check_match(signature) + + # Using the public key belonging to 'key_dict' + # (i.e., rsakey_dict['keyval']['public']), verify whether 'signature' + # was produced by key_dict's corresponding private key + # key_dict['keyval']['private']. + method = signature['method'] + sig = signature['sig'] + sig = binascii.unhexlify(sig) + public = key_dict['keyval']['public'] + keytype = key_dict['keytype'] + valid_signature = False + + # Convert 'data' to canonical JSON format so that repeatable signatures are + # generated across different platforms and Python key dictionaries. The + # resulting 'data' is a string encoded in UTF-8 and compatible with the input + # expected by the cryptography functions called below. + data = tuf.formats.encode_canonical(data) + + # Call the appropriate cryptography libraries for the supported key types, + # otherwise raise an exception. + if keytype == 'rsa': + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + valid_signature = tuf.pycrypto_keys.verify_rsa_signature(sig, method, + public, data) + else: + message = 'Unsupported "tuf.conf.RSA_CRYPTO_LIBRARY": '+\ + repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + elif keytype == 'ed25519': + public = binascii.unhexlify(public) + if _ED25519_CRYPTO_LIBRARY == 'pynacl' or \ + 'pynacl' in _available_crypto_libraries: + valid_signature = tuf.ed25519_keys.verify_signature(public, + method, sig, data, + use_pynacl=True) + # Fall back to the optimized pure python implementation of ed25519. + else: + valid_signature = tuf.ed25519_keys.verify_signature(public, + method, sig, data, + use_pynacl=False) + else: + raise TypeError('Unsupported key type.') + + return valid_signature + + + + + +def import_rsakey_from_encrypted_pem(encrypted_pem, password): + """ + + Generate public and private RSA keys, with modulus length 'bits'. In + addition, a keyid identifier for the RSA key is generated. The object + returned conforms to 'tuf.formats.RSAKEY_SCHEMA' and has the + form: + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are strings in PEM format. + + Although the PyCrypto crytography library called sets a 1024-bit minimum + key size, generate() enforces a minimum key size of 2048 bits. If 'bits' is + unspecified, a 3072-bit RSA key is generated, which is the key size + recommended by TUF. + + >>> rsa_key = generate_rsa_key() + >>> private = rsa_key['keyval']['private'] + >>> passphrase = 'secret' + >>> encrypted_pem = create_rsa_encrypted_pem(private, passphrase) + >>> rsa_key2 = import_rsakey_from_encrypted_pem(encrypted_pem, passphrase) + >>> rsa_key == rsa_key2 + True + + + encrypted_pem: + The key size, or key length, of the RSA key. 'bits' must be 2048, or + greater, and a multiple of 256. + + password: + + + tuf.FormatError, if 'bits' is improperly or invalid (i.e., not an integer + and not at least 2048). + + tuf.UnsupportedLibraryError, if any of the cryptography libraries specified + in 'tuf.conf.py' are unsupported or unavailable. + + ValueError, if an exception occurs after calling the RSA key generation + routine. 'bits' must be a multiple of 256. The 'ValueError' exception is + raised by the key generation function of the cryptography library called. + + + The RSA keys are generated by calling PyCrypto's + Crypto.PublicKey.RSA.generate(). + + + A dictionary containing the RSA keys and other identifying information. + Conforms to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'encrypted_pem' have the correct format? + # This check will ensure 'encrypted_pem' conforms to + # 'tuf.formats.PEMRSA_SCHEMA'. + tuf.formats.PEMRSA_SCHEMA.check_match(encrypted_pem) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY' and 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Begin building the RSA key dictionary. + rsakey_dict = {} + keytype = 'rsa' + public = None + private = None + + # Generate the public and private RSA keys. The PyCrypto module performs + # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 + # or not a multiple of 256, although a 2048-bit minimum is enforced by + # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + public, private = \ + tuf.pycrypto_keys.create_rsa_public_and_private_from_encrypted_pem(encrypted_pem, + password) + else: + message = 'Invalid crypto library: '+repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + # Generate the keyid of the RSA key. 'key_value' corresponds to the + # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key + # information is not included in the generation of the 'keyid' identifier. + key_value = {'public': public, + 'private': ''} + keyid = _get_keyid(keytype, key_value) + + # Build the 'rsakey_dict' dictionary. Update 'key_value' with the RSA + # private key prior to adding 'key_value' to 'rsakey_dict'. + key_value['private'] = private + + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def format_rsakey_from_pem(pem): + """ + + Generate public and private RSA keys, with modulus length 'bits'. In + addition, a keyid identifier for the RSA key is generated. The object + returned conforms to 'tuf.formats.RSAKEY_SCHEMA' and has the + form: + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': ''}} + + The public and private keys are strings in PEM format. + + Although the PyCrypto crytography library called sets a 1024-bit minimum + key size, generate() enforces a minimum key size of 2048 bits. If 'bits' is + unspecified, a 3072-bit RSA key is generated, which is the key size + recommended by TUF. + + >>> + >>> + >>> + + + pem: + The key size, or key length, of the RSA key. 'bits' must be 2048, or + greater, and a multiple of 256. + + + tuf.FormatError, if 'bits' is improperly or invalid (i.e., not an integer + and not at least 2048). + + tuf.UnsupportedLibraryError, if any of the cryptography libraries specified + in 'tuf.conf.py' are unsupported or unavailable. + + ValueError, if an exception occurs after calling the RSA key generation + routine. 'bits' must be a multiple of 256. The 'ValueError' exception is + raised by the key generation function of the cryptography library called. + + + The RSA keys are generated by calling PyCrypto's + Crypto.PublicKey.RSA.generate(). + + + A dictionary containing the RSA keys and other identifying information. + Conforms to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'pem' have the correct format? + # This check will ensure 'pem' conforms to + # 'tuf.formats.PEMRSA_SCHEMA'. + tuf.formats.PEMRSA_SCHEMA.check_match(pem) + + # Begin building the RSA key dictionary. + rsakey_dict = {} + keytype = 'rsa' + public = pem + + # Generate the keyid of the RSA key. 'key_value' corresponds to the + # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key + # information is not included in the generation of the 'keyid' identifier. + key_value = {'public': public, + 'private': ''} + keyid = _get_keyid(keytype, key_value) + + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def encrypt_key(key_object, password): + """ + + Return a string containing 'key_object' in encrypted form. Encrypted strings + may be safely saved to a file. The corresponding decrypt_key() function can + be applied to the encrypted string to restore the original key object. + 'key_object' is a TUF key (e.g., RSAKEY_SCHEMA, ED25519KEY_SCHEMA). This + function calls the appropriate cryptography module (e.g., pycrypto_keys.py) + to perform the encryption. + + The currently supported general-purpose crypto module, 'pycrypto_keys.py', + performs the actual cryptographic operation on 'key_object'. Whereas + an encrypted PEM file uses the Triple Data Encryption Algorithm (3DES), the + Cipher-block chaining (CBC) mode of operation, and the Password-Based Key + Derivation Function 1 (PBKF1) + MD5 to strengthen 'password', encrypted + TUF keys use AES-256-CTR-Mode and passwords strengthened with + PBKDF2-HMAC-SHA256 (100K iterations by default, but may be overriden in + 'tuf.conf.PBKDF2_ITERATIONS' by the user). + + http://en.wikipedia.org/wiki/Advanced_Encryption_Standard + http://en.wikipedia.org/wiki/CTR_mode#Counter_.28CTR.29 + https://en.wikipedia.org/wiki/PBKDF2 + + >>> ed25519_key = generate_ed25519_key() + >>> password = 'secret' + >>> encrypted_key = encrypt_key(ed25519_key, password) + >>> tuf.formats.ENCRYPTEDKEY_SCHEMA.matches(encrypted_key) + True + + + key_object: + A TUF key (containing also the private key portion) of the form + 'tuf.formats.ANYKEY_SCHEMA' + + password: + The password, or passphrase, to encrypt 'the private part of the RSA + key. 'password' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if 'key_object' cannot be encrypted. + + tuf.UnsupportedLibraryError, if the general-purpose cryptography library + specified in 'tuf.conf.GENERAL_CRYPTO_LIBRARY' is unsupported. + + + Perform crytographic operations using the library specified in + 'tuf.formats.GENERAL_CRYPTO_LIBRARY' and 'password'. + + + An encrypted string of the form: 'tuf.formats.ENCRYPTEDKEY_SCHEMA'. + """ + + # Does 'key_object' have the correct format? + # This check will ensure 'key_object' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ANYKEY_SCHEMA.check_match(key_object) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.GENERAL_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Encrypted string of 'key_object'. The encrypted string may be safely saved + # to a file and stored offline. + encrypted_key = None + + # Generate an encrypted string of 'key_object' using AES-256-CTR-Mode, where + # 'password' is strengthened with PBKDF2-HMAC-SHA256. Ensure the general- + # purpose library specified in 'tuf.conf.GENERAL_CRYPTO_LIBRARY' is supported. + if _GENERAL_CRYPTO_LIBRARY == 'pycrypto': + encrypted_key = \ + tuf.pycrypto_keys.encrypt_key(key_object, password) + else: + message = 'Invalid crypto library: '+repr(_GENERAL_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + return encrypted_key + + + + + +def decrypt_key(encrypted_key, passphrase): + """ + + Return a string containing 'encrypted_key' in non-encrypted form. + The decrypt_key() function can be applied to the encrypted string to restore + the original key object, a TUF key (e.g., RSAKEY_SCHEMA, ED25519KEY_SCHEMA). + This function calls the appropriate cryptography module (e.g., + pycrypto_keys.py) to perform the decryption. + + The currently supported general-purpose crypto module, 'pycrypto_keys.py', + performs the actual cryptographic operation on 'key_object'. Whereas + an encrypted PEM file uses the Triple Data Encryption Algorithm (3DES), the + Cipher-block chaining (CBC) mode of operation, and the Password-Based Key + Derivation Function 1 (PBKF1) + MD5 to strengthen 'password', encrypted + TUF keys use AES-256-CTR-Mode and passwords strengthened with + PBKDF2-HMAC-SHA256 (100K iterations be default, but may be overriden in + 'tuf.conf.py' by the user). + + http://en.wikipedia.org/wiki/Advanced_Encryption_Standard + http://en.wikipedia.org/wiki/CTR_mode#Counter_.28CTR.29 + https://en.wikipedia.org/wiki/PBKDF2 + + >>> ed25519_key = generate_ed25519_key() + >>> password = 'secret' + >>> encrypted_key = encrypt_key(ed25519_key, password) + >>> decrypted_key = decrypt_key(encrypted_key, password) + >>> tuf.formats.ANYKEY_SCHEMA.matches(decrypted_key) + True + >>> decrypted_key == ed25519_key + True + + + encrypted_key: + An encrypted TUF key (additional data is also included, such as salt, + number of password iterations used for the derived encryption key, etc) + of the form 'tuf.formats.ENCRYPTEDKEY_SCHEMA'. 'encrypted_key' should + have been generated with encrypted_key(). + + password: + The password, or passphrase, to decrypt 'encrypted_key'. 'password' is + not used directly as the encryption key, a stronger encryption key is + derived from it. The supported general-purpose module takes care of + re-deriving the encryption key. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if 'encrypted_key' cannot be decrypted. + + tuf.UnsupportedLibraryError, if the general-purpose cryptography library + specified in 'tuf.conf.GENERAL_CRYPTO_LIBRARY' is unsupported. + + + Perform crytographic operations using the library specified in + 'tuf.formats.GENERAL_CRYPTO_LIBRARY' and 'password'. + + + A TUF key object of the form: 'tuf.formats.ANYKEY_SCHEMA' (e.g., + RSAKEY_SCHEMA, ED25519KEY_SCHEMA). + """ + + # Does 'encrypted_key' have the correct format? + # This check ensures 'encrypted_key' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ENCRYPTEDKEY_SCHEMA.check_match(encrypted_key) + + # Does 'passphrase' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.GENERAL_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Store and return the decrypted key object. + key_object = None + + # Decrypt 'encrypted_key' so that the original key object is restored. + # encrypt_key() generates an encrypted string of the TUF key object using + # AES-256-CTR-Mode, where 'password' is strengthened with PBKDF2-HMAC-SHA256. + # Ensure the general-purpose library specified in + # 'tuf.conf.GENERAL_CRYPTO_LIBRARY' is supported. + if _GENERAL_CRYPTO_LIBRARY == 'pycrypto': + key_object = \ + tuf.pycrypto_keys.decrypt_key(encrypted_key, passphrase) + else: + message = 'Invalid crypto library: '+repr(_GENERAL_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + # The corresponding encrypt_key() encrypts and stores key objects in + # non-metadata format (i.e., original format of key object argument to + # encrypt_key()) prior to returning. + + return key_object + + + + + +def create_rsa_encrypted_pem(private_key, passphrase): + """ + + Return a string in PEM format, where the private part of the RSA key is + encrypted. The private part of the RSA key is encrypted by the Triple + Data Encryption Algorithm (3DES) and Cipher-block chaining (CBC) for the + mode of operation. Password-Based Key Derivation Function 1 (PBKF1) + MD5 + is used to strengthen 'passphrase'. + + https://en.wikipedia.org/wiki/Triple_DES + https://en.wikipedia.org/wiki/PBKDF2 + + >>> rsa_key = generate_rsa_key() + >>> private = rsa_key['keyval']['private'] + >>> passphrase = 'secret' + >>> encrypted_pem = create_rsa_encrypted_pem(private, passphrase) + >>> tuf.formats.PEMRSA_SCHEMA.matches(encrypted_pem) + True + + + private_key: + The private key string in PEM format. + + passphrase: + The passphrase, or password, to encrypt the private part of the RSA + key. 'passphrase' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if an RSA key in encrypted PEM format cannot be created. + + TypeError, 'private_key' is unset. + + + PyCrypto's Crypto.PublicKey.RSA.exportKey() called to perform the actual + generation of the PEM-formatted output. + + + A string in PEM format, where the private RSA key is encrypted. + Conforms to 'tuf.formats.PEMRSA_SCHEMA'. + """ + + # Does 'private_key' have the correct format? + # This check will ensure 'private_key' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(private_key) + + # Does 'passphrase' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + encrypted_pem = None + + # Generate the public and private RSA keys. The PyCrypto module performs + # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 + # or not a multiple of 256, although a 2048-bit minimum is enforced by + # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + encrypted_pem = \ + tuf.pycrypto_keys.create_rsa_encrypted_pem(private_key, passphrase) + else: + message = 'Invalid crypto library: '+repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + return encrypted_pem + + + +if __name__ == '__main__': + # The interactive sessions of the documentation strings can + # be tested by running 'keys.py' as a standalone module: + # $ python keys.py + import doctest + doctest.testmod() diff --git a/tuf/libtuf.py b/tuf/libtuf.py new file mode 100755 index 00000000..1dee636a --- /dev/null +++ b/tuf/libtuf.py @@ -0,0 +1,4339 @@ +""" + + libtuf.py + + + Vladimir Diaz + + + October 19, 2013 + + + See LICENSE for licensing information. + + + See 'tuf/README' for a complete guide on using 'tuf.libtuf.py'. +""" + +# Help with Python 3 compatibility, where the print statement is a function, an +# implicit relative import is invalid, and the '/' operator performs true +# division. Example: print 'hello world' raises a 'SyntaxError' exception. +from __future__ import print_function +from __future__ import absolute_import +from __future__ import division + +import os +import errno +import sys +import time +import getpass +import logging +import tempfile +import shutil +import json +import gzip +import random + +import tuf +import tuf.formats +import tuf.util +import tuf.keydb +import tuf.roledb +import tuf.keys +import tuf.sig +import tuf.log +import tuf.conf + + +# See 'log.py' to learn how logging is handled in TUF. +logger = logging.getLogger('tuf.libtuf') + +# Recommended RSA key sizes: +# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 +# According to the document above, revised May 6, 2003, RSA keys of +# size 3072 provide security through 2031 and beyond. 2048-bit keys +# are the recommended minimum and are good from the present through 2030. +DEFAULT_RSA_KEY_BITS = 3072 + +# The algorithm used by the repository to generate the hashes of the +# target filepaths. The repository may optionally organize targets into +HASH_FUNCTION = 'sha256' + +# The extension of TUF metadata. +METADATA_EXTENSION = '.txt' + +# The metadata filenames of the top-level roles. +ROOT_FILENAME = 'root' + METADATA_EXTENSION +TARGETS_FILENAME = 'targets' + METADATA_EXTENSION +RELEASE_FILENAME = 'release' + METADATA_EXTENSION +TIMESTAMP_FILENAME = 'timestamp' + METADATA_EXTENSION + +# The targets and metadata directory names. Metadata files are written +# to the staged metadata directory instead of the "live" one. +METADATA_STAGED_DIRECTORY_NAME = 'metadata.staged' +METADATA_DIRECTORY_NAME = 'metadata' +TARGETS_DIRECTORY_NAME = 'targets' + +# The full list of supported TUF metadata extensions. +METADATA_EXTENSIONS = ['.txt', '.txt.gz'] + +# The recognized compression extensions. +SUPPORTED_COMPRESSION_EXTENSIONS = ['.gz'] + +# Supported key types. +SUPPORTED_KEY_TYPES = ['rsa', 'ed25519'] + +# Expiration date delta, in seconds, of the top-level roles. A metadata +# expiration date is set by taking the current time and adding the expiration +# seconds listed below. + +# Initial 'root.txt' expiration time of 1 year. +ROOT_EXPIRATION = 31556900 + +# Initial 'targets.txt' expiration time of 3 months. +TARGETS_EXPIRATION = 7889230 + +# Initial 'release.txt' expiration time of 1 week. +RELEASE_EXPIRATION = 604800 + +# Initial 'timestamp.txt' expiration time of 1 day. +TIMESTAMP_EXPIRATION = 86400 + + +class Repository(object): + """ + + Represent a TUF repository that contains the metadata of the top-level + roles, including all those delegated from the 'targets.txt' role. The + repository object returned provides access to the top-level roles, and any + delegated targets that are added as the repository is modified. For + example, a Repository object named 'repository' provides the following + access by default: + + repository.root.version = 2 + repository.timestamp.expiration = "2015-08-08 12:00:00" + repository.release.add_key(...) + repository.targets.delegate('unclaimed', ...) + + Delegating a role from 'targets' updates the attributes of the parent + delegation, which then provides: + + repository.targets('unclaimed').add_key(...) + + + + repository_directory: + The root folder of the repository that contains the metadata and targets + sub-directories. + + metadata_directory: + The metadata sub-directory contains the files of the top-level + roles, including all roles delegated from 'targets.txt'. + + targets_directory: + The targets sub-directory contains all the target files that are + downloaded by clients and are referenced in TUF Metadata. The hashes and + file lengths are listed in Metadata files so that they are securely + downloaded. Metadata files are similarly referenced in the top-level + metadata. + + + tuf.FormatError, if the arguments are improperly formatted. + + + Creates top-level role objects and assigns them as attributes. + + + A Repository object that contains default Metadata objects for the top-level + roles. + """ + + def __init__(self, repository_directory, metadata_directory, targets_directory): + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.PATH_SCHEMA.check_match(repository_directory) + tuf.formats.PATH_SCHEMA.check_match(metadata_directory) + tuf.formats.PATH_SCHEMA.check_match(targets_directory) + + self._repository_directory = repository_directory + self._metadata_directory = metadata_directory + self._targets_directory = targets_directory + + # Set the top-level role objects. + self.root = Root() + self.release = Release() + self.timestamp = Timestamp() + self.targets = Targets(self._targets_directory, 'targets') + + + + def write(self, write_partial=False, consistent_snapshots=False): + """ + + Write all the JSON Metadata objects to their corresponding files. + write() raises an exception if any of the role metadata to be written to + disk is invalid, such as an insufficient threshold of signatures, missing + private keys, etc. + + + write_partial: + A boolean indicating whether partial metadata should be written to + disk. Partial metadata may be written to allow multiple maintainters + to independently sign and update role metadata. write() raises an + exception if a metadata role cannot be written due to not having enough + signatures. + + consistent_snapshots: + A boolean indicating whether written metadata and target files should + include a digest in the filename (i.e., root..txt, + targets..txt.gz, README..txt, where is the + file's SHA256 digest. Example: + 'root.1f4e35a60c8f96d439e27e858ce2869c770c1cdd54e1ef76657ceaaf01da18a3.txt' + + + tuf.Error, if any of the top-level roles do not have a minimum + threshold of signatures. + + + Creates metadata files in the repository's metadata directory. + + + None. + """ + + # Does 'write_partial' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.BOOLEAN_SCHEMA.check_match(write_partial) + tuf.formats.BOOLEAN_SCHEMA.check_match(consistent_snapshots) + + # At this point the tuf.keydb and tuf.roledb stores must be fully + # populated, otherwise write() throwns a 'tuf.Repository' exception if + # any of the top-level roles are missing signatures, keys, etc. + + # Write the metadata files of all the delegated roles. + delegated_rolenames = tuf.roledb.get_delegated_rolenames('targets') + for delegated_rolename in delegated_rolenames: + roleinfo = tuf.roledb.get_roleinfo(delegated_rolename) + delegated_filename = os.path.join(self._metadata_directory, + delegated_rolename + METADATA_EXTENSION) + + # Ensure the parent directories of 'metadata_filepath' exist, otherwise an + # IO exception is raised if 'metadata_filepath' is written to a + # sub-directory. + tuf.util.ensure_parent_dir(delegated_filename) + + _generate_and_write_metadata(delegated_rolename, delegated_filename, + write_partial, self._targets_directory, + self._metadata_directory, + consistent_snapshots) + + # Generate the 'root.txt' metadata file. + # _generate_and_write_metadata() raises a 'tuf.Error' exception if the + # metadata cannot be written. + root_filename = 'root' + METADATA_EXTENSION + root_filename = os.path.join(self._metadata_directory, root_filename) + signable_junk, root_filename = \ + _generate_and_write_metadata('root', root_filename, write_partial, + self._targets_directory, + self._metadata_directory, + consistent_snapshots) + + # Generate the 'targets.txt' metadata file. + targets_filename = 'targets' + METADATA_EXTENSION + targets_filename = os.path.join(self._metadata_directory, targets_filename) + signable_junk, targets_filename = \ + _generate_and_write_metadata('targets', targets_filename, write_partial, + self._targets_directory, + self._metadata_directory, + consistent_snapshots) + + # Generate the 'release.txt' metadata file. + release_filename = os.path.join(self._metadata_directory, 'release') + release_filename = 'release' + METADATA_EXTENSION + release_filename = os.path.join(self._metadata_directory, release_filename) + filenames = {'root': root_filename, 'targets': targets_filename} + release_signable, release_filename = \ + _generate_and_write_metadata('release', release_filename, write_partial, + self._targets_directory, + self._metadata_directory, + consistent_snapshots, filenames) + + # Generate the 'timestamp.txt' metadata file. + timestamp_filename = 'timestamp' + METADATA_EXTENSION + timestamp_filename = os.path.join(self._metadata_directory, timestamp_filename) + filenames = {'release': release_filename} + _generate_and_write_metadata('timestamp', timestamp_filename, write_partial, + self._targets_directory, + self._metadata_directory, consistent_snapshots, + filenames) + + # Delete the metadata of roles no longer in 'tuf.roledb'. Obsolete roles + # may have been revoked. + _delete_obsolete_metadata(self._metadata_directory, + release_signable['signed'], consistent_snapshots) + + + + def write_partial(self): + """ + + Write all the JSON Metadata objects to their corresponding files, but + allow metadata files to contain an invalid threshold of signatures. + + + None. + + + None. + + + Creates metadata files in the repository's metadata directory. + + + None. + """ + + self.write(write_partial=True) + + + + def status(self): + """ + + Determine the status of the top-level roles, including those delegated. + status() checks if each role provides sufficient public keys, signatures, + and that a valid metadata file is generated if write() were to be called. + Metadata files are temporary written to check that proper metadata files + are written, where file hashes and lengths are calculated and referenced + by the top-level roles. status() does not do a simple check for number + of threshold keys and signatures. + + + None. + + + None. + + + Generates and writes temporary metadata files. + + + None. + """ + + temp_repository_directory = None + + try: + temp_repository_directory = tempfile.mkdtemp() + metadata_directory = os.path.join(temp_repository_directory, + METADATA_STAGED_DIRECTORY_NAME) + os.mkdir(metadata_directory) + + filenames = get_metadata_filenames(metadata_directory) + + # Delegated roles. + delegated_roles = tuf.roledb.get_delegated_rolenames('targets') + insufficient_keys = [] + insufficient_signatures = [] + + for delegated_role in delegated_roles: + try: + _check_role_keys(delegated_role) + except tuf.InsufficientKeysError, e: + insufficient_keys.append(delegated_role) + continue + + roleinfo = tuf.roledb.get_roleinfo(delegated_role) + try: + write_delegated_metadata_file(temp_repository_directory, + self._targets_directory, + delegated_role, roleinfo, + write_partial=False) + except tuf.Error, e: + insufficient_signatures.append(delegated_role) + + if len(insufficient_keys): + message = 'Delegated roles with insufficient keys: '+ \ + repr(insufficient_keys) + print(message) + return + + if len(insufficient_signatures): + message = 'Delegated roles with insufficient signatures: '+ \ + repr(insufficient_signatures) + print(message) + return + + # Root role. + try: + _check_role_keys(self.root.rolename) + except tuf.InsufficientKeysError, e: + print(str(e)) + return + + try: + signable = _generate_and_write_metadata(self.root.rolename, + filenames, False, + self._targets_directory, + metadata_directory) + _print_status(self.root.rolename, signable) + except tuf.Error, e: + signable = e[1] + _print_status(self.root.rolename, signable) + return + + # Targets role. + try: + _check_role_keys(self.targets.rolename) + except tuf.InsufficientKeysError, e: + print(str(e)) + return + + try: + signable = _generate_and_write_metadata(self.targets.rolename, + filenames, False, + self._targets_directory, + metadata_directory) + _print_status(self.targets.rolename, signable) + except tuf.Error, e: + signable = e[1] + _print_status(self.targets.rolename, signable) + return + + # Release role. + try: + _check_role_keys(self.release.rolename) + except tuf.InsufficientKeysError, e: + print(str(e)) + return + + try: + signable = _generate_and_write_metadata(self.release.rolename, + filenames, False, + self._targets_directory, + metadata_directory) + _print_status(self.release.rolename, signable) + except tuf.Error, e: + signable = e[1] + _print_status(self.release.rolename, signable) + return + + # Timestamp role. + try: + _check_role_keys(self.timestamp.rolename) + except tuf.InsufficientKeysError, e: + print(str(e)) + return + + try: + signable = _generate_and_write_metadata(self.timestamp.rolename, + filenames, False, + self._targets_directory, + metadata_directory) + _print_status(self.timestamp.rolename, signable) + except tuf.Error, e: + signable = e[1] + _print_status(self.timestamp.rolename, signable) + return + + finally: + shutil.rmtree(temp_repository_directory, ignore_errors=True) + + + + def get_filepaths_in_directory(self, files_directory, recursive_walk=False, + followlinks=True): + """ + + Walk the given 'files_directory' and build a list of target files found. + + + files_directory: + The path to a directory of target files. + + recursive_walk: + To recursively walk the directory, set recursive_walk=True. + + followlinks: + To follow symbolic links, set followlinks=True. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.Error, if 'file_directory' is not a valid directory. + + Python IO exceptions. + + + None. + + + A list of absolute paths to target files in the given 'files_directory'. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.PATH_SCHEMA.check_match(files_directory) + tuf.formats.BOOLEAN_SCHEMA.check_match(recursive_walk) + tuf.formats.BOOLEAN_SCHEMA.check_match(followlinks) + + # Ensure a valid directory is given. + if not os.path.isdir(files_directory): + message = repr(files_directory)+' is not a directory.' + raise tuf.Error(message) + + # A list of the target filepaths found in 'file_directory'. + targets = [] + + # FIXME: We need a way to tell Python 2, but not Python 3, to return + # filenames in Unicode; see #61 and: + # http://docs.python.org/2/howto/unicode.html#unicode-filenames + for dirpath, dirnames, filenames in os.walk(files_directory, + followlinks=followlinks): + for filename in filenames: + full_target_path = os.path.join(dirpath, filename) + targets.append(full_target_path) + + # Prune the subdirectories to walk right now if we do not wish to + # recursively walk files_directory. + if recursive_walk is False: + del dirnames[:] + + return targets + + + + + +class Metadata(object): + """ + + Provide a base class to represent a TUF Metadata role. There are four + top-level roles: Root, Targets, Release, and Timestamp. The Metadata class + provides methods that are needed by all top-level roles, such as adding + and removing public keys, private keys, and signatures. Metadata + attributes, such as rolename, version, threshold, expiration, key list, and + compressions, is also provided by the Metadata base class. + + + None. + + + None. + + + None. + + + None. + """ + + def __init__(self): + self._rolename = None + + + + def add_key(self, key): + """ + + Add 'key' to the role. Adding a key, which should contain only the public + portion, signifies the corresponding private key and signatures the role + is expected to provide. A threshold of signatures is required for a role + to be considered properly signed. If a metadata file contains an + insufficient threshold of signatures, it must not be accepted. + + >>> + >>> + >>> + + + key: + The role key to be added, conformant to 'tuf.formats.ANYKEY_SCHEMA'. + Adding a public key to a role means that its corresponding private key + must generate and add its signature to the role. A threshold number of + signatures is required for a role to be fully signed. + + + tuf.FormatError, if the 'key' argument is improperly formatted. + + + The role's entries in 'tuf.keydb.py' and 'tuf.roledb.py' are updated. + + + None. + """ + + # Does 'key' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ANYKEY_SCHEMA.check_match(key) + + # Ensure 'key', which should contain the public portion, is added to + # 'tuf.keydb.py'. + try: + tuf.keydb.add_key(key) + except tuf.KeyAlreadyExistsError, e: + pass + + keyid = key['keyid'] + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + # Add 'key' to the role's entry in 'tuf.roledb.py' and avoid duplicates. + if keyid not in roleinfo['keyids']: + roleinfo['keyids'].append(keyid) + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + + + def remove_key(self, key): + """ + + Remove 'key' from the role's currently recognized list of role keys. + The role expects a threshold number of signatures + + >>> + >>> + >>> + + + key: + The role's key, conformant to 'tuf.formats.ANYKEY_SCHEMA'. 'key' + should contain the only the public portion, as only the public key + is needed. The 'add_key()' method should have previously added 'key'. + + + tuf.FormatError, if the 'key' argument is improperly formatted. + + + Updates the role's 'tuf.roledb.py' entry. + + + None. + """ + + # Does 'key' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ANYKEY_SCHEMA.check_match(key) + + keyid = key['keyid'] + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + if keyid in roleinfo['keyids']: + roleinfo['keyids'].remove(keyid) + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + + + def load_signing_key(self, key): + """ + + Load the role key, which must contain the private portion, so that role + signatures may be generated when the role's metadata file is eventually + written to disk. + + >>> + >>> + >>> + + + key: + The role's key, conformant to 'tuf.formats.ANYKEY_SCHEMA'. It must + contain the private key, so that role signatures may be generated when + write() or write_partial() is eventually called to generate valid + metadata files. + + + tuf.FormatError, if 'key' is improperly formatted. + + tuf.Error, if the private key is not found in 'key'. + + + Updates the role's 'tuf.keydb.py' and 'tuf.roledb.py' entries. + + + None. + """ + + # Does 'key' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ANYKEY_SCHEMA.check_match(key) + + # Ensure the private portion of the key is available, otherwise signatures + # cannot be generated when the metadata file is written to disk. + if not len(key['keyval']['private']): + message = 'The private key is unavailable.' + raise tuf.Error(message) + + # Has the key, with the private portion included, been added to the keydb? + # The public version of the key may have been previously added. + try: + tuf.keydb.add_key(key) + except tuf.KeyAlreadyExistsError, e: + tuf.keydb.remove_key(key['keyid']) + tuf.keydb.add_key(key) + + # Update the role's 'signing_keys' field in 'tuf.roledb.py'. + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + if key['keyid'] not in roleinfo['signing_keyids']: + roleinfo['signing_keyids'].append(key['keyid']) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def unload_signing_key(self, key): + """ + + Remove a previously loaded role private key (i.e., load_signing_key()). + The keyid of the 'key' is removed the list of signing keys recognized. + + >>> + >>> + >>> + + + key: + The role key to be unloaded, conformant to 'tuf.formats.ANYKEY_SCHEMA'. + + + tuf.FormatError, if the 'key' argument is improperly formatted. + + + Updates the signing keys of the role in 'tuf.roledb.py'. + + + None. + """ + + # Does 'key' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ANYKEY_SCHEMA.check_match(key) + + # Update the role's 'signing_keys' field in 'tuf.roledb.py'. + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + if key['keyid'] in roleinfo['signing_keyids']: + roleinfo['signing_keyids'].remove(key['keyid']) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def add_signature(self, signature): + """ + + Add a signature to the role. A role is considered fully signed if it + contains a threshold of signatures. The 'signature' should have been + generated by the private key corresponding to one of the role's expected + keys. + + >>> + >>> + >>> + + + signature: + The signature to be added to the role, conformant to + 'tuf.formats.SIGNATURE_SCHEMA'. + + + tuf.FormatError, if the 'signature' argument is improperly formatted. + + + Adds 'signature', if not already added, to the role's 'signatures' field + in 'tuf.roledb.py'. + + + None. + """ + + # Does 'signature' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.SIGNATURE_SCHEMA.check_match(signature) + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + # Ensure the roleinf contains a 'signatures' field. + if 'signatures' not in roleinfo: + roleinfo['signatures'] = [] + + # Update the role's roleinfo by adding 'signature', if it has not been + # added. + if signature not in roleinfo['signatures']: + roleinfo['signatures'].append(signature) + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def remove_signature(self, signature): + """ + + Remove a previously loaded, or added, role 'signature'. A role must + contain a threshold number of signatures to be considered fully signed. + + >>> + >>> + >>> + + + signature: + The role signature to remove, conformant to + 'tuf.formats.SIGNATURE_SCHEMA'. + + + tuf.FormatError, if the 'signature' argument is improperly formatted. + + + Updates the 'signatures' field of the role in 'tuf.roledb.py'. + + + None. + """ + + # Does 'signature' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.SIGNATURE_SCHEMA.check_match(signature) + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + if signature in roleinfo['signatures']: + roleinfo['signatures'].remove(signature) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + @property + def signatures(self): + """ + + A getter method that returns the role's signatures. A role is considered + fully signed if it contains a threshold number of signatures, where each + signature must be provided by the generated by the private key. Keys + are added to a role with the add_key() method. + + + None. + + + None. + + + None. + + + A list of signatures, conformant to 'tuf.formats.SIGNATURES_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + signatures = roleinfo['signatures'] + + return signatures + + + + @property + def keys(self): + """ + + A getter method that returns the role's keyids of the keys. The role + is expected to eventually contain a threshold of signatures generated + by the private keys of each of the role's keys (returned here as a keyid). + + + None. + + + None. + + + None. + + + A list of the role's keyids (i.e., keyids of the keys). + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + keyids = roleinfo['keyids'] + + return keyids + + + + @property + def rolename(self): + """ + + Return the role's name. + Examples: 'root', 'timestamp', 'targets/unclaimed/django'. + + + None. + + + None. + + + None. + + + The role's name, conformant to 'tuf.formats.ROLENAME_SCHEMA'. + Examples: 'root', 'timestamp', 'targets/unclaimed/django'. + """ + + return self._rolename + + + + @property + def version(self): + """ + + A getter method that returns the role's version number, conformant to + 'tuf.formats.VERSION_SCHEMA'. + + + None. + + + None. + + + None. + + + The role's version number, conformant to 'tuf.formats.VERSION_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + version = roleinfo['version'] + + return version + + + + @version.setter + def version(self, version): + """ + + A setter method that updates the role's version number. TUF clients + download new metadata with version number greater than the version + currently trusted. New metadata start at version 1 when either write() + or write_partial() is called. Version numbers are automatically + incremented, when the write methods are called, as follows: + + 1. write_partial==True and the metadata is the first to be written. + + 2. write_partial=False (i.e., write()), the metadata was not loaded as + partially written, and a write_partial is not needed. + + >>> + >>> + >>> + + + version: + The role's version number, conformant to 'tuf.formats.VERSION_SCHEMA'. + + + tuf.FormatError, if the 'version' argument is improperly formatted. + + + Modifies the 'version' attribute of the Repository object and updates + the role's version in 'tuf.roledb.py'. + + + None. + """ + + # Does 'version' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + roleinfo['version'] = version + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + + + @property + def threshold(self): + """ + + Return the role's threshold value. A role is considered fully signed if + a threshold number of signatures is available. + + + None. + + + None. + + + None. + + + The role's threshold value, conformant to 'tuf.formats.THRESHOLD_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + threshold = roleinfo['threshold'] + + return threshold + + + + @threshold.setter + def threshold(self, threshold): + """ + + A setter method that modified the threshold value of the role. Metadata + is considered fully signed if a 'threshold' number of signatures is + available. + + >>> + >>> + >>> + + + threshold: + An integer value that sets the role's threshold value, or the miminum + number of signatures needed for metadata to be considered fully + signed. Conformant to 'tuf.formats.THRESHOLD_SCHEMA'. + + + tuf.FormatError, if the 'threshold' argument is improperly formatted. + + + Modifies the threshold attribute of the Repository object and updates + the roles threshold in 'tuf.roledb.py'. + + + None. + """ + + # Does 'threshold' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.THRESHOLD_SCHEMA.check_match(threshold) + + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + roleinfo['threshold'] = threshold + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + + @property + def expiration(self): + """ + + A getter method that returns the role's expiration datetime. + + >>> + >>> + >>> + + + None. + + + None. + + + None. + + + The role's expiration datetime, conformant to + 'tuf.formats.DATETIME_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + return roleinfo['expires'] + + + + @expiration.setter + def expiration(self, expiration_datetime_utc): + """ + + A setter method for the role's expiration datetime. The top-level + roles have a default expiration (e.g., ROOT_EXPIRATION), but may later + be modified by this setter method. + + TODO: expiration_datetime_utc in ISO 8601 format. + + >>> + >>> + >>> + + + expiration_datetime_utc: + The datetime expiration of the role, conformant to + 'tuf.formats.DATETIME_SCHEMA'. + + + tuf.FormatError, if 'expiration_datetime_utc' is improperly formatted. + + + Modifies the expiration attribute of the Repository object. + + + None. + """ + + # Does 'expiration_datetime_utc' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.DATETIME_SCHEMA.check_match(expiration_datetime_utc) + + # Further validate the datetime, such as a correct date, time, expiration. + # Convert 'expiration_datetime_utc' to a unix timestamp so that it can be + # compared with time.time(). + expiration_datetime_utc = expiration_datetime_utc+' UTC' + try: + unix_timestamp = tuf.formats.parse_time(expiration_datetime_utc) + except (tuf.FormatError, ValueError), e: + message = 'Invalid datetime argument: '+repr(expiration_datetime_utc) + raise tuf.FormatError(message) + + # Ensure the expiration has not already passed. + if unix_timestamp < time.time(): + message = 'The expiration date must occur after the current date.' + raise tuf.FormatError(message) + + # Update the role's 'expires' entry in 'tuf.roledb.py'. + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + roleinfo['expires'] = expiration_datetime_utc + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + @property + def signing_keys(self): + """ + + A getter method that returns a list of the role's signing keys. + + >>> + >>> + >>> + + + None. + + + None. + + + None. + + + A list of keyids of the role's signing keys, conformant to + 'tuf.formats.KEYIDS_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + signing_keyids = roleinfo['signing_keyids'] + + return signing_keyids + + + + @property + def compressions(self): + """ + + A getter method that returns a list of the file compression algorithms + used when the metadata is written to disk. If ['gz'] is set for the + 'targets.txt' role, the metadata files 'targets.txt' and 'targets.txt.gz' + are written. + + >>> + >>> + >>> + + + None. + + + None. + + + None. + + + A list of compression algorithms, conformant to + 'tuf.formats.COMPRESSIONS_SCHEMA'. + """ + + tuf.roledb.get_roleinfo(self.rolename) + compressions = roleinfo['compressions'] + + return compressions + + + + @compressions.setter + def compressions(self, compression_list): + """ + + A setter method for the file compression algorithms used when the + metadata is written to disk. If ['gz'] is set for the 'targets.txt' role + the metadata files 'targets.txt' and 'targets.txt.gz' are written. + + >>> + >>> + >>> + + + compression_list: + A list of file compression algorithms, conformant to + 'tuf.formats.COMPRESSIONS_SCHEMA'. + + + tuf.FormatError, if 'compression_list' is improperly formatted. + + + Updates the role's compression algorithms listed in 'tuf.roledb.py'. + + + None. + """ + + # Does 'compression_name' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.COMPRESSIONS_SCHEMA.check_match(compression_list) + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + # Add the compression algorithms of 'compression_list' to the role's + # entry in 'tuf.roledb.py'. + for compression in compression_list: + if compression not in roleinfo['compressions']: + roleinfo['compressions'].append(compression) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + + +class Root(Metadata): + """ + + Represent a Root role object. The root role is responsible for + listing the public keys and threshold of all the top-level roles, including + itself. Top-level metadata is rejected if it does not comply with what is + specified by the Root role. + + This Root object sub-classes Metadata, so the expected Metadata + operations like adding/removing public keys, signatures, private keys, and + updating metadata attributes (e.g., version and expiration) is supported. + Since Root is a top-level role and must exist, a default Root object + is instantiated when a new Repository object is created. + + >>> + >>> + >>> + + + None. + + + None. + + + A 'root' role is added to 'tuf.roledb.py'. + + + None. + """ + + def __init__(self): + + super(Root, self).__init__() + + self._rolename = 'root' + + # By default, 'release' metadata is set to expire 1 week from the current + # time. The expiration may be modified. + expiration = tuf.formats.format_time(time.time()+ROOT_EXPIRATION) + + roleinfo = {'keyids': [], 'signing_keyids': [], 'threshold': 1, + 'signatures': [], 'version': 0, 'consistent_snapshots': False, + 'compressions': [''], 'expires': expiration, + 'partial_loaded': False} + try: + tuf.roledb.add_role(self._rolename, roleinfo) + except tuf.RoleAlreadyExistsError, e: + pass + + + + + +class Timestamp(Metadata): + """ + + Represent a Timestamp role object. The timestamp role is responsible for + referencing the latest version of the Release role. Under normal + conditions, it is the only role to be downloaded from a remote repository + without a known file length and hash. An upper length limit is set, though. + Also, its signatures are also verified to be valid according to the Root + role. If invalid metadata can only be downloaded by the client, Root + is the only other role that is downloaded without a known length and hash. + This case may occur if a role's signing keys have been revoked and a newer + Root file is needed to list the updated keys. + + This Timestamp object sub-classes Metadata, so the expected Metadata + operations like adding/removing public keys, signatures, private keys, and + updating metadata attributes (e.g., version and expiration) is supported. + Since Release is a top-level role and must exist, a default Timestamp object + is instantiated when a new Repository object is created. + + >>> + >>> + >>> + + + None. + + + None. + + + A 'timestamp' role is added to 'tuf.roledb.py'. + + + None. + """ + + def __init__(self): + + super(Timestamp, self).__init__() + + self._rolename = 'timestamp' + + # By default, 'release' metadata is set to expire 1 week from the current + # time. The expiration may be modified. + expiration = tuf.formats.format_time(time.time()+TIMESTAMP_EXPIRATION) + + roleinfo = {'keyids': [], 'signing_keyids': [], 'threshold': 1, + 'signatures': [], 'version': 0, 'compressions': [''], + 'expires': expiration, 'partial_loaded': False} + + try: + tuf.roledb.add_role(self.rolename, roleinfo) + except tuf.RoleAlreadyExistsError, e: + pass + + + + + +class Release(Metadata): + """ + + Represent a Release role object. The release role is responsible for + referencing the other top-level roles (excluding Timestamp) and all + delegated roles. + + This Release object sub-classes Metadata, so the expected + Metadata operations like adding/removing public keys, signatures, private + keys, and updating metadata attributes (e.g., version and expiration) is + supported. Since Release is a top-level role and must exist, a default + Release object is instantiated when a new Repository object is created. + + >>> + >>> + >>> + + + None. + + + None. + + + A 'release' role is added to 'tuf.roledb.py'. + + + None. + """ + + def __init__(self): + + super(Release, self).__init__() + + self._rolename = 'release' + + # By default, 'release' metadata is set to expire 1 week from the current + # time. The expiration may be modified. + expiration = tuf.formats.format_time(time.time()+RELEASE_EXPIRATION) + + roleinfo = {'keyids': [], 'signing_keyids': [], 'threshold': 1, + 'signatures': [], 'version': 0, 'compressions': [''], + 'expires': expiration, 'partial_loaded': False} + + try: + tuf.roledb.add_role(self._rolename, roleinfo) + except tuf.RoleAlreadyExistsError, e: + pass + + + + + +class Targets(Metadata): + """ + + Represent a Targets role object. Targets roles include the top-level role + 'targets.txt' and all delegated roles (e.g., 'targets/unclaimed/django'). + The expected operations of Targets metadata is included, such as adding + and removing repository target files, making and revoking delegations, and + listing the target files provided by it. + + Adding or removing a delegation causes the attributes of the Targets object + to be updated. That is, if the 'django' Targets object is delegated by + 'targets/unclaimed', a new attribute is added so that the following + code statement is supported: + repository.targets('unclaimed')('django').version = 2 + + Likewise, revoking a delegation causes removal of the delegation attribute. + + This Targets object sub-classes Metadata, so the expected + Metadata operations like adding/removing public keys, signatures, private + keys, and updating metadata attributes (e.g., version and expiration) is + supported. Since Targets is a top-level role and must exist, a default + Targets object (for 'targets.txt', not delegated roles) is instantiated when + a new Repository object is created. + + >>> + >>> + >>> + + + targets_directory: + The targets directory of the Repository object. + + rolename: + The rolename of this Targets object. + + roleinfo: + An already populated roleinfo object of 'rolename'. Conformant to + 'tuf.formats.ROLEDB_SCHEMA'. + + + tuf.FormatError, if the arguments are improperly formatted. + + + Modifies the roleinfo of the targets role in 'tuf.roledb'. + + + None. + """ + + def __init__(self, targets_directory, rolename, roleinfo=None): + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.PATH_SCHEMA.check_match(targets_directory) + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + + if roleinfo is not None: + tuf.formats.ROLEDB_SCHEMA.check_match(roleinfo) + + super(Targets, self).__init__() + self._targets_directory = targets_directory + self._rolename = rolename + self._target_files = [] + self._delegated_roles = {} + + # By default, Targets objects are set to expire 3 months from the current + # time. May be later modified. + expiration = tuf.formats.format_time(time.time()+TARGETS_EXPIRATION) + + # If 'roleinfo' is not provided, set an initial default. + if roleinfo is None: + roleinfo = {'keyids': [], 'signing_keyids': [], 'threshold': 1, + 'version': 0, 'compressions': [''], 'expires': expiration, + 'signatures': [], 'paths': [], 'path_hash_prefixes': [], + 'partial_loaded': False, 'delegations': {'keys': {}, + 'roles': []}} + + # Add the new role to the 'tuf.roledb'. + try: + tuf.roledb.add_role(self.rolename, roleinfo) + except tuf.RoleAlreadyExistsError, e: + pass + + + + def __call__(self, rolename): + """ + + Allow callable Targets object so that delegated roles may be referenced + by their string rolenames. Rolenames may include characters like '-' and + are not restricted to Python identifiers. + + + rolename: + The rolename of the delegated role. 'rolename' must be a role + previously delegated by this Targets role. + + + tuf.FormatError, if the arguments are improperly formatted. + + + Modifies the roleinfo of the targets role in 'tuf.roledb'. + + + None. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + + if rolename in self._delegated_roles: + return self._delegated_roles[rolename] + else: + message = repr(rolename)+' has not been delegated by '+repr(self.rolename) + raise tuf.UnknownRoleError(message) + + + + @property + def target_files(self): + """ + + A getter method that returns the target files added thus far to this + Targets object. + + >>> + >>> + >>> + + + None. + + + None. + + + None. + + + None. + """ + + target_files = tuf.roledb.get_roleinfo(self._rolename)['paths'] + + return target_files + + + + def add_target(self, filepath): + """ + + Add a filepath (must be under the repository's targets directory) to the + Targets object. + + This method does not actually create 'filepath' on the file + system. 'filepath' must already exist on the file system. + + >>> + >>> + >>> + + + filepath: + The path of the target file. It must be located in the repository's + targets directory. + + + tuf.FormatError, if 'filepath' is improperly formatted. + + tuf.Error, if 'filepath' is not found under the repository's targets + directory. + + + Adds 'filepath' to this role's list of targets. This role's + 'tuf.roledb.py' is also updated. + + + None. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + filepath = os.path.abspath(filepath) + + # Ensure 'filepath' is found under the repository's targets directory. + if not os.path.commonprefix([self._targets_directory, filepath]) == \ + self._targets_directory: + message = repr(filepath)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + # TODO: Ensure 'filepath' is an allowed target path according to the + # parent's delegation. + + # Add 'filepath' (i.e., relative to the targets directory) to the role's + # list of targets. + if os.path.isfile(filepath): + + # Update the role's 'tuf.roledb.py' entry and avoid duplicates. + targets_directory_length = len(self._targets_directory) + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + relative_path = filepath[targets_directory_length+1:] + if relative_path not in roleinfo['paths']: + roleinfo['paths'].append(relative_path) + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + else: + message = repr(filepath)+' is not a valid file.' + raise tuf.Error(message) + + + + def add_targets(self, list_of_targets): + """ + + Add a list of target filepaths (all relative to 'self.targets_directory'). + This method does not actually create files on the file system. The + list of target must already exist. + + >>> + >>> + >>> + + + list_of_targets: + A list of target filepaths that are added to the paths of this Targets + object. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.Error, if any of the paths listed in 'list_of_targets' is not found + under the repository's targets directory or is invalid. + + + This Targets' roleinfo is updated with the paths in 'list_of_targets'. + + + None. + """ + + # Does 'list_of_targets' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.RELPATHS_SCHEMA.check_match(list_of_targets) + + # TODO: Ensure list of targets allowed paths according to the parent's + # delegation. + + # Update the tuf.roledb entry. + targets_directory_length = len(self._targets_directory) + absolute_list_of_targets = [] + relative_list_of_targets = [] + + # Ensure the paths in 'list_of_targets' are valid and fall under the + # repository's targets directory. + for target in list_of_targets: + filepath = os.path.abspath(target) + + if not os.path.commonprefix([self._targets_directory, filepath]) == \ + self._targets_directory: + message = repr(filepath)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + if os.path.isfile(filepath): + absolute_list_of_targets.append(filepath) + relative_list_of_targets.append(filepath[targets_directory_length+1:]) + else: + message = repr(filepath)+' is not a valid file.' + raise tuf.Error(message) + + # Update this Targets 'tuf.roledb.py' entry. + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + for relative_target in relative_list_of_targets: + if relative_target not in roleinfo['paths']: + roleinfo['paths'].append(relative_target) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def remove_target(self, filepath): + """ + + Remove the target 'filepath' from this Targets' 'paths' field. 'filepath' + is relative to the targets directory. + + >>> + >>> + >>> + + + filepath: + The target to remove from this Targets object, relative to the + repository's targets directory. + + + tuf.FormatError, if 'filepath' is improperly formatted. + + tuf.Error, if 'filepath' is not under the repository's targets directory. + + + Modifies this Targets 'tuf.roledb.py' entry. + + + None. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.RELPATH_SCHEMA.check_match(filepath) + + filepath = os.path.abspath(filepath) + targets_directory_length = len(self._targets_directory) + + # Ensure 'filepath' is under the repository targets directory. + if not os.path.commonprefix([self._targets_directory, filepath]) == \ + self._targets_directory: + message = repr(filepath)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + # The relative filepath is listed in 'paths'. + relative_filepath = filepath[targets_directory_length+1:] + + # Remove 'relative_filepath', if found, and update this Targets roleinfo. + fileinfo = tuf.roledb.get_roleinfo(self.rolename) + if relative_filepath in fileinfo['paths']: + fileinfo['paths'].remove(relative_filepath) + + tuf.roledb.update_roleinfo(self.rolename, fileinfo) + + + + def clear_targets(self): + """ + + Remove all the target filepaths in the "paths" field of this Targets. + + >>> + >>> + >>> + + + None + + + None. + + + Modifies this Targets' 'tuf.roledb.py' entry. + + + None. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + roleinfo['paths'] = [] + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def get_delegated_rolenames(self): + """ + + Return all delegations of a role, including any made by child delegations. + If ['a/b/', 'a/b/c/', 'a/b/c/d'] have been delegated, + repository.a.get_delegated_rolenames() returns: + ['a/b', 'a/b/c', 'a/b/c/d']. + + + None. + + + None. + + + None. + + + A list of rolenames. + """ + + return tuf.roledb.get_delegated_rolenames(self.rolename) + + + + def delegate(self, rolename, public_keys, list_of_targets, + threshold=1, restricted_paths=None, path_hash_prefixes=None): + """ + + Create a new delegation, where 'rolename' is a child delegation of this + Targets object. The keys and roles database is updated, including the + delegations field of this Targets. The delegation of 'rolename' is added + and accessible (e.g., 'repository.targets(rolename). + + Actual metadata files are not updated, only when repository.write() or + repository.write_partial() is called. + + >>> + >>> + >>> + + + rolename: + The name of the delegated role, as in 'django' (i.e., not the full + rolename). + + public_keys: + A list of TUF keys objects in 'ANYKEYLIST_SCHEMA' format. The list + may contain any of the supported key types: RSAKEY_SCHEMA, + ED25519KEY_SCHEMA, etc. + + list_of_targets: + A list of target filepaths that are added to the paths of 'rolename'. + 'targets' is a list of target filepaths, and can be empty. + + threshold: + The threshold number of keys of 'rolename'. + + restricted_paths: + A list of restricted directory or file paths of 'rolename'. Any target + files added to 'rolename' must fall under one of the restricted paths. + + path_hash_prefixes: + A list of hash prefixes in PATH_HASH_PREFIXES_SCHEMA format, used in + hashed bin delegations. Targets may be located and stored in hashed + bins by calculating the target path's hash prefix. + + + tuf.FormatError, if any of the arguments are improperly formatted. + + tuf.Error, if the delegated role already exists or if any of the arguments + is an invalid path (i.e., not under the repository's targets directory). + + + A new Target object is created for 'rolename' that is accessible to the + caller (i.e., targets.unclaimed.). The 'tuf.keydb.py' and + 'tuf.roledb.py' stores are updated with 'public_keys'. + + + None. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + tuf.formats.ANYKEYLIST_SCHEMA.check_match(public_keys) + tuf.formats.RELPATHS_SCHEMA.check_match(list_of_targets) + tuf.formats.THRESHOLD_SCHEMA.check_match(threshold) + if restricted_paths is not None: + tuf.formats.RELPATHS_SCHEMA.check_match(restricted_paths) + if path_hash_prefixes is not None: + tuf.formats.PATH_HASH_PREFIXES_SCHEMA.check_match(path_hash_prefixes) + + # Check if 'rolename' is not already a delegation. 'tuf.roledb' expects the + # full rolename. + full_rolename = self._rolename+'/'+rolename + + if tuf.roledb.role_exists(full_rolename): + raise tuf.Error(repr(full_rolename)+' already delegated.') + + # Keep track of the valid keyids (added to the new Targets object) and their + # keydicts (added to this Targets delegations). + keyids = [] + keydict = {} + + # Add all the keys of 'public_keys' to tuf.keydb. + for key in public_keys: + + try: + tuf.keydb.add_key(key) + except tuf.KeyAlreadyExistsError, e: + pass + + keyid = key['keyid'] + key_metadata_format = tuf.keys.format_keyval_to_metadata(key['keytype'], + key['keyval']) + # Update 'keyids' and 'keydict'. + new_keydict = {keyid: key_metadata_format} + keydict.update(new_keydict) + keyids.append(keyid) + + # Ensure the paths of 'list_of_targets' all fall under the repository's + # targets. + relative_targetpaths = [] + targets_directory_length = len(self._targets_directory) + + for target in list_of_targets: + target = os.path.abspath(target) + if not os.path.commonprefix([self._targets_directory, target]) == \ + self._targets_directory: + message = repr(target)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + relative_targetpaths.append(target[targets_directory_length+1:]) + + # Ensure the paths of 'restricted_paths' all fall under the repository's + # targets. + relative_restricted_paths = [] + + if restricted_paths is not None: + for target in restricted_paths: + target = os.path.abspath(target) + if not os.path.commonprefix([self._targets_directory, target]) == \ + self._targets_directory: + message = repr(target)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + relative_restricted_paths.append(target[targets_directory_length+1:]) + + # Create a new Targets object for the 'rolename' delegation. An initial + # expiration is set (3 months from the current time). + expiration = tuf.formats.format_time(time.time()+TARGETS_EXPIRATION) + roleinfo = {'name': full_rolename, 'keyids': keyids, 'signing_keyids': [], + 'threshold': threshold, 'version': 0, 'compressions': [''], + 'expires': expiration, 'signatures': [], 'partial_loaded': False, + 'paths': relative_targetpaths, 'delegations': {'keys': {}, + 'roles': []}} + + # The new targets object is added as an attribute to this Targets object. + new_targets_object = Targets(self._targets_directory, full_rolename, + roleinfo) + + # Update the 'delegations' field of the current role. + current_roleinfo = tuf.roledb.get_roleinfo(self.rolename) + current_roleinfo['delegations']['keys'].update(keydict) + + # Update the roleinfo of this role. A ROLE_SCHEMA object requires only + # 'keyids', 'threshold', and 'paths'. + roleinfo = {'name': full_rolename, + 'keyids': roleinfo['keyids'], + 'threshold': roleinfo['threshold'], + 'paths': roleinfo['paths']} + if restricted_paths is not None: + roleinfo['paths'] = relative_restricted_paths + if path_hash_prefixes is not None: + roleinfo['path_hash_prefixes'] = path_hash_prefixes + + current_roleinfo['delegations']['roles'].append(roleinfo) + tuf.roledb.update_roleinfo(self.rolename, current_roleinfo) + + # Update the public keys of 'new_targets_object'. + for key in public_keys: + new_targets_object.add_key(key) + + # Add the new delegation to this Targets object. For example, 'django' is + # added to 'repository.targets' (i.e., repository.targets('django'). + self._delegated_roles[rolename] = new_targets_object + + + + def revoke(self, rolename): + """ + + Revoke this Targets' 'rolename' delegation. Its 'rolename' attribute is + deleted, including the entries in its 'delegations' field and in + 'tuf.roledb'. + + Actual metadata files are not updated, only when repository.write() or + repository.write_partial() is called. + + >>> + >>> + >>> + + + rolename: + The rolename (e.g., 'Django' in 'targets/unclaimed/Django') of + the child delegation the parent role (this role) wants to revoke. + + + tuf.FormatError, if 'rolename' is improperly formatted. + + + The delegations dictionary of 'rolename' is modified, and its 'tuf.roledb' + entry is updated. This Targets' 'rolename' delegation attribute is also + deleted. + + + None. + """ + + # Does 'rolename' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + + # Remove 'rolename' from this Target's delegations dict. + # The child delegation's full rolename is required to locate in the parent's + # delegations list. + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + full_rolename = self.rolename+'/'+rolename + + for role in roleinfo['delegations']['roles']: + if role['name'] == full_rolename: + roleinfo['delegations']['roles'].remove(role) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + # Remove 'rolename' from 'tuf.roledb.py'. The delegations of 'rolename' are + # also removed. + tuf.roledb.remove_role(full_rolename) + + # Remove the rolename delegation from the current role. For example, the + # 'django' role is removed from 'repository.targets('unclaimed')('django'). + del self._delegated_roles[rolename] + + + + def delegate_hashed_bins(self, list_of_targets, keys_of_hashed_bins, + number_of_bins=1024): + """ + + Split the large number of target files of 'list_of_targets' into + multiple delegated roles (hashed bins). The size of all the delegated + roles will be nearly equal. The updater client will use "lazy bin walk" + to find a target file's hashed bin destination. The parent role lists + the hashed bins as either a direct delegation, or as a path hash prefix + of another hashed bin. See the following link for more information: + http://www.python.org/dev/peps/pep-0458/#metadata-scalability + + >>> + >>> + >>> + + + list_of_targets: + The target filepaths of the targets that should be stored in the hashed + bins (i.e., delegated roles). + + keys_of_hashed_bins: + The public keys of the delegated roles. + + number_of_bins: + The number of delegated roles listed in the parent role's + 'delegations' field. Must be a multiple of 16. Each bin may contain + multiple roles. + + + tuf.FormatError, if the arguments are improperly formatted, + 'number_of_bins' is not a multiple of 16, or one of the targets + in 'list_of_targets' is not located under the repository's targets + directory. + + + Delegates multiple target roles from the current parent role. Others + may be generated/added as a role and only linked with the parent. + + + None. + """ + + # Does 'rolename' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATHS_SCHEMA.check_match(list_of_targets) + tuf.formats.ANYKEYLIST_SCHEMA.check_match(keys_of_hashed_bins) + tuf.formats.NUMBINS_SCHEMA.check_match(number_of_bins) + + # Strip the '0x' from the Python hex representation. + prefix_length = len(hex(number_of_bins - 1)[2:]) + max_number_of_bins = 16 ** prefix_length + + # For simplicity, ensure that we can evenly distribute 'max_number_of_bins' + # over 'number_of_bins'. + if max_number_of_bins % number_of_bins != 0: + message = 'The number of bins argument must be a multiple of 16.' + raise tuf.FormatError(message) + + logger.info('There are '+len(list_of_targets)+' total targets.') + + # Store the target paths that fall into each bin. + target_paths_in_bin = {} + for bin_index in xrange(max_number_of_bins): + target_paths_in_bin[bin_index] = [] + + # Assign every path to its bin. Ensure every target is located under the + # repository's targets directory. + for target_path in list_of_targets: + if not target_path.startswith(self._targets_directory+'/'): + message = 'A path in the list of targets arguments is not '+\ + 'under the repository\'s targets directory: '+repr(target_path) + raise tuf.FormatError(message) + + # Determine the hash prefix of 'target_path' by computing the digest of + # its path relative to the targets directory. Example: + # '{repository_root}/targets/file1.txt' -> 'file1.txt'. + relative_path = target_path[len(self._targets_directory)+1:] + digest_object = tuf.hash.digest(algorithm=HASH_FUNCTION) + digest_object.update(relative_path) + relative_path_hash = digest.hexdigest() + relative_path_hash_prefix = relative_path_hash[:prefix_length] + + # 'target_paths_in_bin' store bin indices in base-10, so convert the + # 'relative_path_hash_prefix' base-16 (hex) number to a base-10 (dec) + # number. + bin_index = int(relative_path_hash_prefix, 16) + + # Add the 'target_path' (absolute) to the bin. + target_paths_in_bin[bin_index] = \ + target_paths_in_bin[bin_index].append(target_path) + + # Calculate the path hash prefixes of each bin_offset stored in the parent + # role. For example: 'targets/unclaimed/004' may list the path hash + # prefixes "000", "001", "002", "003" in the delegations dict of + # 'targets/unclaimed'. + bin_offset = max_number_of_bins // number_of_bins + + # The parent roles will list bin roles starting from "0" to + # 'max_number_of_bins' in 'bin_offset' increments. The skipped bin roles + # are listed in 'path_hash_prefixes' of 'outer_bin_index. + for outer_bin_index in xrange(0, max_number_of_bins, bin_offset): + # The bin index in hex padded from the left with zeroes for up to the + # 'prefix_lengthn'. + bin_rolename = hex(outer_bin_index)[2:].zfill(prefix_length) + + # The hash prefixes of the skipped bin roles, or the roles not directly + # delegated from the parent role. + path_hash_prefixes = [] + + for inner_bin_index in xrange(outer_bin_index, outer_bin_index+bin_offset): + # 'inner_bin_rolename' in padded hex. For example, "00b". + inner_bin_rolename = hex(inner_bin_index)[2:].zfill(prefix_length) + path_hash_prefixes.append(inner_bin_rolename) + + # Delegate from the "unclaimed" targets role to each 'bin_rolename' + # (i.e., outer_bin_index). + bin_rolename_targets = target_paths_in_bin[outer_bin_index] + self.delegate(bin_rolename, keys_of_hashed_bins, + list_of_targets=bin_rolename_targets, + path_hash_prefixes=path_hash_prefixes) + + message = 'Delegated from '+repr(self.rolename)+' to '+repr(binned_rolename) + logger.debug(message) + + + + @property + def delegations(self): + """ + + A getter method that returns the delegations made by this Targets role. + + >>> + >>> + >>> + + + None. + + + tuf.UnknownRoleError, if this Targets' rolename does not exist in + 'tuf.roledb'. + + + None. + + + A dictionary containing the rolenames (as dict keys) and role Targets + objects of this Targets' delegations. + Example: {'targets/unclaimed-role/django': Targets(), ...} + """ + + return self._delegated_roles + + + + + +def _generate_and_write_metadata(rolename, metadata_filename, write_partial, + targets_directory, metadata_directory, + consistent_snapshots, filenames=None): + """ + Non-public function that can generate and write the metadata of the specified + top-level 'rolename'. It also increments version numbers if: + + 1. write_partial==True and the metadata is the first to be written. + + 2. write_partial=False (i.e., write()), the metadata was not loaded as + partially written, and a write_partial is not needed. + """ + + metadata = None + + # Retrieve the roleinfo of 'rolename' to extract the needed metadata + # attributes, such as version number, expiration, etc. + roleinfo = tuf.roledb.get_roleinfo(rolename) + release_compressions = tuf.roledb.get_roleinfo('release')['compressions'] + + # Generate the appropriate role metadata for 'rolename'. + if rolename == 'root': + metadata = generate_root_metadata(roleinfo['version'], + roleinfo['expires'], consistent_snapshots) + elif rolename.startswith('targets'): + metadata = generate_targets_metadata(targets_directory, + roleinfo['paths'], + roleinfo['version'], + roleinfo['expires'], + roleinfo['delegations'], + consistent_snapshots) + elif rolename == 'release': + root_filename = filenames['root'] + targets_filename = filenames['targets'] + metadata = generate_release_metadata(metadata_directory, + roleinfo['version'], + roleinfo['expires'], root_filename, + targets_filename, + consistent_snapshots ) + elif rolename == 'timestamp': + release_filename = filenames['release'] + metadata = generate_timestamp_metadata(release_filename, + roleinfo['version'], + roleinfo['expires'], + release_compressions) + + signable = sign_metadata(metadata, roleinfo['signing_keyids'], + metadata_filename) + + # Check if the version number of 'rolename' may be automatically incremented, + # depending on whether if partial metadata is loaded or if the metadata is + # written with write() / write_partial(). + # Increment the version number if this is the first partial write. + if write_partial: + temp_signable = sign_metadata(metadata, [], metadata_filename) + temp_signable['signatures'].extend(roleinfo['signatures']) + status = tuf.sig.get_signature_status(temp_signable, rolename) + if len(status['good_sigs']) == 0: + metadata['version'] = metadata['version'] + 1 + signable = sign_metadata(metadata, roleinfo['signing_keyids'], + metadata_filename) + # non-partial write() + else: + if tuf.sig.verify(signable, rolename) and not roleinfo['partial_loaded']: + metadata['version'] = metadata['version'] + 1 + signable = sign_metadata(metadata, roleinfo['signing_keyids'], + metadata_filename) + + # Write the metadata to file if contains a threshold of signatures. + signable['signatures'].extend(roleinfo['signatures']) + + if tuf.sig.verify(signable, rolename) or write_partial: + _remove_invalid_and_duplicate_signatures(signable) + compressions = roleinfo['compressions'] + filename = write_metadata_file(signable, metadata_filename, compressions, + consistent_snapshots) + + # The root and timestamp files should also be written without a digest if + # 'consistent_snaptshots' is True. Client may request a timestamp and root + # file without knowing its digest and file size. + if rolename == 'root' or rolename == 'timestamp': + write_metadata_file(signable, metadata_filename, compressions, + consistent_snapshots=False) + + return signable, filename + + # 'signable' contains an invalid threshold of signatures. + else: + message = 'Not enough signatures for '+repr(metadata_filename) + raise tuf.Error(message, signable) + + + + + +def _print_status(rolename, signable): + """ + Non-public function prints the number of (good/threshold) signatures of + 'rolename'. + """ + + status = tuf.sig.get_signature_status(signable, rolename) + + message = repr(rolename)+' role contains '+ \ + repr(len(status['good_sigs']))+' / '+ \ + repr(status['threshold'])+' signatures.' + print(message) + + + + + +def _prompt(message, result_type=str): + """ + Non-public function that prompts the user for input by printing 'message', + converting the input to 'result_type', and returning the value to the + caller. + """ + + return result_type(raw_input(message)) + + + + + +def _get_password(prompt='Password: ', confirm=False): + """ + Non-public function that returns the password entered by the user. If + 'confirm' is True, the user is asked to enter the previously entered + password once again. If they match, the password is returned to the caller. + """ + + while True: + # getpass() prompts the user for a password without echoing + # the user input. + password = getpass.getpass(prompt, sys.stderr) + if not confirm: + return password + password2 = getpass.getpass('Confirm: ', sys.stderr) + if password == password2: + return password + else: + print('Mismatch; try again.') + + + + + +def _check_if_partial_loaded(rolename, signable, roleinfo): + """ + Non-public function that determines whether 'rolename' is loaded with + at least 1 good signatures, but an insufficient threshold (which means + 'rolename' was written to disk with repository.write_partial(). If 'rolename' + is found to be partially loaded, mark it as partially loaded in its + 'tuf.roledb' roleinfo. This function exists to assist in deciding whether + a role's version number should be incremented when write() or write_parital() + is called. + """ + + # The signature status lists the number of good signatures, including + # bad, untrusted, unknown, etc. + status = tuf.sig.get_signature_status(signable, rolename) + + if len(status['good_sigs']) < status['threshold'] and \ + len(status['good_sigs']) >= 1: + roleinfo['partial_loaded'] = True + + + + + +def _check_directory(directory): + """ + + Non-public function that ensures 'directory' is valid and it exists. This + is not a security check, but a way for the caller to determine the cause of + an invalid directory provided by the user. If the directory argument is + valid, it is returned normalized and as an absolute path. + + + directory: + The directory to check. + + + tuf.Error, if 'directory' could not be validated. + + tuf.FormatError, if 'directory' is not properly formatted. + + + None. + + + The normalized absolutized path of 'directory'. + """ + + # Does 'directory' have the correct format? + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(directory) + + # Check if the directory exists. + if not os.path.isdir(directory): + raise tuf.Error(repr(directory)+' directory does not exist.') + + directory = os.path.abspath(directory) + + return directory + + + + + +def _check_role_keys(rolename): + """ + Non-public function that verifies the public and signing keys of 'rolename'. + If either contain an invalid threshold number of keys, raise an exception. + 'rolename' is the full rolename (e.g., 'targets/unclaimed/django'). + """ + + # Extract the total number of public and private keys of 'rolename' from its + # roleinfo in 'tuf.roledb'. + roleinfo = tuf.roledb.get_roleinfo(rolename) + total_keyids = len(roleinfo['keyids']) + threshold = roleinfo['threshold'] + total_signatures = len(roleinfo['signatures']) + total_signing_keys = len(roleinfo['signing_keyids']) + + # Raise an exception for an invalid threshold of public keys. + if total_keyids < threshold: + message = repr(rolename)+' role contains '+repr(total_keyids)+' / '+ \ + repr(threshold)+' public keys.' + raise tuf.InsufficientKeysError(message) + + # Raise an exception for an invalid threshold of signing keys. + if total_signatures == 0 and total_signing_keys < threshold: + message = repr(rolename)+' role contains '+repr(total_signing_keys)+' / '+ \ + repr(threshold)+' signing keys.' + raise tuf.InsufficientKeysError(message) + + + + + +def _remove_invalid_and_duplicate_signatures(signable): + """ + Non-public function that removes invalid signatures from 'signable'. + 'signable' may contain signatures (invalid) from previous versions + of the metadata that were loaded with load_repository(). Invalid, or + duplicate signatures are removed from 'signable'. + """ + + # Store the keyids of valid signatures. 'signature_keyids' is checked + # for duplicates rather than comparing signature objects because PSS may + # generate duplicate valid signatures of the same data, yet contain different + # signatures. + signature_keyids = [] + + for signature in signable['signatures']: + signed = signable['signed'] + keyid = signature['keyid'] + key = None + + # Remove 'signature' from 'signable' if the listed keyid does not exist + # in 'tuf.keydb'. + try: + key = tuf.keydb.get_key(keyid) + except tuf.UnknownKeyError, e: + signable['signatures'].remove(signature) + + # Remove 'signature' from 'signable' if it is an invalid signature. + if not tuf.keys.verify_signature(key, signature, signed): + signable['signatures'].remove(signature) + + # Although valid, it may still need removal if it is a duplicate. Check + # the keyid, rather than the signature, to remove duplicate PSS signatures. + # PSS may generate multiple different signatures for the same keyid. + else: + if keyid in signature_keyids: + signable['signatures'].remove(signature) + + # 'keyid' is valid and not a duplicate, so add it to 'signature_keyids'. + else: + signature_keyids.append(keyid) + + + + + +def _delete_obsolete_metadata(metadata_directory, release_metadata, + consistent_snapshots): + """ + Non-public function that deletes metadata files marked as removed by + libtuf.py. Metadata files marked as removed are not actually deleted + until this function is called. + """ + + # Walk the repository's metadata 'targets' sub-directory, where all the + # metadata of delegated roles is stored. + targets_metadata = os.path.join(metadata_directory, 'targets') + + # The 'targets.txt' metadata is not visited, only its child delegations. + # The 'targets/unclaimed/django.txt' role would be located in the + # '{repository_directory}/metadata/targets/unclaimed/' directory. + if os.path.exists(targets_metadata) and os.path.isdir(targets_metadata): + for directory_path, junk_directories, files in os.walk(targets_metadata): + + # 'files' here is a list of target file names. + for basename in files: + metadata_path = os.path.join(directory_path, basename) + # Strip the metadata dirname and the leading path separator. + # '{repository_directory}/metadata/targets/unclaimed/django.txt' --> + # 'targets/unclaimed/django.txt' + metadata_name = \ + metadata_path[len(metadata_directory):].lstrip(os.path.sep) + + # Strip the digest if 'consistent_snapshots' is True. + # Example: 'targets/unclaimed/13df98ab0.django.txt' --> + # 'targets/unclaimed/django.txt' + metadata_name, embeded_digest = \ + _strip_consistent_snapshots_digest(metadata_name, consistent_snapshots) + + # Strip filename extensions. The role database does not include the + # metadata extension. + for metadata_extension in METADATA_EXTENSIONS: + if metadata_name.endswith(metadata_extension): + metadata_name_without_extension = \ + metadata_name[:-len(metadata_extension)] + + # Delete the metadata file if it does not exist in 'tuf.roledb'. + # libtuf.py might have marked 'metadata_name' as removed, but its + # metadata file is not actually deleted yet. Do it now. + if not tuf.roledb.role_exists(metadata_name_without_extension): + logger.info('Removing outdated metadata: ' + repr(metadata_path)) + os.remove(metadata_path) + + # Delete outdated consistent snapshots. release metadata includes + # the file extension of roles. + if consistent_snapshots: + #metadata_name_extension = metadata_name + METADATA_EXTENSION + file_hashes = release_metadata['meta'][metadata_name] \ + ['hashes'].values() + if embeded_digest not in file_hashes: + logger.info('Removing outdated metadata: ' + repr(metadata_path)) + os.remove(metadata_path) + + + + + +def _get_written_metadata_and_digests(metadata_signable): + """ + Non-public function that returns the actual content of written metadata and + its digest. + """ + + written_metadata_content = unicode(json.dumps(metadata_signable, indent=1, + sort_keys=True)) + written_metadata_digests = {} + + for hash_algorithm in tuf.conf.REPOSITORY_HASH_ALGORITHMS: + digest_object = tuf.hash.digest(hash_algorithm) + digest_object.update(written_metadata_content) + written_metadata_digests.update({hash_algorithm: digest_object.hexdigest()}) + + return written_metadata_content, written_metadata_digests + + + + + +def _strip_consistent_snapshots_digest(metadata_filename, consistent_snapshots): + """ + Strip from 'metadata_filename' any digest data (in the expected + '{dirname}/digest.filename' format) that it may contain, and return it. + """ + + embeded_digest = '' + + # Strip the digest if 'consistent_snapshots' is True. + # Example: 'targets/unclaimed/13df98ab0.django.txt' --> + # 'targets/unclaimed/django.txt' + if consistent_snapshots: + dirname, basename = os.path.split(metadata_filename) + embeded_digest = basename[:basename.find('.')] + basename = basename[basename.find('.')+1:] + metadata_filename = os.path.join(dirname, basename) + + + return metadata_filename, embeded_digest + + + + + + +def create_new_repository(repository_directory): + """ + + Create a new repository, instantiate barebones metadata for the top-level + roles, and return a Repository object. On disk, create_new_repository() + only creates the directories needed to hold the metadata and targets files. + The repository object returned may be modified to update the newly created + repository. The methods of the returned object may be called to create + actual repository files (e.g., repository.write()). + + + repository_directory: + The directory that will eventually hold the metadata and target files of + the TUF repository. + + + tuf.FormatError, if the arguments are improperly formatted. + + + The 'repository_directory' is created if it does not exist, including its + metadata and targets sub-directories. + + + A 'tuf.libtuf.Repository' object. + """ + + # Does 'repository_directory' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(repository_directory) + + # Set the repository, metadata, and targets directories. These directories + # are created if they do not exist. + repository_directory = os.path.abspath(repository_directory) + metadata_directory = None + targets_directory = None + + # Try to create 'repository_directory' if it does not exist. + try: + message = 'Creating '+repr(repository_directory) + logger.info(message) + os.makedirs(repository_directory) + + # 'OSError' raised if the leaf directory already exists or cannot be created. + # Check for case where 'repository_directory' has already been created. + except OSError, e: + if e.errno == errno.EEXIST: + pass + else: + raise + + # Set the metadata and targets directories. The metadata directory is a + # staged one so that the "live" repository is not affected. The + # staged metadata changes may be moved over to "live" after all updated + # have been completed. + metadata_directory = \ + os.path.join(repository_directory, METADATA_STAGED_DIRECTORY_NAME) + targets_directory = \ + os.path.join(repository_directory, TARGETS_DIRECTORY_NAME) + + # Try to create the metadata directory that will hold all of the metadata + # files, such as 'root.txt' and 'release.txt'. + try: + message = 'Creating '+repr(metadata_directory) + logger.info(message) + os.mkdir(metadata_directory) + + # 'OSError' raised if the leaf directory already exists or cannot be created. + except OSError, e: + if e.errno == errno.EEXIST: + pass + else: + raise + + # Try to create the targets directory that will hold all of the target files. + try: + message = 'Creating '+repr(targets_directory) + logger.info(message) + os.mkdir(targets_directory) + except OSError, e: + if e.errno == errno.EEXIST: + pass + else: + raise + + # Create the bare bones repository object, where only the top-level roles + # have been set and contain default values (e.g., Root roles has a threshold + # of 1, expires 1 year into the future, etc.) + repository = Repository(repository_directory, metadata_directory, + targets_directory) + + return repository + + + +def load_repository(repository_directory): + """ + + Return a repository object containing the contents of metadata files loaded + from the repository. + + + repository_directory: + + + tuf.FormatError, if 'repository_directory' or any of the metadata files + are improperly formatted. Also raised if, at a minimum, the Root role + cannot be found. + + + All the metadata files found in the repository are loaded and their contents + stored in a libtuf.Repository object. + + + libtuf.Repository object. + """ + + # Does 'repository_directory' have the correct format? + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(repository_directory) + + # Load top-level metadata. + repository_directory = os.path.abspath(repository_directory) + metadata_directory = os.path.join(repository_directory, + METADATA_STAGED_DIRECTORY_NAME) + targets_directory = os.path.join(repository_directory, + TARGETS_DIRECTORY_NAME) + + # The Repository() object loaded (i.e., containing all the metadata roles + # found) and returned. + repository = Repository(repository_directory, metadata_directory, + targets_directory) + + filenames = get_metadata_filenames(metadata_directory) + + # The Root file is always available without a consistent snapshots digest + # attached to the filename. Store the 'consistent_snapshots' value read the + # loaded Root file so that other metadata files may be located. + # 'consistent_snapshots' value. + consistent_snapshots = False + + # Load the metadata of the top-level roles (i.e., Root, Timestamp, Targets, + # and Release). + repository, consistent_snapshots = _load_top_level_metadata(repository, + filenames) + + # Load delegated targets metadata. + # Walk the 'targets/' directory and generate the fileinfo of all the files + # listed. This information is stored in the 'meta' field of the release + # metadata object. + targets_objects = {} + loaded_metadata = [] + targets_objects['targets'] = repository.targets + targets_metadata_directory = os.path.join(metadata_directory, + TARGETS_DIRECTORY_NAME) + if os.path.exists(targets_metadata_directory) and \ + os.path.isdir(targets_metadata_directory): + for root, directories, files in os.walk(targets_metadata_directory): + + # 'files' here is a list of target file names. + for basename in files: + metadata_path = os.path.join(root, basename) + metadata_name = \ + metadata_path[len(metadata_directory):].lstrip(os.path.sep) + + # Strip the digest if 'consistent_snapshots' is True. + # Example: 'targets/unclaimed/13df98ab0.django.txt' --> + # 'targets/unclaimed/django.txt' + metadata_name, digest_junk = \ + _strip_consistent_snapshots_digest(metadata_name, consistent_snapshots) + + if metadata_name.endswith(METADATA_EXTENSION): + extension_length = len(METADATA_EXTENSION) + metadata_name = metadata_name[:-extension_length] + else: + continue + + # Keep a store metadata previously loaded metadata to prevent + # re-loading duplicate versions. Duplicate versions may occur with + # consistent_snapshots, where the same metadata may be available in + # multiples files (the different hash is included in each filename. + if metadata_name in loaded_metadata: + continue + + signable = None + try: + signable = tuf.util.load_json_file(metadata_path) + except (ValueError, IOError), e: + continue + + metadata_object = signable['signed'] + + roleinfo = tuf.roledb.get_roleinfo(metadata_name) + roleinfo['signatures'].extend(signable['signatures']) + roleinfo['version'] = metadata_object['version'] + roleinfo['expires'] = metadata_object['expires'] + roleinfo['paths'] = metadata_object['targets'].keys() + roleinfo['delegations'] = metadata_object['delegations'] + + if os.path.exists(metadata_path+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded(metadata_name, signable, roleinfo) + tuf.roledb.update_roleinfo(metadata_name, roleinfo) + loaded_metadata.append(metadata_name) + + new_targets_object = Targets(targets_directory, metadata_name, roleinfo) + targets_object = \ + targets_objects[tuf.roledb.get_parent_rolename(metadata_name)] + targets_objects[metadata_name] = new_targets_object + + targets_object._delegated_roles[(os.path.basename(metadata_name))] = \ + new_targets_object + + # Add the keys specified in the delegations field of the Targets role. + for key_metadata in metadata_object['delegations']['keys'].values(): + key_object = tuf.keys.format_metadata_to_key(key_metadata) + try: + tuf.keydb.add_key(key_object) + except tuf.KeyAlreadyExistsError, e: + pass + + for role in metadata_object['delegations']['roles']: + rolename = role['name'] + roleinfo = {'name': role['name'], 'keyids': role['keyids'], + 'threshold': role['threshold'], + 'compressions': [''], 'signing_keyids': [], + 'signatures': [], + 'partial_loaded': False, + 'delegations': {'keys': {}, + 'roles': []}} + tuf.roledb.add_role(rolename, roleinfo) + + return repository + + + + + +def _load_top_level_metadata(repository, top_level_filenames): + """ + Load the metadata of the Root, Timestamp, Targets, and Release roles. + At a minimum, the Root role must exist and successfully loaded. + """ + + root_filename = top_level_filenames[ROOT_FILENAME] + targets_filename = top_level_filenames[TARGETS_FILENAME] + release_filename = top_level_filenames[RELEASE_FILENAME] + timestamp_filename = top_level_filenames[TIMESTAMP_FILENAME] + + root_metadata = None + targets_metadata = None + release_metadata = None + timestamp_metadata = None + + # Load ROOT.txt. A Root role file without a digest is always written. + if os.path.exists(root_filename): + # Initialize the key and role metadata of the top-level roles. + signable = tuf.util.load_json_file(root_filename) + tuf.formats.check_signable_object_format(signable) + root_metadata = signable['signed'] + tuf.keydb.create_keydb_from_root_metadata(root_metadata) + tuf.roledb.create_roledb_from_root_metadata(root_metadata) + + # Load Root's roleinfo and update 'tuf.roledb'. + roleinfo = tuf.roledb.get_roleinfo('root') + roleinfo['signatures'] = [] + for signature in signable['signatures']: + if signature not in roleinfo['signatures']: + roleinfo['signatures'].append(signature) + + if os.path.exists(root_filename+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded('root', signable, roleinfo) + tuf.roledb.update_roleinfo('root', roleinfo) + + # Ensure the 'consistent_snapshots' field is extracted. + consistent_snapshots = root_metadata['consistent_snapshots'] + + else: + message = 'Cannot load the required root file: '+repr(root_filename) + raise tuf.RepositoryError(message) + + # Load TIMESTAMP.txt. A Timestamp role file without a digest is always + # written. + if os.path.exists(timestamp_filename): + signable = tuf.util.load_json_file(timestamp_filename) + timestamp_metadata = signable['signed'] + for signature in signable['signatures']: + repository.timestamp.add_signature(signature) + + # Load Timestamp's roleinfo and update 'tuf.roledb'. + roleinfo = tuf.roledb.get_roleinfo('timestamp') + roleinfo['expires'] = timestamp_metadata['expires'] + roleinfo['version'] = timestamp_metadata['version'] + if os.path.exists(timestamp_filename+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded('timestamp', signable, roleinfo) + tuf.roledb.update_roleinfo('timestamp', roleinfo) + + else: + pass + + # Load RELEASE.txt. A consistent snapshot of Release must be calculated + # if 'consistent_snapshots' is True. + if consistent_snapshots: + release_hashes = timestamp_metadata['meta'][RELEASE_FILENAME]['hashes'] + release_digest = random.choice(release_hashes.values()) + dirname, basename = os.path.split(release_filename) + release_filename = os.path.join(dirname, release_digest + '.' + basename) + + if os.path.exists(release_filename): + signable = tuf.util.load_json_file(release_filename) + tuf.formats.check_signable_object_format(signable) + release_metadata = signable['signed'] + for signature in signable['signatures']: + repository.release.add_signature(signature) + + # Load Release's roleinfo and update 'tuf.roledb'. + roleinfo = tuf.roledb.get_roleinfo('release') + roleinfo['expires'] = release_metadata['expires'] + roleinfo['version'] = release_metadata['version'] + if os.path.exists(release_filename+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded('release', signable, roleinfo) + tuf.roledb.update_roleinfo('release', roleinfo) + + else: + pass + + # Load TARGETS.txt. A consistent snapshot of Targets must be calculated if + # 'consistent_snapshots' is True. + if consistent_snapshots: + targets_hashes = release_metadata['meta'][TARGETS_FILENAME]['hashes'] + targets_digest = random.choice(targets_hashes.values()) + dirname, basename = os.path.split(targets_filename) + targets_filename = os.path.join(dirname, targets_digest + '.' + basename) + + if os.path.exists(targets_filename): + signable = tuf.util.load_json_file(targets_filename) + tuf.formats.check_signable_object_format(signable) + targets_metadata = signable['signed'] + + for signature in signable['signatures']: + repository.targets.add_signature(signature) + + # Update 'targets.txt' in 'tuf.roledb.py' + roleinfo = tuf.roledb.get_roleinfo('targets') + roleinfo['paths'] = targets_metadata['targets'].keys() + roleinfo['version'] = targets_metadata['version'] + roleinfo['expires'] = targets_metadata['expires'] + roleinfo['delegations'] = targets_metadata['delegations'] + if os.path.exists(targets_filename+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded('targets', signable, roleinfo) + tuf.roledb.update_roleinfo('targets', roleinfo) + + # Add the keys specified in the delegations field of the Targets role. + # TODO: Delegated role's are only missing the threshold value, which the + # parent role sets. Remember to request threshold value from parent role. + for key_metadata in targets_metadata['delegations']['keys'].values(): + key_object = tuf.keys.format_metadata_to_key(key_metadata) + tuf.keydb.add_key(key_object) + + for role in targets_metadata['delegations']['roles']: + rolename = role['name'] + roleinfo = {'name': role['name'], 'keyids': role['keyids'], + 'threshold': role['threshold'], 'compressions': [''], + 'signing_keyids': [], 'signatures': [], + 'delegations': {'keys': {}, + 'roles': []}} + tuf.roledb.add_role(rolename, roleinfo) + + else: + pass + + return repository, consistent_snapshots + + + + + +def generate_and_write_rsa_keypair(filepath, bits=DEFAULT_RSA_KEY_BITS, + password=None): + """ + + Generate an RSA key file, create an encrypted PEM string (using 'password' + as the pass phrase), and store it in 'filepath'. The public key portion of + the generated RSA key is stored in <'filepath'>.pub. Which cryptography + library performs the cryptographic decryption is determined by the string + set in 'tuf.conf.RSA_CRYPTO_LIBRARY'. PyCrypto currently supported. The + PEM private key is encrypted with 3DES and CBC the mode of operation. The + password is strengthened with PBKDF1-MD5. + + + filepath: + The public and private key files are saved to .pub, , + respectively. + + bits: + The number of bits of the generated RSA key. + + password: + The password used to encrypt 'filepath'. + + + tuf.FormatError, if the arguments are improperly formatted. + + + Writes key files to '' and '.pub'. + + + None. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of + # objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # Does 'bits' have the correct format? + tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) + + # If the caller does not provide a password argument, prompt for one. + if password is None: + message = 'Enter a password for the RSA key file: ' + password = _get_password(message, confirm=True) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Generate public and private RSA keys, encrypted the private portion + # and store them in PEM format. + rsa_key = tuf.keys.generate_rsa_key(bits) + public = rsa_key['keyval']['public'] + private = rsa_key['keyval']['private'] + encrypted_pem = tuf.keys.create_rsa_encrypted_pem(private, password) + + # Write public key (i.e., 'public', which is in PEM format) to + # '.pub'. If the parent directory of filepath does not exist, + # create it (and all its parent directories, if necessary). + tuf.util.ensure_parent_dir(filepath) + + # Create a tempororary file, write the contents of the public key, and move + # to final destination. + file_object = tuf.util.TempFile() + file_object.write(public) + + # The temporary file is closed after the final move. + file_object.move(filepath+'.pub') + + # Write the private key in encrypted PEM format to ''. + # Unlike the public key file, the private key does not have a file + # extension. + file_object = tuf.util.TempFile() + file_object.write(encrypted_pem) + file_object.move(filepath) + + + + + +def import_rsa_privatekey_from_file(filepath, password=None): + """ + + Import the encrypted PEM file in 'filepath', decrypt it, and return the key + object in 'tuf.formats.RSAKEY_SCHEMA' format. + + Which cryptography library performs the cryptographic decryption is + determined by the string set in 'tuf.conf.RSA_CRYPTO_LIBRARY'. PyCrypto + currently supported. + + The PEM private key is encrypted with 3DES and CBC the mode of operation. + The password is strengthened with PBKDF1-MD5. + + + filepath: + file, an RSA encrypted PEM file. Unlike the public RSA PEM + key file, 'filepath' does not have an extension. + + password: + The passphrase to decrypt 'filepath'. + + + tuf.FormatError, if the arguments are improperly formatted. + + + The contents of 'filepath' is read, decrypted, and the key stored. + + + An RSA key object, conformant to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # If the caller does not provide a password argument, prompt for one. + if password is None: + message = 'Enter a password for the encrypted RSA key file: ' + password = _get_password(message, confirm=True) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + encrypted_pem = None + + # Read the contents of 'filepath' that should be an encrypted PEM. + with open(filepath, 'rb') as file_object: + encrypted_pem = file_object.read() + + # Convert 'encrypted_pem' to 'tuf.formats.RSAKEY_SCHEMA' format. + rsa_key = tuf.keys.import_rsakey_from_encrypted_pem(encrypted_pem, password) + + return rsa_key + + + + + +def import_rsa_publickey_from_file(filepath): + """ + + Import the RSA key stored in 'filepath'. The key object returned is a TUF + key, specifically 'tuf.formats.RSAKEY_SCHEMA'. If the RSA PEM in 'filepath' + contains a private key, it is discarded. + + Which cryptography library performs the cryptographic decryption is + determined by the string set in 'tuf.conf.RSA_CRYPTO_LIBRARY'. PyCrypto + currently supported. If the RSA PEM in 'filepath' contains a private key, + it is discarded. + + + filepath: + .pub file, an RSA PEM file. + + + tuf.FormatError, if 'filepath' is improperly formatted. + + + 'filepath' is read and its contents extracted. + + + An RSA key object conformant to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # Read the contents of the key file that should be in PEM format and contains + # the public portion of the RSA key. + with open(filepath, 'rb') as file_object: + rsa_pubkey_pem = file_object.read() + + # Convert 'rsa_pubkey_pem' in 'tuf.formats.RSAKEY_SCHEMA' format. + rsakey_dict = tuf.keys.format_rsakey_from_pem(rsa_pubkey_pem) + + return rsakey_dict + + + + + +def generate_and_write_ed25519_keypair(filepath, password=None): + """ + + Generate an ED25519 key file, create an encrypted TUF key (using 'password' + as the pass phrase), and store it in 'filepath'. The public key portion of + the generated ED25519 key is stored in <'filepath'>.pub. Which cryptography + library performs the cryptographic decryption is determined by the string + set in 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + + PyCrypto currently supported. The ED25519 private key is encrypted with + AES-256 and CTR the mode of operation. The password is strengthened with + PBKDF2-HMAC-SHA256. + + + filepath: + The public and private key files are saved to .pub and + , respectively. + + password: + The password, or passphrase, to encrypt the private portion of the + generated ed25519 key. A symmetric encryption key is derived from + 'password', so it is not directly used. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if 'filepath' cannot be encrypted. + + tuf.UnsupportedLibraryError, if 'filepath' cannot be encrypted due to an + invalid configuration setting (i.e., invalid 'tuf.conf.py' setting). + + + Writes key files to '' and '.pub'. + + + None. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # If the caller does not provide a password argument, prompt for one. + if password is None: + message = 'Enter a password for the ED25519 key: ' + password = _get_password(message, confirm=True) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Generate a new ED25519 key object and encrypt it. The cryptography library + # used is determined by the user, or by default (set in + # 'tuf.conf.ED25519_CRYPTO_LIBRARY'). Raise 'tuf.CryptoError' or + # 'tuf.UnsupportedLibraryError', if 'ed25519_key' cannot be encrypted. + ed25519_key = tuf.keys.generate_ed25519_key() + encrypted_key = tuf.keys.encrypt_key(ed25519_key, password) + + # ed25519 public key file contents in metadata format (i.e., does not include + # the keyid portion). + keytype = ed25519_key['keytype'] + keyval = ed25519_key['keyval'] + ed25519key_metadata_format = \ + tuf.keys.format_keyval_to_metadata(keytype, keyval, private=False) + + # Write the public key, conformant to 'tuf.formats.KEY_SCHEMA', to + # '.pub'. + tuf.util.ensure_parent_dir(filepath) + + # Create a tempororary file, write the contents of the public key, and move + # to final destination. + file_object = tuf.util.TempFile() + file_object.write(json.dumps(ed25519key_metadata_format)) + + # The temporary file is closed after the final move. + file_object.move(filepath+'.pub') + + # Write the encrypted key string, conformant to + # 'tuf.formats.ENCRYPTEDKEY_SCHEMA', to ''. + file_object = tuf.util.TempFile() + file_object.write(encrypted_key) + file_object.move(filepath) + + + + + +def import_ed25519_publickey_from_file(filepath): + """ + + Load the ED25519 public key object (conformant to 'tuf.formats.KEY_SCHEMA') + stored in 'filepath'. Return 'filepath' in tuf.formats.ED25519KEY_SCHEMA + format. + + If the TUF key object in 'filepath' contains a private key, it is discarded. + + + filepath: + .pub file, a TUF public key file. + + + tuf.FormatError, if 'filepath' is improperly formatted or is an unexpected + key type. + + + The contents of 'filepath' is read and saved. + + + An ED25519 key object conformant to 'tuf.formats.ED25519KEY_SCHEMA'. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # ED25519 key objects are saved in json and metadata format. Return the + # loaded key object in tuf.formats.ED25519KEY_SCHEMA' format that also + # includes the keyid. + ed25519_key_metadata = tuf.util.load_json_file(filepath) + ed25519_key = tuf.keys.format_metadata_to_key(ed25519_key_metadata) + + # Raise an exception if an unexpected key type is imported. + if ed25519_key['keytype'] != 'ed25519': + message = 'Invalid key type loaded: '+repr(ed25519_key['keytype']) + raise tuf.FormatError(message) + + return ed25519_key + + + + + +def import_ed25519_privatekey_from_file(filepath, password=None): + """ + + Import the encrypted ed25519 TUF key file in 'filepath', decrypt it, and + return the key object in 'tuf.formats.ED25519KEY_SCHEMA' format. + + Which cryptography library performs the cryptographic decryption is + determined by the string set in 'tuf.conf.ED25519_CRYPTO_LIBRARY'. PyCrypto + currently supported. + + The TUF private key (may also contain the public part) is encrypted with AES + 256 and CTR the mode of operation. The password is strengthened with + PBKDF2-HMAC-SHA256. + + + filepath: + file, an RSA encrypted TUF key file. + + password: + The password, or passphrase, to import the private key (i.e., the + encrypted key file 'filepath' must be decrypted before the ed25519 key + object can be returned. + + + tuf.FormatError, if the arguments are improperly formatted or the imported + key object contains an invalid key type (i.e., not 'ed25519'). + + tuf.CryptoError, if 'filepath' cannot be decrypted. + + tuf.UnsupportedLibraryError, if 'filepath' cannot be decrypted due to an + invalid configuration setting (i.e., invalid 'tuf.conf.py' setting). + + + 'password' is used to decrypt the 'filepath' key file. + + + An ed25519 key object of the form: 'tuf.formats.ED25519KEY_SCHEMA'. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # If the caller does not provide a password argument, prompt for one. + if password is None: + message = 'Enter a password for the encrypted ED25519 key file: ' + password = _get_password(message, confirm=True) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Store the encrypted contents of 'filepath' prior to calling the decryption + # routine. + encrypted_key = None + + with open(filepath, 'rb') as file_object: + encrypted_key = file_object.read() + + # Decrypt the loaded key file, calling the appropriate cryptography library + # (i.e., set by the user) and generating the derived encryption key from + # 'password'. Raise 'tuf.CryptoError' or 'tuf.UnsupportedLibraryError' if the + # decryption fails. + key_object = tuf.keys.decrypt_key(encrypted_key, password) + + # Raise an exception if an unexpected key type is imported. + if key_object['keytype'] != 'ed25519': + message = 'Invalid key type loaded: '+repr(key_object['keytype']) + raise tuf.FormatError(message) + + return key_object + + + + + +def get_metadata_filenames(metadata_directory=None): + """ + + Return a dictionary containing the filenames of the top-level roles. + If 'metadata_directory' is set to 'metadata', the dictionary + returned would contain: + + filenames = {'root': 'metadata/root.txt', + 'targets': 'metadata/targets.txt', + 'release': 'metadata/release.txt', + 'timestamp': 'metadata/timestamp.txt'} + + If the metadata directory is not set by the caller, the current + directory is used. + + + metadata_directory: + The directory containing the metadata files. + + + tuf.FormatError, if 'metadata_directory' is improperly formatted. + + + None. + + + A dictionary containing the expected filenames of the top-level + metadata files, such as 'root.txt' and 'release.txt'. + """ + + # Does 'metadata_directory' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(metadata_directory) + + if metadata_directory is None: + metadata_directory = '.' + + # Does 'metadata_directory' have the correct format? + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(metadata_directory) + + # Store the filepaths of the top-level roles, including the + # 'metadata_directory' for each one. + filenames = {} + + filenames[ROOT_FILENAME] = \ + os.path.join(metadata_directory, ROOT_FILENAME) + + filenames[TARGETS_FILENAME] = \ + os.path.join(metadata_directory, TARGETS_FILENAME) + + filenames[RELEASE_FILENAME] = \ + os.path.join(metadata_directory, RELEASE_FILENAME) + + filenames[TIMESTAMP_FILENAME] = \ + os.path.join(metadata_directory, TIMESTAMP_FILENAME) + + return filenames + + + + + +def get_metadata_file_info(filename): + """ + + Retrieve the file information of 'filename'. The object returned + conforms to 'tuf.formats.FILEINFO_SCHEMA'. The information + generated for 'filename' is stored in metadata files like 'targets.txt'. + The fileinfo object returned has the form: + fileinfo = {'length': 1024, + 'hashes': {'sha256': 1233dfba312, ...}, + 'custom': {...}} + + + filename: + The metadata file whose file information is needed. It must exist. + + + tuf.FormatError, if 'filename' is improperly formatted. + + tuf.Error, if 'filename' doesn't exist. + + + The file is opened and information about the file is generated, + such as file size and its hash. + + + A dictionary conformant to 'tuf.formats.FILEINFO_SCHEMA'. This + dictionary contains the length, hashes, and custom data about the + 'filename' metadata file. + """ + + # Does 'filename' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filename) + + if not os.path.isfile(filename): + message = repr(filename)+' is not a file.' + raise tuf.Error(message) + + # Note: 'filehashes' is a dictionary of the form + # {'sha256': 1233dfba312, ...}. 'custom' is an optional + # dictionary that a client might define to include additional + # file information, such as the file's author, version/revision + # numbers, etc. + filesize, filehashes = \ + tuf.util.get_file_details(filename, tuf.conf.REPOSITORY_HASH_ALGORITHMS) + custom = None + + return tuf.formats.make_fileinfo(filesize, filehashes, custom) + + + + + + +def get_target_hash(self, target_filepath): + """ + + Compute the hash of 'target_filepath'. This is useful in conjunction with + the "path_hash_prefixes" attribute in a delegated targets role, which + tells us which paths it is implicitly responsible for. + + The repository may optionally organize targets into hashed bins to ease + target delegations and role metadata management. The use of consistent + hashing allows for a uniform distribution of targets into bins. + + + target_filepath: + The path to the target file on the repository. This will be relative to + the 'targets' (or equivalent) directory on a given mirror. + + + None. + + + None. + + + The hash of 'target_filepath'. + """ + + # Calculate the hash of the filepath to determine which bin to find the + # target. The client currently assumes the repository uses + # 'HASH_FUNCTION' to generate hashes. + + digest_object = tuf.hash.digest(HASH_FUNCTION) + + try: + digest_object.update(target_filepath) + except UnicodeEncodeError: + # Sometimes, there are Unicode characters in target paths. We assume a + # UTF-8 encoding and try to hash that. + digest_object = tuf.hash.digest(HASH_FUNCTION) + encoded_target_filepath = target_filepath.encode('utf-8') + digest_object.update(encoded_target_filepath) + + target_filepath_hash = digest_object.hexdigest() + + return target_filepath_hash + + + + + +def generate_root_metadata(version, expiration_date, consistent_snapshots): + """ + + Create the root metadata. 'tuf.roledb.py' and 'tuf.keydb.py' are read and + the information returned by these modules is used to generate the root + metadata object. + + + version: + The metadata version number. Clients use the version number to + determine if the downloaded version is newer than the one currently + trusted. + + expiration_date: + The expiration date, in UTC, of the metadata file. Conformant to + 'tuf.formats.TIME_SCHEMA'. + + consistent_snapshots: + + + tuf.FormatError, if the generated root metadata object could not + be generated with the correct format. + + tuf.Error, if an error is encountered while generating the root + metadata object. + + + The contents of 'tuf.keydb.py' and 'tuf.roledb.py' are read. + + + A root metadata object, conformant to 'tuf.formats.ROOT_SCHEMA'. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any of the arguments are improperly formatted. + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + tuf.formats.TIME_SCHEMA.check_match(expiration_date) + tuf.formats.BOOLEAN_SCHEMA.check_match(consistent_snapshots) + + # The role and key dictionaries to be saved in the root metadata object. + # Conformant to 'ROLEDICT_SCHEMA' and 'KEYDICT_SCHEMA', respectively. + roledict = {} + keydict = {} + + # Extract the role, threshold, and keyid information of the top-level roles, + # which Root stores in its metadata. The necessary role metadata is generated + # from this information. + for rolename in ['root', 'targets', 'release', 'timestamp']: + + # If a top-level role is missing from 'tuf.roledb.py', raise an exception. + if not tuf.roledb.role_exists(rolename): + raise tuf.Error(repr(rolename)+' not in "tuf.roledb".') + + # Keep track of the keys loaded so that duplicates is avoided. + keyids = [] + + # Generate keys for the keyids listed by the role being processed. + for keyid in tuf.roledb.get_role_keyids(rolename): + key = tuf.keydb.get_key(keyid) + + # If 'key' is an RSA key, it would conform to 'tuf.formats.RSAKEY_SCHEMA', + # and have the form: + # {'keytype': 'rsa', + # 'keyid': keyid, + # 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + # 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + keyid = key['keyid'] + if keyid not in keydict: + + # This appears to be a new keyid. Let's generate the key for it. + if key['keytype'] in ['rsa', 'ed25519']: + keytype = key['keytype'] + keyval = key['keyval'] + keydict[keyid] = \ + tuf.keys.format_keyval_to_metadata(keytype, keyval) + + # This is not a recognized key. Raise an exception. + else: + raise tuf.Error('Unsupported keytype: '+keyid) + + # Do we have a duplicate? + if keyid in keyids: + raise tuf.Error('Same keyid listed twice: '+keyid) + + # Add the loaded keyid for the role being processed. + keyids.append(keyid) + + # Generate and store the role data belonging to the processed role. + role_threshold = tuf.roledb.get_role_threshold(rolename) + role_metadata = tuf.formats.make_role_metadata(keyids, role_threshold) + roledict[rolename] = role_metadata + + # Generate the root metadata object. + root_metadata = tuf.formats.RootFile.make_metadata(version, expiration_date, + keydict, roledict, + consistent_snapshots) + + return root_metadata + + + + + +def generate_targets_metadata(targets_directory, target_files, version, + expiration_date, delegations=None, + write_consistent_targets=False): + """ + + Generate the targets metadata object. The targets in 'target_files' must + exist at the same path they should on the repo. 'target_files' is a list of + targets. The 'custom' field of the targets metadata is not currently + supported. + + + targets_directory: + The directory containing the target files and directories of the + repository. + + target_files: + The target files tracked by 'targets.txt'. 'target_files' is a list of + target paths that are relative to the targets directory (e.g., + ['file1.txt', 'Django/module.py']). + + version: + The metadata version number. Clients use the version number to + determine if the downloaded version is newer than the one currently + trusted. + + expiration_date: + The expiration date, in UTC, of the metadata file. Conformant to + 'tuf.formats.TIME_SCHEMA'. + + delegations: + The delegations made by the targets role to be generated. 'delegations' + must match 'tuf.formats.DELEGATIONS_SCHEMA'. + + write_consitent_targets: + Boolean. + + + tuf.FormatError, if an error occurred trying to generate the targets + metadata object. + + tuf.Error, if any of the target files cannot be read. + + + The target files are read and file information generated about them. + + + A targets metadata object, conformant to 'tuf.formats.TARGETS_SCHEMA'. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(targets_directory) + tuf.formats.PATHS_SCHEMA.check_match(target_files) + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + tuf.formats.TIME_SCHEMA.check_match(expiration_date) + tuf.formats.BOOLEAN_SCHEMA.check_match(write_consistent_targets) + + if delegations is not None: + tuf.formats.DELEGATIONS_SCHEMA.check_match(delegations) + + # Store the file attributes of targets in 'target_files'. 'filedict', + # conformant to 'tuf.formats.FILEDICT_SCHEMA', is added to the targets + # metadata object returned. + filedict = {} + + # Ensure the user is aware of a non-existent 'target_directory', and convert + # it to its abosolute path, if it exists. + targets_directory = _check_directory(targets_directory) + + # Generate the fileinfo of all the target files listed in 'target_files'. + for target in target_files: + + # The root-most folder of the targets directory should not be included. + # (e.g., 'targets/more_targets/somefile.txt' -> 'more_targets/somefile.txt') + relative_targetpath = target + target_path = os.path.join(targets_directory, target) + + # Ensure all target files listed in 'target_files' exist. If just one of + # these files does not exist, raise an exception. + if not os.path.exists(target_path): + message = repr(target_path)+' cannot be read. Unable to generate '+ \ + 'targets metadata.' + raise tuf.Error(message) + + filedict[relative_targetpath] = get_metadata_file_info(target_path) + + if write_consistent_targets: + for target_digest in filedict[relative_targetpath]['hashes'].values(): + dirname, basename = os.path.split(target_path) + digest_filename = target_digest + '.' + basename + digest_target = os.path.join(dirname, digest_filename) + + if not os.path.exists(digest_target): + logger.warn('Copying target file to ' + repr(digest_target)) + shutil.copy(target_path, digest_target) + + # Generate the targets metadata object. + targets_metadata = tuf.formats.TargetsFile.make_metadata(version, + expiration_date, + filedict, + delegations) + + return targets_metadata + + + + + +def generate_release_metadata(metadata_directory, version, expiration_date, + root_filename, targets_filename, + consistent_snapshots): + """ + + Create the release metadata. The minimum metadata must exist + (i.e., 'root.txt' and 'targets.txt'). This will also look through + the 'targets/' directory in 'metadata_directory' and the resulting + release file will list all the delegated roles. + + + metadata_directory: + The directory containing the 'root.txt' and 'targets.txt' metadata + files. + + version: + The metadata version number. Clients use the version number to + determine if the downloaded version is newer than the one currently + trusted. + + expiration_date: + The expiration date, in UTC, of the metadata file. + Conformant to 'tuf.formats.TIME_SCHEMA'. + + root_filename: + + targets_filename: + + consistent_snapshots: + + + tuf.FormatError, if 'metadata_directory' is improperly formatted. + + tuf.Error, if an error occurred trying to generate the release metadata + object. + + + The 'root.txt' and 'targets.txt' files are read. + + + The release metadata object, conformant to 'tuf.formats.RELEASE_SCHEMA'. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PATH_SCHEMA.check_match(metadata_directory) + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + tuf.formats.TIME_SCHEMA.check_match(expiration_date) + tuf.formats.PATH_SCHEMA.check_match(root_filename) + tuf.formats.PATH_SCHEMA.check_match(targets_filename) + + metadata_directory = _check_directory(metadata_directory) + + # Retrieve the fileinfo of 'root.txt' and 'targets.txt'. This file + # information includes data such as file length, hashes of the file, etc. + filedict = {} + filedict[ROOT_FILENAME] = get_metadata_file_info(root_filename) + filedict[TARGETS_FILENAME] = get_metadata_file_info(targets_filename) + + # Add compressed versions of the 'targets.txt' and 'root.txt' metadata, + # if they exist. + for extension in SUPPORTED_COMPRESSION_EXTENSIONS: + compressed_root_filename = root_filename+extension + compressed_targets_filename = targets_filename+extension + + # If the compressed versions of the root and targets metadata is found, + # add their file attributes to 'filedict'. + if os.path.exists(compressed_root_filename): + filedict[ROOT_FILENAME+extension] = \ + get_metadata_file_info(compressed_root_filename) + if os.path.exists(compressed_targets_filename): + filedict[TARGETS_FILENAME+extension] = \ + get_metadata_file_info(compressed_targets_filename) + + # Walk the 'targets/' directory and generate the fileinfo of all the role + # files found. This information is stored in the 'meta' field of the release + # metadata object. + targets_metadata = os.path.join(metadata_directory, 'targets') + if os.path.exists(targets_metadata) and os.path.isdir(targets_metadata): + for directory_path, junk_directories, files in os.walk(targets_metadata): + + # 'files' here is a list of file names. + for basename in files: + metadata_path = os.path.join(directory_path, basename) + metadata_name = \ + metadata_path[len(metadata_directory):].lstrip(os.path.sep) + + # Strip the digest if 'consistent_snapshots' is True. + # Example: 'targets/unclaimed/13df98ab0.django.txt' --> + # 'targets/unclaimed/django.txt' + metadata_name, digest_junk = \ + _strip_consistent_snapshots_digest(metadata_name, consistent_snapshots) + + # All delegated roles are added to the release file, including + # compressed versions. + for metadata_extension in METADATA_EXTENSIONS: + if metadata_name.endswith(metadata_extension): + rolename = metadata_name[:-len(metadata_extension)] + + # Obsolete role files may still be found. Ensure only roles loaded + # in the roledb are included in the release metadata. + if tuf.roledb.role_exists(rolename): + filedict[metadata_name] = get_metadata_file_info(metadata_path) + + # Generate the release metadata object. + release_metadata = tuf.formats.ReleaseFile.make_metadata(version, + expiration_date, + filedict) + + return release_metadata + + + + + +def generate_timestamp_metadata(release_filename, version, + expiration_date, compressions=()): + """ + + Generate the timestamp metadata object. The 'release.txt' file must exist. + + + release_filename: + The required filename of the release metadata file. + + version: + The timestamp's version number. Clients use the version number to + determine if the downloaded version is newer than the one currently + trusted. + + expiration_date: + The expiration date, in UTC, of the metadata file, conformant to + 'tuf.formats.TIME_SCHEMA'. + + release_filename: + + compressions: + Compression extensions (e.g., 'gz'). If 'release.txt' is also saved in + compressed form, these compression extensions should be stored in + 'compressions' so the compressed timestamp files can be added to the + timestamp metadata object. + + + tuf.FormatError, if the generated timestamp metadata object cannot be + formatted correctly, or one of the arguments is improperly formatted. + + + None. + + + A timestamp metadata object, conformant to 'tuf.formats.TIMESTAMP_SCHEMA'. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PATH_SCHEMA.check_match(release_filename) + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + tuf.formats.TIME_SCHEMA.check_match(expiration_date) + tuf.formats.COMPRESSIONS_SCHEMA.check_match(compressions) + + # Retrieve the fileinfo of the release metadata file. + # This file information contains hashes, file length, custom data, etc. + fileinfo = {} + fileinfo[RELEASE_FILENAME] = get_metadata_file_info(release_filename) + + # Save the fileinfo of the compressed versions of 'timestamp.txt' + # in 'fileinfo'. Log the files included in 'fileinfo'. + for file_extension in compressions: + if not len(file_extension): + continue + + compressed_filename = release_filename + '.' + file_extension + try: + compressed_fileinfo = get_metadata_file_info(compressed_filename) + + except: + logger.warn('Cannot get fileinfo about '+str(compressed_filename)) + + else: + logger.info('Including fileinfo about '+str(compressed_filename)) + fileinfo[RELEASE_FILENAME+'.'+file_extension] = compressed_fileinfo + + # Generate the timestamp metadata object. + timestamp_metadata = tuf.formats.TimestampFile.make_metadata(version, + expiration_date, + fileinfo) + + return timestamp_metadata + + + + + +def sign_metadata(metadata_object, keyids, filename): + """ + + Sign a metadata object. If any of the keyids have already signed the file, + the old signature is replaced. The keys in 'keyids' must already be + loaded in 'tuf.keydb'. + + + metadata_object: + The metadata object to sign. For example, 'metadata' might correspond to + 'tuf.formats.ROOT_SCHEMA' or 'tuf.formats.TARGETS_SCHEMA'. + + keyids: + The keyids list of the signing keys. + + filename: + The intended filename of the signed metadata object. + For example, 'root.txt' or 'targets.txt'. This function + does NOT save the signed metadata to this filename. + + + tuf.FormatError, if a valid 'signable' object could not be generated or + the arguments are improperly formatted. + + tuf.Error, if an invalid keytype was found in the keystore. + + + None. + + + A signable object conformant to 'tuf.formats.SIGNABLE_SCHEMA'. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ANYROLE_SCHEMA.check_match(metadata_object) + tuf.formats.KEYIDS_SCHEMA.check_match(keyids) + tuf.formats.PATH_SCHEMA.check_match(filename) + + # Make sure the metadata is in 'signable' format. That is, + # it contains a 'signatures' field containing the result + # of signing the 'signed' field of 'metadata' with each + # keyid of 'keyids'. + signable = tuf.formats.make_signable(metadata_object) + + # Sign the metadata with each keyid in 'keyids'. + for keyid in keyids: + + # Load the signing key. + key = tuf.keydb.get_key(keyid) + logger.info('Signing '+repr(filename)+' with '+key['keyid']) + + # Create a new signature list. If 'keyid' is encountered, + # do not add it to new list. + signatures = [] + for signature in signable['signatures']: + if not keyid == signature['keyid']: + signatures.append(signature) + signable['signatures'] = signatures + + # Generate the signature using the appropriate signing method. + if key['keytype'] in SUPPORTED_KEY_TYPES: + if len(key['keyval']['private']): + signed = signable['signed'] + signature = tuf.keys.create_signature(key, signed) + signable['signatures'].append(signature) + else: + logger.warn('Private key unset. Skipping: '+repr(keyid)) + + else: + raise tuf.Error('The keydb contains a key with an invalid key type.') + + # Raise 'tuf.FormatError' if the resulting 'signable' is not formatted + # correctly. + tuf.formats.check_signable_object_format(signable) + + return signable + + + + + +def write_metadata_file(metadata, filename, compressions, consistent_snapshots): + """ + + If necessary, write the 'metadata' signable object to 'filename', and the + compressed version of the metadata file if 'compression' is set. + Note: Compression algorithms like gzip attach a timestamp to compressed + files, so a metadata file compressed multiple times may generate different + digests even though the uncompressed content has not changed. + + + metadata: + The object that will be saved to 'filename', conformant to + 'tuf.formats.SIGNABLE_SCHEMA'. + + filename: + The filename of the metadata to be written (e.g., 'root.txt'). + If a compression algorithm is specified in 'compressions', the + compression extention is appended to 'filename'. + + compressions: + Specify the algorithms, as a list of strings, used to compress the file; + The only currently available compression option is 'gz' (gzip). + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.Error, if the directory of 'filename' does not exist. + + Any other runtime (e.g., IO) exception. + + + The 'filename' (or the compressed filename) file is created, or overwritten + if it exists. + + + None. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.SIGNABLE_SCHEMA.check_match(metadata) + tuf.formats.PATH_SCHEMA.check_match(filename) + tuf.formats.COMPRESSIONS_SCHEMA.check_match(compressions) + tuf.formats.BOOLEAN_SCHEMA.check_match(consistent_snapshots) + + # Verify the directory of 'filename', and convert 'filename' to its absolute + # path so that temporary files are moved to their expected destinations. + filename = os.path.abspath(filename) + written_filename = filename + _check_directory(os.path.dirname(filename)) + consistent_filenames = [] + + # Generate the actual metadata file content of 'metadata'. Metadata is + # saved as json and includes formatting, such as indentation and sorted + # objects. The new digest of 'metadata' is also calculated to help determine + # if re-saving is required. + file_content, new_digests = _get_written_metadata_and_digests(metadata) + + if consistent_snapshots: + for new_digest in new_digests.values(): + dirname, basename = os.path.split(filename) + digest_and_filename = new_digest + '.' + basename + consistent_filenames.append(os.path.join(dirname, digest_and_filename)) + written_filename = consistent_filenames.pop() + + # Verify whether new metadata needs to be written (i.e., has not been + # previously written or has changed. + write_new_metadata = False + + # Has the uncompressed metadata changed? Does it exist? If so, set + # 'write_compressed_version' to True so that it is written. + # compressed metadata should only be written if it does not exist or the + # uncompressed version has changed). + try: + file_length_junk, old_digests = tuf.util.get_file_details(written_filename) + if old_digests != new_digests: + write_new_metadata = True + + # 'tuf.Error' raised if 'filename' does not exist. + except tuf.Error, e: + write_new_metadata = True + + if write_new_metadata: + # The 'metadata' object is written to 'file_object', including compressed + # versions. To avoid partial metadata from being written, 'metadata' is + # first written to a temporary location (i.e., 'file_object') and then moved + # to 'filename'. + file_object = tuf.util.TempFile() + + # Serialize 'metadata' to the file-like object and then write + # 'file_object' to disk. The dictionary keys of 'metadata' are sorted + # and indentation is used. The 'tuf.util.TempFile' file-like object is + # automically closed after the final move. + file_object.write(file_content) + logger.info('Saving ' + repr(written_filename)) + file_object.move(written_filename) + + for consistent_filename in consistent_filenames: + logger.info('Saving ' + repr(consistent_filename)) + shutil.copy(written_filename, consistent_filename) + + + # Generate the compressed versions of 'metadata', if necessary. A compressed + # file may be written (without needed to write the uncompressed version) if + # the repository maintainer adds compression after writting the the + # uncompressed version. + for compression in compressions: + file_object = None + + # Ignore the empty string that signifies non-compression. The uncompressed + # file was previously written above, if necessary. + if not len(compression): + continue + + elif compression == 'gz': + file_object = tuf.util.TempFile() + compressed_filename = filename + '.gz' + + # Instantiate a gzip object, but save compressed content to + # 'file_object' (i.e., GzipFile instance is based on its 'fileobj' + # argument). + with gzip.GzipFile(fileobj=file_object, mode='wb') as gzip_object: + gzip_object.write(file_content) + + else: + raise tuf.FormatError('Unknown compression algorithm: '+repr(compression)) + + # Save the compressed version, ensuring an unchanged file is not re-saved. + # Re-savign the same compressed version may cause its digest to unexpectedly + # change (gzip includes a timestamp) even though content has not changed. + _write_compressed_metadata(file_object, compressed_filename, + consistent_snapshots) + return written_filename + + + + + +def _write_compressed_metadata(file_object, compressed_filename, + consistent_snapshots): + """ + Write compressed versions of metadata, ensuring compressed file that have + not changed are not re-written, the digest of the compressed file is properly + added to the compressed filename, and consistent snapshots are also saved. + Ensure compressed files are written to a temporary location, and then + moved to their destinations. + """ + + # If a consistent snapshot is unneeded, 'file_object' may be simply moved + # 'compressed_filename' if not already written. + if not consistent_snapshots: + if not os.path.exists(compressed_filename): + file_object.move(compressed_filename) + + # The temporary file must be closed if 'file_object.move()' is not used. + # tuf.util.TempFile() automatically closes the temp file when move() is + # called + else: + file_object.close_temp_file() + + # Consistent snapshots = True. Ensure the file's digest is included in the + # compressed filename written, provided it does not already exist. + else: + compressed_content = file_object.read() + new_digests = [] + consistent_filenames = [] + + # Multiple snapshots may be written if the repository uses multiple + # hash algorithms. Generate the digest of the compressed content. + for hash_algorithm in tuf.conf.REPOSITORY_HASH_ALGORITHMS: + digest_object = tuf.hash.digest(hash_algorithm) + digest_object.update(compressed_content) + new_digests.append(digest_object.hexdigest()) + + # Attach each digest to the compressed consistent snapshot filename. + for new_digest in new_digests: + dirname, basename = os.path.split(compressed_filename) + digest_and_filename = new_digest + '.' + basename + consistent_filenames.append(os.path.join(dirname, digest_and_filename)) + + # Move the 'tuf.util.TempFile' object to one of the filenames so that it is + # saved and the temporary file closed. Any remaining consistent snapshots + # may still need to be copied or linked. + compressed_filename = consistent_filenames.pop() + if not os.path.exists(compressed_filename): + logger.info('Saving ' + repr(compressed_filename)) + file_object.move(compressed_filename) + + # Save any remaining compressed consistent snapshots. + for consistent_filename in consistent_filenames: + if not os.path.exists(consistent_filename): + logger.info('Saving ' + repr(consistent_filename)) + shutil.copy(compressed_filename, consistent_filename) + + + + + +def create_tuf_client_directory(repository_directory, client_directory): + """ + + Create a client directory structure that the 'tuf.interposition' package + and 'tuf.client.updater' module expect of clients. Metadata files + downloaded from a remote TUF repository are saved to 'client_directory'. + The Root file must initially exist before an update request can be + satisfied. create_tuf_client_directory() ensures the minimum metadata + is copied and that required directories ('previous' and 'current') are + created in 'client_directory'. Software updaters integrating TUF may + use the client directory created as an initial copy of the repository's + metadadata. + + + repository_directory: + The path of the root repository directory. The 'metadata' and 'targets' + sub-directories should be available in 'repository_directory'. The + metadata files of 'repository_directory' are copied to 'client_directory'. + + client_directory: + The path of the root client directory. The 'current' and 'previous' + sub-directies are created and will store the metadata files copied + from 'repository_directory'. 'client_directory' will store metadata + and target files downloaded from a TUF repository. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.RepositoryError, if the metadata directory in 'client_directory' + already exists. + + + Copies metadata files and directories from 'repository_directory' to + 'client_directory'. Parent directories are created if they do not exist. + + + None. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PATH_SCHEMA.check_match(repository_directory) + tuf.formats.PATH_SCHEMA.check_match(client_directory) + + # Set the absolute path of the Repository's metadata directory. The metadata + # directory should be the one served by the Live repository. At a minimum, + # the repository's root file must be copied. + repository_directory = os.path.abspath(repository_directory) + metadata_directory = os.path.join(repository_directory, + METADATA_DIRECTORY_NAME) + + # Set the client's metadata directory, which will store the metadata copied + # from the repository directory set above. + client_directory = os.path.abspath(client_directory) + client_metadata_directory = os.path.join(client_directory, + METADATA_DIRECTORY_NAME) + + # If the client's metadata directory does not already exist, create it and + # any of its parent directories, otherwise raise an exception. An exception + # is raised to avoid accidently overwritting previous metadata. + try: + os.makedirs(client_metadata_directory) + except OSError, e: + if e.errno == errno.EEXIST: + message = 'Cannot create a fresh client metadata directory: '+ \ + repr(client_metadata_directory)+'. Already exists.' + raise tuf.RepositoryError(message) + else: + raise + + # Move all metadata to the client's 'current' and 'previous' directories. + # The root metadata file MUST exist in '{client_metadata_directory}/current'. + # 'tuf.interposition' and 'tuf.client.updater.py' expect the 'current' and + # 'previous' directories to exist under 'metadata'. + client_current = os.path.join(client_metadata_directory, 'current') + client_previous = os.path.join(client_metadata_directory, 'previous') + shutil.copytree(metadata_directory, client_current) + shutil.copytree(metadata_directory, client_previous) + + + +if __name__ == '__main__': + # The interactive sessions of the documentation strings can + # be tested by running libtuf.py as a standalone module: + # $ python libtuf.py. + import doctest + doctest.testmod() diff --git a/tuf/log.py b/tuf/log.py index 6ba9ece5..e7ba03c6 100755 --- a/tuf/log.py +++ b/tuf/log.py @@ -77,7 +77,7 @@ # Example format for '_FORMAT_STRING': # [2013-08-13 15:21:18,068 UTC] [tuf] [INFO][_update_metadata:851@updater.py] _FORMAT_STRING = '[%(asctime)s UTC] [%(name)s] [%(levelname)s]'+\ - '[%(funcName)s:%(lineno)s@%(filename)s] %(message)s' + '[%(funcName)s:%(lineno)s@%(filename)s]\n%(message)s\n' # Ask all Formatter instances to talk GMT. Set the 'converter' attribute of # 'logging.Formatter' so that all formatters use Greenwich Mean Time. diff --git a/tuf/pycrypto_keys.py b/tuf/pycrypto_keys.py new file mode 100755 index 00000000..ba6533bc --- /dev/null +++ b/tuf/pycrypto_keys.py @@ -0,0 +1,895 @@ +""" + + pycrypto_keys.py + + + Vladimir Diaz + + + October 7, 2013. + + + See LICENSE for licensing information. + + + The goal of this module is to support public-key and general-purpose + cryptography through the PyCrypto library. The RSA-related functions provided: + generate_rsa_public_and_private() + create_rsa_signature() + verify_rsa_signature() + create_rsa_encrypted_pem() + create_rsa_public_and_private_from_encrypted_pem() + + The general-purpose functions include: + encrypt_key() + decrypt_key() + + PyCrypto (i.e., the 'Crypto' package) performs the actual cryptographic + operations and the functions listed above can be viewed as the easy-to-use + public interface. + + https://github.com/dlitz/pycrypto + https://en.wikipedia.org/wiki/RSA_(algorithm) + https://en.wikipedia.org/wiki/Advanced_Encryption_Standard + https://en.wikipedia.org/wiki/3des + https://en.wikipedia.org/wiki/PBKDF + + TUF key files are encrypted with the AES-256-CTR-Mode symmetric key + algorithm. User passwords are strengthened with PBKDF2, currently set to + 100,000 passphrase iterations. The previous evpy implementation used 1,000 + iterations. + + PEM-encrypted RSA key files use the Triple Data Encryption Algorithm (3DES) + and Cipher-block chaining (CBC) for the mode of operation. Password-Based Key + Derivation Function 1 (PBKF1) + MD5. + """ + +import os +import binascii +import json + +# Crypto.PublicKey (i.e., PyCrypto's public-key cryptography modules) supports +# algorithms like the Digital Signature Algorithm (DSA) and the ElGamal +# encryption system. 'Crypto.PublicKey.RSA' is needed here to generate, sign, +# and verify RSA keys. +import Crypto.PublicKey.RSA + +# PyCrypto requires 'Crypto.Hash' hash objects to generate PKCS#1 PSS +# signatures (i.e., Crypto.Signature.PKCS1_PSS). +import Crypto.Hash.SHA256 + +# RSA's probabilistic signature scheme with appendix (RSASSA-PSS). +# PKCS#1 v1.5 is available for compatibility with existing applications, but +# RSASSA-PSS is encouraged for newer applications. RSASSA-PSS generates +# a random salt to ensure the signature generated is probabilistic rather than +# deterministic (e.g., PKCS#1 v1.5). +# http://en.wikipedia.org/wiki/RSA-PSS#Schemes +# https://tools.ietf.org/html/rfc3447#section-8.1 +import Crypto.Signature.PKCS1_PSS + +# Import PyCrypto's Key Derivation Function (KDF) module. 'keystore.py' +# needs this module to derive a secret key according to the Password-Based +# Key Derivation Function 2 specification. The derived key is used as the +# symmetric key to encrypt TUF key information. PyCrypto's implementation: +# Crypto.Protocol.KDF.PBKDF2(). PKCS#5 v2.0 PBKDF2 specification: +# http://tools.ietf.org/html/rfc2898#section-5.2 +import Crypto.Protocol.KDF + +# PyCrypto's AES implementation. AES is a symmetric key algorithm that +# operates on fixed block sizes of 128-bits. +# https://en.wikipedia.org/wiki/Advanced_Encryption_Standard +import Crypto.Cipher.AES + +# 'Crypto.Random' is a cryptographically strong version of Python's standard +# "random" module. Random bits of data is needed for salts and +# initialization vectors suitable for the encryption algorithms used in +# 'pycrypto_keys.py'. +import Crypto.Random + +# The mode of operation is presently set to CTR (CounTeR Mode) for symmetric +# block encryption (AES-256, where the symmetric key is 256 bits). PyCrypto +# provides a callable stateful block counter that can update successive blocks +# when needed. The initial random block, or initialization vector (IV), can +# be set to begin the process of incrementing the 128-bit blocks and allowing +# the AES algorithm to perform cipher block operations on them. +import Crypto.Util.Counter + +# Import the TUF package and TUF-defined exceptions in __init__.py. +import tuf + +# Digest objects needed to generate hashes. +import tuf.hash + +# Perform object format-checking. +import tuf.formats + +# Extract the cryptography library settings. +import tuf.conf + +# Import key files containing json data. +import tuf.util + +# Recommended RSA key sizes: +# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 +# According to the document above, revised May 6, 2003, RSA keys of +# size 3072 provide security through 2031 and beyond. +_DEFAULT_RSA_KEY_BITS = 3072 + +# The delimiter symbol used to separate the different sections +# of encrypted files (i.e., salt, iterations, hmac, IV, ciphertext). +# This delimiter is arbitrarily chosen and should not occur in +# the hexadecimal representations of the fields it is separating. +_ENCRYPTION_DELIMITER = '@@@@' + +# AES key size. Default key size = 32 bytes = AES-256. +_AES_KEY_SIZE = 32 + +# Default salt size, in bytes. A 128-bit salt (i.e., a random sequence of data +# to protect against attacks that use precomputed rainbow tables to crack +# password hashes) is generated for PBKDF2. +_SALT_SIZE = 16 + +# Default PBKDF2 passphrase iterations. The current "good enough" number +# of passphrase iterations. We recommend that important keys, such as root, +# be kept offline. 'tuf.conf.PBKDF2_ITERATIONS' should increase as CPU +# speeds increase, set here at 100,000 iterations by default (in 2013). +# Repository maintainers may opt to modify the default setting according to +# their security needs and computational restrictions. A strong user password +# is still important. Modifying the number of iterations will result in a new +# derived key+PBDKF2 combination if the key is loaded and re-saved, overriding +# any previous iteration setting used by the old '.key'. +# https://en.wikipedia.org/wiki/PBKDF2 +_PBKDF2_ITERATIONS = tuf.conf.PBKDF2_ITERATIONS + + +def generate_rsa_public_and_private(bits=_DEFAULT_RSA_KEY_BITS): + """ + + Generate public and private RSA keys with modulus length 'bits'. + The public and private keys returned conform to 'tuf.formats.PEMRSA_SCHEMA' + and have the form: + '-----BEGIN RSA PUBLIC KEY----- ...' + + or + + '-----BEGIN RSA PRIVATE KEY----- ...' + + The public and private keys are returned as strings in PEM format. + + Although PyCrypto sets a 1024-bit minimum key size, + generate_rsa_public_and_private() enforces a minimum key size of 2048 bits. + If 'bits' is unspecified, a 3072-bit RSA key is generated, which is the key + size recommended by TUF. + + >>> public, private = generate_rsa_public_and_private(2048) + >>> tuf.formats.PEMRSA_SCHEMA.matches(public) + True + >>> tuf.formats.PEMRSA_SCHEMA.matches(private) + True + + + bits: + The key size, or key length, of the RSA key. 'bits' must be 2048, or + greater, and a multiple of 256. + + + tuf.FormatError, if 'bits' does not contain the correct format. + + ValueError, if an exception occurs in the RSA key generation routine. + 'bits' must be a multiple of 256. The 'ValueError' exception is raised by + the PyCrypto key generation function. + + + The RSA keys are generated by PyCrypto's Crypto.PublicKey.RSA.generate(). + + + A (public, private) tuple containing the RSA keys in PEM format. + """ + + # Does 'bits' have the correct format? + # This check will ensure 'bits' conforms to 'tuf.formats.RSAKEYBITS_SCHEMA'. + # 'bits' must be an integer object, with a minimum value of 2048. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) + + # Generate the public and private RSA keys. The PyCrypto module performs + # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 + # or not a multiple of 256, although a 2048-bit minimum is enforced by + # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). + rsa_key_object = Crypto.PublicKey.RSA.generate(bits) + + # Extract the public & private halves of the RSA key and generate their + # PEM-formatted representations. Return the key pair as a (public, private) + # tuple, where each RSA is a string in PEM format. + private = rsa_key_object.exportKey(format='PEM') + rsa_pubkey = rsa_key_object.publickey() + public = rsa_pubkey.exportKey(format='PEM') + + return public, private + + + + + +def create_rsa_signature(private_key, data): + """ + + Generate an RSASSA-PSS signature. The signature, and the method (signature + algorithm) used, is returned as a (signature, method) tuple. + + The signing process will use 'private_key' and 'data' to generate the + signature. + + RFC3447 - RSASSA-PSS + http://www.ietf.org/rfc/rfc3447.txt + + >>> public, private = generate_rsa_public_and_private(2048) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature, method = create_rsa_signature(private, data) + >>> tuf.formats.NAME_SCHEMA.matches(method) + True + >>> method == 'RSASSA-PSS' + True + >>> tuf.formats.PYCRYPTOSIGNATURE_SCHEMA.matches(method) + True + + + private_key: + The private RSA key, a string in PEM format. + + data: + Data object used by create_rsa_signature() to generate the signature. + + + tuf.FormatError, if 'private_key' is improperly formatted. + + TypeError, if 'private_key' is unset. + + tuf.CryptoError, if the signature cannot be generated. + + + PyCrypto's 'Crypto.Signature.PKCS1_PSS' called to generate the signature. + + + A (signature, method) tuple, where the signature is a string and the method + is 'RSASSA-PSS'. + """ + + # Does 'private_key' have the correct format? + # This check will ensure 'private_key' conforms to 'tuf.formats.PEMRSA_SCHEMA'. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(private_key) + + # Signing the 'data' object requires a private key. + # The 'RSASSA-PSS' (i.e., PyCrypto module) signing method is the + # only method currently supported. + method = 'RSASSA-PSS' + signature = None + + # Verify the signature, but only if the private key has been set. The private + # key is a NULL string if unset. Although it may be clearer to explicit check + # that 'private_key' is not '', we can/should check for a value and not + # compare identities with the 'is' keyword. + if len(private_key): + # Calculate the SHA256 hash of 'data' and generate the hash's PKCS1-PSS + # signature. + try: + rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) + sha256_object = Crypto.Hash.SHA256.new(data) + pkcs1_pss_signer = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) + signature = pkcs1_pss_signer.sign(sha256_object) + except (ValueError, IndexError, TypeError), e: + message = 'An RSA signature could not be generated.' + raise tuf.CryptoError(message) + else: + raise TypeError('The required private key is unset.') + + return signature, method + + + + + +def verify_rsa_signature(signature, signature_method, public_key, data): + """ + + Determine whether the corresponding private key of 'public_key' produced + 'signature'. verify_signature() will use the public key, signature method, + and 'data' to complete the verification. + + >>> public, private = generate_rsa_public_and_private(2048) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature, method = create_rsa_signature(private, data) + >>> verify_rsa_signature(signature, method, public, data) + True + >>> verify_rsa_signature(signature, method, public, 'bad_data') + False + + + signature: + An RSASSA PSS signature as a string. This is the signature returned + by create_rsa_signature(). + + signature_method: + A string that indicates the signature algorithm used to generate + 'signature'. 'RSASSA-PSS' is currently supported. + + public_key: + The RSA public key, a string in PEM format. + + data: + Data object used by tuf.keys.create_signature() to generate + 'signature'. 'data' is needed here to verify the signature. + + + tuf.UnknownMethodError. Raised if the signing method used by + 'signature' is not one supported by tuf.keys.create_signature(). + + tuf.FormatError. Raised if 'signature', 'signature_method', or 'public_key' + is improperly formatted. + + + Crypto.Signature.PKCS1_PSS.verify() called to do the actual verification. + + + Boolean. True if the signature is valid, False otherwise. + """ + + # Does 'public_key' have the correct format? + # This check will ensure 'public_key' conforms to 'tuf.formats.PEMRSA_SCHEMA'. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(public_key) + + # Does 'signature_method' have the correct format? + tuf.formats.NAME_SCHEMA.check_match(signature_method) + + # Does 'signature' have the correct format? + tuf.formats.PYCRYPTOSIGNATURE_SCHEMA.check_match(signature) + + # Verify whether the private key of 'public_key' produced the signature. + # Before returning the Boolean result, ensure 'RSASSA-PSS' was used + # as the signing method. + signature = signature + method = signature_method + public = public_key + valid_signature = False + + # Verify the signature with PyCrypto if the signature method is valid, else + # raise 'tuf.UnknownMethodError'. + if method == 'RSASSA-PSS': + try: + rsa_key_object = Crypto.PublicKey.RSA.importKey(public_key) + pkcs1_pss_verifier = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) + sha256_object = Crypto.Hash.SHA256.new(data) + valid_signature = pkcs1_pss_verifier.verify(sha256_object, signature) + except (ValueError, IndexError, TypeError), e: + message = 'The RSA signature could not be verified.' + raise tuf.CryptoError(message) + else: + raise tuf.UnknownMethodError(method) + + return valid_signature + + + + + +def create_rsa_encrypted_pem(private_key, passphrase): + """ + + Return a string in PEM format, where the private part of the RSA key is + encrypted. The private part of the RSA key is encrypted by the Triple + Data Encryption Algorithm (3DES) and Cipher-block chaining (CBC) for the + mode of operation. Password-Based Key Derivation Function 1 (PBKF1) + MD5 + is used to strengthen 'passphrase'. + + https://en.wikipedia.org/wiki/Triple_DES + https://en.wikipedia.org/wiki/PBKDF2 + + >>> public, private = generate_rsa_public_and_private(2048) + >>> passphrase = 'secret' + >>> encrypted_pem = create_rsa_encrypted_pem(private, passphrase) + >>> tuf.formats.PEMRSA_SCHEMA.matches(encrypted_pem) + True + + + private_key: + The private key string in PEM format. + + passphrase: + The passphrase, or password, to encrypt the private part of the RSA + key. 'passphrase' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if an RSA key in encrypted PEM format cannot be created. + + TypeError, 'private_key' is unset. + + + PyCrypto's Crypto.PublicKey.RSA.exportKey() called to perform the actual + generation of the PEM-formatted output. + + + A string in PEM format, where the private RSA key is encrypted. + Conforms to 'tuf.formats.PEMRSA_SCHEMA'. + """ + + # Does 'private_key' have the correct format? + # This check will ensure 'private_key' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(private_key) + + # Does 'passphrase' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) + + # 'private_key' is in PEM format and unencrypted. The extracted key will be + # imported and converted to PyCrypto's RSA key object + # (i.e., Crypto.PublicKey.RSA). Use PyCrypto's exportKey method, with a + # passphrase specified, to create the string. PyCrypto uses PBKDF1+MD5 to + # strengthen 'passphrase', and 3DES with CBC mode for encryption. + # 'private_key' may still be a NULL string after the tuf.formats check. + if len(private_key): + try: + rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) + encrypted_pem = rsa_key_object.exportKey(format='PEM', + passphrase=passphrase) + except (ValueError, IndexError, TypeError), e: + message = 'An encrypted RSA key in PEM format could not be generated.' + raise tuf.CryptoError(message) + else: + raise TypeError('The required private key is unset.') + + + return encrypted_pem + + + + + +def create_rsa_public_and_private_from_encrypted_pem(encrypted_pem, passphrase): + """ + + Generate public and private RSA keys from an encrypted PEM. + The public and private keys returned conform to 'tuf.formats.PEMRSA_SCHEMA' + and have the form: + '-----BEGIN RSA PUBLIC KEY----- ...' + + or + + '-----BEGIN RSA PRIVATE KEY----- ...' + + The public and private keys are returned as strings in PEM format. + + The private key part of 'encrypted_pem' is encrypted. PyCrypto's importKey + method is used, where a passphrase is specified. PyCrypto uses PBKDF1+MD5 + to strengthen 'passphrase', and 3DES with CBC mode for encryption/decryption. + Alternatively, key data may be encrypted with AES-CTR-Mode and the passphrase + strengthened with PBKDF2+SHA256. See 'keystore.py'. + + >>> public, private = generate_rsa_public_and_private(2048) + >>> passphrase = 'secret' + >>> encrypted_pem = create_rsa_encrypted_pem(private, passphrase) + >>> returned_public, returned_private = \ + create_rsa_public_and_private_from_encrypted_pem(encrypted_pem, passphrase) + >>> tuf.formats.PEMRSA_SCHEMA.matches(returned_public) + True + >>> tuf.formats.PEMRSA_SCHEMA.matches(returned_private) + True + >>> public == returned_public + True + >>> private == returned_private + True + + + encrypted_pem: + A byte string in PEM format, where the private key is encrypted. It has + the form: + + '-----BEGIN RSA PRIVATE KEY-----\n + Proc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC ...' + + passphrase: + The passphrase, or password, to decrypt the private part of the RSA + key. 'passphrase' is not directly used as the encryption key, instead + it is used to derive a stronger symmetric key. + + + tuf.FormatError, if the arguments are improperly formatted. + + + PyCrypto's 'Crypto.PublicKey.RSA.importKey()' called to perform the actual + conversion from an encrypted RSA private key. + + + A (public, private) tuple containing the RSA keys in PEM format. + """ + + # Does 'encryped_pem' have the correct format? + # This check will ensure 'encrypted_pem' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(encrypted_pem) + + # Does 'passphrase' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) + + try: + rsa_key_object = Crypto.PublicKey.RSA.importKey(encrypted_pem, passphrase) + except (ValueError, IndexError, TypeError), e: + message = 'An RSA key object could not be generated from the encrypted '+\ + 'PEM string.' + # Raise 'tuf.CryptoError' instead of PyCrypto's exception to avoid + # revealing sensitive error, such as a decryption error due to an + # invalid passphrase. + raise tuf.CryptoError(message) + + # Extract the public and private halves of the RSA key and generate their + # PEM-formatted representations. The dictionary returned contains the + # private and public RSA keys in PEM format, as strings. + private = rsa_key_object.exportKey(format='PEM') + rsa_pubkey = rsa_key_object.publickey() + public = rsa_pubkey.exportKey(format='PEM') + + return public, private + + + + + +def encrypt_key(key_object, password): + """ + + Return a string containing 'key_object' in encrypted form. Encrypted strings + may be safely saved to a file. The corresponding decrypt_key() function can + be applied to the encrypted string to restore the original key object. + 'key_object' is a TUF key (e.g., RSAKEY_SCHEMA, ED25519KEY_SCHEMA). This + function calls the PyCrypto library to perform the encryption and derive + a suitable encryption key. + + Whereas an encrypted PEM file uses the Triple Data Encryption Algorithm + (3DES), the Cipher-block chaining (CBC) mode of operation, and the Password + Based Key Derivation Function 1 (PBKF1) + MD5 to strengthen 'password', + encrypted TUF keys use AES-256-CTR-Mode and passwords strengthened with + PBKDF2-HMAC-SHA256 (100K iterations by default, but may be overriden in + 'tuf.conf.PBKDF2_ITERATIONS' by the user). + + http://en.wikipedia.org/wiki/Advanced_Encryption_Standard + http://en.wikipedia.org/wiki/CTR_mode#Counter_.28CTR.29 + https://en.wikipedia.org/wiki/PBKDF2 + + >>> ed25519_key = {'keytype': 'ed25519', \ + 'keyid': \ + 'd62247f817883f593cf6c66a5a55292488d457bcf638ae03207dbbba9dbe457d', \ + 'keyval': {'public': \ + '74addb5ad544a4306b34741bc1175a3613a8d7dc69ff64724243efdec0e301ad', \ + 'private': \ + '1f26964cc8d4f7ee5f3c5da2fbb7ab35811169573ac367b860a537e47789f8c4'}} + >>> passphrase = 'secret' + >>> encrypted_key = encrypt_key(ed25519_key, passphrase) + >>> tuf.formats.ENCRYPTEDKEY_SCHEMA.matches(encrypted_key) + True + + + key_object: + The TUF key object that should contain the private portion of the ED25519 + key. + + password: + The password, or passphrase, to encrypt the private part of the RSA + key. 'password' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if any of the arguments are improperly formatted or + 'key_object' does not contain the private portion of the key. + + tuf.CryptoError, if an ED25519 key in encrypted TUF format cannot be + created. + + + PyCrypto cryptographic operations called to perform the actual encryption of + 'key_object'. 'password' used to derive a suitable encryption key. + + + An encrypted string in 'tuf.formats.ENCRYPTEDKEY_SCHEMA' format. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ANYKEY_SCHEMA.check_match(key_object) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Ensure the private portion of the key is included in 'key_object'. + if not key_object['keyval']['private']: + message = 'Key object does not contain a private part.' + raise tuf.FormatError(message) + + # Derive a key (i.e., an appropriate encryption key and not the + # user's password) from the given 'password'. Strengthen 'password' with + # PBKDF2-HMAC-SHA256 (100K iterations by default, but may be overriden in + # 'tuf.conf.PBKDF2_ITERATIONS' by the user). + salt, iterations, derived_key = _generate_derived_key(password) + + # Store the derived key info in a dictionary, the object expected + # by the non-public _encrypt() routine. + derived_key_information = {'salt': salt, 'iterations': iterations, + 'derived_key': derived_key} + + # Convert the key object to json string format and encrypt it with the + # derived key. + encrypted_key = _encrypt(json.dumps(key_object), derived_key_information) + + return encrypted_key + + + + + +def decrypt_key(encrypted_key, password): + """ + + + Return a string containing 'encrypted_key' in non-encrypted form. + The decrypt_key() function can be applied to the encrypted string to restore + the original key object, a TUF key (e.g., RSAKEY_SCHEMA, ED25519KEY_SCHEMA). + This function calls the appropriate cryptography module (e.g., + pycrypto_keys.py) to perform the decryption. + + Encrypted TUF keys use AES-256-CTR-Mode and passwords strengthened with + PBKDF2-HMAC-SHA256 (100K iterations be default, but may be overriden in + 'tuf.conf.py' by the user). + + http://en.wikipedia.org/wiki/Advanced_Encryption_Standard + http://en.wikipedia.org/wiki/CTR_mode#Counter_.28CTR.29 + https://en.wikipedia.org/wiki/PBKDF2 + + >>> ed25519_key = {'keytype': 'ed25519', \ + 'keyid': \ + 'd62247f817883f593cf6c66a5a55292488d457bcf638ae03207dbbba9dbe457d', \ + 'keyval': {'public': \ + '74addb5ad544a4306b34741bc1175a3613a8d7dc69ff64724243efdec0e301ad', \ + 'private': \ + '1f26964cc8d4f7ee5f3c5da2fbb7ab35811169573ac367b860a537e47789f8c4'}} + >>> passphrase = 'secret' + >>> encrypted_key = encrypt_key(ed25519_key, passphrase) + >>> decrypted_key = decrypt_key(encrypted_key, passphrase) + >>> tuf.formats.ED25519KEY_SCHEMA.matches(decrypted_key) + True + >>> decrypted_key == ed25519_key + True + + + encrypted_key: + An encrypted TUF key (additional data is also included, such as salt, + number of password iterations used for the derived encryption key, etc) + of the form 'tuf.formats.ENCRYPTEDKEY_SCHEMA'. 'encrypted_key' should + have been generated with encrypted_key(). + + password: + The password, or passphrase, to encrypt the private part of the RSA + key. 'password' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if a TUF key cannot be decrypted from 'encrypted_key'. + + + The PyCrypto library called to perform the actual decryption of + 'encrypted_key'. The key derivation data stored in 'encrypted_key' is used + to re-derive the encryption/decryption key. + + + The decrypted key object in 'tuf.formats.ANYKEY_SCHEMA' format. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ENCRYPTEDKEY_SCHEMA.check_match(encrypted_key) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Decrypt 'encrypted_key', using 'password' (and additional key derivation + # data like salts and password iterations) to re-derive the decryption key. + json_data = _decrypt(encrypted_key, password) + key_object = tuf.util.load_json_string(json_data) + + return key_object + + + + + +def _generate_derived_key(password, salt=None, iterations=None): + """ + Generate a derived key by feeding 'password' to the Password-Based Key + Derivation Function (PBKDF2). PyCrypto's PBKDF2 implementation is + currently used. 'salt' may be specified so that a previous derived key + may be regenerated. + """ + + if salt is None: + salt = Crypto.Random.new().read(_SALT_SIZE) + + if iterations is None: + iterations = _PBKDF2_ITERATIONS + + + def pseudorandom_function(password, salt): + """ + PyCrypto's PBKDF2() expects a callable function for its optional + 'prf' argument. 'prf' is set to HMAC-SHA1 (in PyCrypto's PBKDF2 function) + by default. 'pseudorandom_function' instead sets 'prf' to HMAC-SHA256. + """ + + return Crypto.Hash.HMAC.new(password, salt, Crypto.Hash.SHA256).digest() + + + # 'dkLen' is the desired key length. 'count' is the number of password + # iterations performed by PBKDF2. 'prf' is a pseudorandom function, which + # must be callable. + derived_key = Crypto.Protocol.KDF.PBKDF2(password, salt, + dkLen=_AES_KEY_SIZE, + count=iterations, + prf=pseudorandom_function) + + return salt, iterations, derived_key + + + + + +def _encrypt(key_data, derived_key_information): + """ + Encrypt 'key_data' using the Advanced Encryption Standard (AES-256) algorithm. + 'derived_key_information' should contain a key strengthened by PBKDF2. The + key size is 256 bits and AES's mode of operation is set to CTR (CounTeR Mode). + The HMAC of the ciphertext is generated to ensure the ciphertext has not been + modified. + + 'key_data' is the JSON string representation of the key. In the case + of RSA keys, this format would be 'tuf.formats.RSAKEY_SCHEMA': + {'keytype': 'rsa', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + 'derived_key_information' is a dictionary of the form: + {'salt': '...', + 'derived_key': '...', + 'iterations': '...'} + + 'tuf.CryptoError' raised if the encryption fails. + """ + + # Generate a random initialization vector (IV). The 'iv' is treated as the + # initial counter block to a stateful counter block function (i.e., + # PyCrypto's 'Crypto.Util.Counter'). The AES block cipher operates on 128-bit + # blocks, so generate a random 16-byte initialization block. PyCrypto expects + # the initial value of the stateful counter to be an integer. + # Follow the provably secure encrypt-then-MAC approach, which affords the + # ability to verify ciphertext without needing to decrypt it and preventing + # an attacker from feeding the block cipher malicious data. Modes like GCM + # provide both encryption and authentication, whereas CTR only provides + # encryption. + iv = Crypto.Random.new().read(16) + stateful_counter_128bit_blocks = Crypto.Util.Counter.new(128, + initial_value=long(iv.encode('hex'), 16)) + symmetric_key = derived_key_information['derived_key'] + aes_cipher = Crypto.Cipher.AES.new(symmetric_key, + Crypto.Cipher.AES.MODE_CTR, + counter=stateful_counter_128bit_blocks) + + # Use AES-256 to encrypt 'key_data'. The key size determines how many cycle + # repetitions are performed by AES, 14 cycles for 256-bit keys. + try: + ciphertext = aes_cipher.encrypt(key_data) + + # Raise generic exception message to avoid revealing sensitive information, + # such as invalid passwords, encryption keys, etc., that an attacker can use + # to his advantage. + except Exception, e: + message = 'The key data cannot be encrypted.' + raise tuf.CryptoError(message) + + # Generate the hmac of the ciphertext to ensure it has not been modified. + # The decryption routine may verify a ciphertext without having to perform + # a decryption operation. + salt = derived_key_information['salt'] + hmac_object = Crypto.Hash.HMAC.new(symmetric_key, ciphertext, + Crypto.Hash.SHA256) + hmac = hmac_object.hexdigest() + + # Store the number of PBKDF2 iterations used to derive the symmetric key so + # that the decryption routine can regenerate the symmetric key successfully. + # The pbkdf2 iterations are allowed to vary for the keys loaded and saved. + iterations = derived_key_information['iterations'] + + # Return the salt, iterations, hmac, initialization vector, and ciphertext + # as a single string. These five values are delimited by + # '_ENCRYPTION_DELIMITER' to make extraction easier. This delimiter is + # arbitrarily chosen and should not occur in the hexadecimal representations + # of the fields it is separating. + return binascii.hexlify(salt) + _ENCRYPTION_DELIMITER + \ + binascii.hexlify(str(iterations)) + _ENCRYPTION_DELIMITER + \ + binascii.hexlify(hmac) + _ENCRYPTION_DELIMITER + \ + binascii.hexlify(iv) + _ENCRYPTION_DELIMITER + \ + binascii.hexlify(ciphertext) + + + + + +def _decrypt(file_contents, password): + """ + The corresponding decryption routine for _encrypt(). + + 'tuf.CryptoError' raised if the decryption fails. + """ + + # Extract the salt, iterations, hmac, initialization vector, and ciphertext + # from 'file_contents'. These five values are delimited by + # '_ENCRYPTION_DELIMITER'. This delimiter is arbitrarily chosen and should + # not occur in the hexadecimal representations of the fields it is separating. + salt, iterations, hmac, iv, ciphertext = \ + file_contents.split(_ENCRYPTION_DELIMITER) + + # Ensure we have the expected raw data for the delimited cryptographic data. + salt = binascii.unhexlify(salt) + iterations = int(binascii.unhexlify(iterations)) + hmac = binascii.unhexlify(hmac) + iv = binascii.unhexlify(iv) + ciphertext = binascii.unhexlify(ciphertext) + + # Generate derived key from 'password'. The salt and iterations are specified + # so that the expected derived key is regenerated correctly. Discard the old + # "salt" and "iterations" values, as we only need the old derived key. + junk_old_salt, junk_old_iterations, derived_key = \ + _generate_derived_key(password, salt, iterations) + + # Verify the hmac to ensure the ciphertext is valid and has not been altered. + # See the encryption routine for why we use the encrypt-then-MAC approach. + generated_hmac_object = Crypto.Hash.HMAC.new(derived_key, ciphertext, + Crypto.Hash.SHA256) + generated_hmac = generated_hmac_object.hexdigest() + + if generated_hmac != hmac: + raise tuf.CryptoError('Decryption failed.') + + # The following decryption routine assumes 'ciphertext' was encrypted with + # AES-256. + stateful_counter_128bit_blocks = Crypto.Util.Counter.new(128, + initial_value=long(iv.encode('hex'), 16)) + aes_cipher = Crypto.Cipher.AES.new(derived_key, + Crypto.Cipher.AES.MODE_CTR, + counter=stateful_counter_128bit_blocks) + try: + key_plaintext = aes_cipher.decrypt(ciphertext) + + # Raise generic exception message to avoid revealing sensitive information, + # such as invalid passwords, encryption keys, etc., that an attacker can + # use to his advantage. + except Exception, e: + raise tuf.CryptoError('Decryption failed.') + + return key_plaintext + + + +if __name__ == '__main__': + # The interactive sessions of the documentation strings can + # be tested by running 'pycrypto_keys.py' as a standalone module: + # $ python pycrypto_keys.py + import doctest + doctest.testmod() diff --git a/tuf/repo/keystore.py b/tuf/repo/keystore.py index 8f5492e5..b16f7ca1 100755 --- a/tuf/repo/keystore.py +++ b/tuf/repo/keystore.py @@ -34,7 +34,6 @@ algorithm. User passwords are strengthened with PBKDF2, currently set to 100,000 passphrase iterations. The previous evpy implementation used 1,000 iterations. - """ import os @@ -68,7 +67,7 @@ # the AES algorithm to perform cipher block operations on them. import Crypto.Util.Counter -import tuf.rsa_key +import tuf.keys import tuf.util import tuf.conf @@ -104,6 +103,9 @@ # https://en.wikipedia.org/wiki/PBKDF2 _PBKDF2_ITERATIONS = tuf.conf.PBKDF2_ITERATIONS +# +_SUPPORTED_KEY_TYPES = ['rsa', 'ed25519'] + # A user password is read and a derived key generated. The derived key returned # by the key derivation function (PBKDF2) is saved in '_derived_keys', along # with the salt and iterations used, which has the form: @@ -159,7 +161,6 @@ def add_rsakey(rsakey_dict, password, keyid=None): None. - """ # Does 'rsakey_dict' have the correct format? @@ -235,7 +236,6 @@ def load_keystore_from_keyfiles(directory_name, keyids, passwords): A list containing the keyids of the loaded keys. - """ # Does 'directory_name' have the correct format? @@ -286,11 +286,11 @@ def load_keystore_from_keyfiles(directory_name, keyids, passwords): # Create the key based on its key type. RSA keys currently # supported. - if keydata['keytype'] == 'rsa': + if keydata['keytype'] in _SUPPORTED_KEY_TYPES: # 'keydata' is stored in KEY_SCHEMA format. Call - # create_from_metadata_format() to get the key in RSAKEY_SCHEMA + # format_metadata_to_key() to get the key in RSAKEY_SCHEMA # format, which is the format expected by 'add_rsakey()'. - rsa_key = tuf.rsa_key.create_from_metadata_format(keydata) + rsa_key = tuf.keys.format_metadata_to_key(keydata) # Ensure the keyid for 'rsa_key' is one of the keys specified in # 'keyids'. If not, do not load the key. @@ -343,7 +343,6 @@ def save_keystore_to_keyfiles(directory_name): None. - """ # Does 'directory_name' have the correct format? @@ -365,9 +364,11 @@ def save_keystore_to_keyfiles(directory_name): file_object = open(basefilename, 'w') # Determine the appropriate format to save the key based on its key type. - if key['keytype'] == 'rsa': + if key['keytype'] in _SUPPORTED_KEY_TYPES: + keytype = key['keytype'] + keyval = key['keyval'] key_metadata_format = \ - tuf.rsa_key.create_in_metadata_format(key['keyval'], private=True) + tuf.keys.format_keyval_to_metadata(keytype, keyval, private=True) else: logger.warn('The keystore has a key with an unrecognized key type.') continue @@ -402,7 +403,6 @@ def clear_keystore(): None. - """ _keystore.clear() @@ -442,7 +442,6 @@ def change_password(keyid, old_password, new_password): None. - """ # Does 'keyid' have the correct format? @@ -506,7 +505,6 @@ def get_key(keyid): The key belonging to 'keyid' (e.g., RSA key). - """ # Does 'keyid' have the correct format? @@ -530,7 +528,6 @@ def _generate_derived_key(password, salt=None, iterations=None): Derivation Function (PBKDF2). PyCrypto's PBKDF2 implementation is currently used. 'salt' may be specified so that a previous derived key may be regenerated. - """ if salt is None: @@ -584,7 +581,6 @@ def _encrypt(key_data, derived_key_information): 'iterations': '...'} 'tuf.CryptoError' raised if the encryption fails. - """ # Generate a random initialization vector (IV). The 'iv' is treated as the @@ -650,7 +646,6 @@ def _decrypt(file_contents, password): The corresponding decryption routine for _encrypt(). 'tuf.CryptoError' raised if the decryption fails. - """ # Extract the salt, iterations, hmac, initialization vector, and ciphertext diff --git a/tuf/repo/signercli.py b/tuf/repo/signercli.py index 5c36377b..09734b8b 100755 --- a/tuf/repo/signercli.py +++ b/tuf/repo/signercli.py @@ -46,7 +46,6 @@ See the parse_options() function for the full list of supported options. - """ import os @@ -92,7 +91,6 @@ def _get_password(prompt='Password: ', confirm=False): is True, the user is asked to enter the previously entered password once again. If they match, the password is returned to the caller. - """ while True: @@ -131,7 +129,6 @@ def _get_metadata_directory(): returned to the caller. 'tuf.FormatError' is raised if the directory is not properly formatted, and 'tuf.Error' if it does not exist. - """ metadata_directory = _prompt('\nEnter the metadata directory: ', str) @@ -151,7 +148,6 @@ def _list_keyids(keystore_directory, metadata_directory): It is assumed the directory arguments exist and have been validated by the caller. The keyids are listed without the '.key' extension, along with their associated roles. - """ # Determine the 'root.txt' filename. This metadata file is needed @@ -233,7 +229,6 @@ def _get_keyids(keystore_directory): key files are stored in encrypted form, the user is asked to enter the password that was used to encrypt the key file. - """ # The keyids list containing the keys loaded. @@ -288,7 +283,6 @@ def _get_all_config_keyids(config_filepath, keystore_directory): loaded_keyids = {'root': [1233d3d, 598djdks, ..], 'release': [sdfsd323, sdsd9090s, ..] ...} - """ # Save the 'load_keystore_from_keyfiles' function call. @@ -338,7 +332,6 @@ def _get_role_config_keyids(config_filepath, keystore_directory, role): tuf.Error, if the required keys could not be loaded. - """ # Save the 'load_keystore_from_keyfiles' function call. @@ -409,7 +402,6 @@ def _get_metadata_version(metadata_filename): 'metadata_filename' does not exist, return a version value of 1. Raise 'tuf.RepositoryError' if 'metadata_filename' cannot be read or validated. - """ # If 'metadata_filename' does not exist on the repository, this means @@ -442,7 +434,6 @@ def _get_metadata_expiration(): tuf.RepositoryError, if the entered expiration date is invalid. - """ message = '\nCurrent time: '+tuf.formats.format_time(time.time())+'.\n'+\ @@ -487,7 +478,6 @@ def change_password(keystore_directory): None. - """ # Save the 'load_keystore_from_keyfiles' function call. @@ -563,7 +553,6 @@ def generate_rsa_key(keystore_directory): None. - """ # Save a reference to the generate_and_save_rsa_key() function. @@ -612,7 +601,6 @@ def list_signing_keys(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -654,7 +642,6 @@ def dump_key(keystore_directory): None. - """ # Save the 'load_keystore_from_keyfiles' function call. @@ -704,8 +691,11 @@ def dump_key(keystore_directory): # Retrieve the key metadata according to the keytype. if key['keytype'] == 'rsa': - key_metadata = tuf.rsa_key.create_in_metadata_format(key['keyval'], - private=show_private) + keytype = key['keytype'] + keyval = key['keyval'] + key_metadata = tuf.keys.format_keyval_to_metadata(keytype, keyval, + #key_metadata = tuf.keys.create_in_metadata_format(keytype, keyval, + private=show_private) else: message = 'The keystore contains an invalid key type.' raise tuf.RepositoryError(message) @@ -737,7 +727,6 @@ def make_root_metadata(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -804,7 +793,6 @@ def make_targets_metadata(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -886,7 +874,6 @@ def make_release_metadata(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -951,7 +938,6 @@ def make_timestamp_metadata(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -1017,7 +1003,6 @@ def sign_metadata_file(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -1084,7 +1069,6 @@ def make_delegation(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -1154,7 +1138,6 @@ def _load_parent_role(metadata_directory, keystore_directory, targets_roles): list of known targets roles and asked to enter the parent role to load. Ensure the parent role is loaded properly and return a string containing the parent role's full rolename and a list of keyids belonging to the parent. - """ # 'load_key' is a reference to the 'load_keystore_from_keyfiles function'. @@ -1210,7 +1193,6 @@ def _get_delegated_role(keystore_directory, metadata_directory): a list of keyids available in the keystore and asked to enter the keyid belonging to the delegated role. Return a string containing the delegated role's full rolename and its keyids. - """ # Retrieve the delegated rolename from the user (e.g., 'role1'). @@ -1240,7 +1222,6 @@ def _make_delegated_metadata(metadata_directory, delegated_targets, role. Determine the target files from the paths in 'delegated_targets' and the other information needed to generate the targets metadata file for delegated_role'. Return the delegated paths to the caller. - """ repository_directory, junk = os.path.split(metadata_directory) @@ -1336,7 +1317,6 @@ def _update_parent_metadata(metadata_directory, delegated_role, metadata file is updated with the key and role information belonging to the newly added delegated role. Finally, the metadata file is signed and written to the metadata directory. - """ # According to the specification, the 'paths' and 'path_hash_prefixes' @@ -1376,8 +1356,10 @@ def _update_parent_metadata(metadata_directory, delegated_role, # Retrieve the key belonging to 'delegated_keyid' from the keystore. role_key = tuf.repo.keystore.get_key(delegated_keyid) if role_key['keytype'] == 'rsa': + keytype = role_key['keytype'] keyval = role_key['keyval'] - keys[delegated_keyid] = tuf.rsa_key.create_in_metadata_format(keyval) + keys[delegated_keyid] = tuf.keys.format_keyval_to_metadata(keytype, keyval) + #keys[delegated_keyid] = tuf.keys.create_in_metadata_format(keytype, keyval) else: message = 'Invalid keytype encountered: '+delegated_keyid+'\n' raise tuf.RepositoryError(message) @@ -1450,7 +1432,6 @@ def process_option(options): None. - """ # Determine which option was chosen and call its corresponding @@ -1506,7 +1487,6 @@ def parse_options(): The options object returned by the parser's parse_args() method. - """ usage = 'usage: %prog [option] ' diff --git a/tuf/repo/signerlib.py b/tuf/repo/signerlib.py index 3f44ea5e..c2a71f63 100755 --- a/tuf/repo/signerlib.py +++ b/tuf/repo/signerlib.py @@ -16,18 +16,18 @@ These functions contain code that can extract or create needed repository data, such as the extraction of role and keyid information from a config file, and the generation of actual metadata content. - """ import gzip import os import ConfigParser import logging +import time import tuf import tuf.formats import tuf.hash -import tuf.rsa_key +import tuf.keys import tuf.repo.keystore import tuf.sig import tuf.util @@ -81,7 +81,6 @@ def read_config_file(filename): A dictionary containing the data loaded from the configuration file. - """ # Does 'filename' have the correct format? @@ -151,7 +150,6 @@ def get_metadata_file_info(filename): A dictionary conformant to 'tuf.formats.FILEINFO_SCHEMA'. This dictionary contains the length, hashes, and custom data about the 'filename' metadata file. - """ # Does 'filename' have the correct format? @@ -204,7 +202,6 @@ def get_metadata_filenames(metadata_directory=None): A dictionary containing the expected filenames of the top-level metadata files, such as 'root.txt' and 'release.txt'. - """ if metadata_directory is None: @@ -255,7 +252,6 @@ def generate_root_metadata(config_filepath, version): A root 'signable' object conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Does 'config_filepath' have the correct format? @@ -290,8 +286,10 @@ def generate_root_metadata(config_filepath, version): keyid = key['keyid'] # This appears to be a new keyid. Let's generate the key for it. if keyid not in keydict: - if key['keytype'] == 'rsa': - keydict[keyid] = tuf.rsa_key.create_in_metadata_format(key['keyval']) + if key['keytype'] in ['rsa', 'ed25519']: + keytype = key['keytype'] + keyval = key['keyval'] + keydict[keyid] = tuf.keys.format_keyval_to_metadata(keytype, keyval) # This is not a recognized key. Raise an exception. else: raise tuf.Error('Unsupported keytype: '+keyid) @@ -312,7 +310,8 @@ def generate_root_metadata(config_filepath, version): 3600 * 24 * expiration['days']) # Generate the root metadata object. - root_metadata = tuf.formats.RootFile.make_metadata(version, expiration_seconds, + expiration_date = tuf.formats.format_time(time.time()+expiration_seconds) + root_metadata = tuf.formats.RootFile.make_metadata(version, expiration_date, keydict, roledict) # Note: make_signable() returns the following dictionary: @@ -364,7 +363,6 @@ def generate_targets_metadata(repository_directory, target_files, version, A targets 'signable' object, conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Do the arguments have the correct format. @@ -433,7 +431,6 @@ def generate_release_metadata(metadata_directory, version, expiration_date): The release 'signable' object, conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Does 'metadata_directory' have the correct format? @@ -510,7 +507,6 @@ def generate_timestamp_metadata(release_filename, version, A timestamp 'signable' object, conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Do the arguments have the correct format? @@ -575,7 +571,6 @@ def write_metadata_file(metadata, filename, compression=None): The path to the written metadata file. - """ # Are the arguments properly formatted? @@ -645,7 +640,6 @@ def read_metadata_file(filename): The metadata object. - """ return tuf.util.load_json_file(filename) @@ -684,7 +678,6 @@ def sign_metadata(metadata, keyids, filename): A signable object conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Does 'keyids' and 'filename' have the correct format? @@ -767,7 +760,6 @@ def generate_and_save_rsa_key(keystore_directory, password, 'keyid': keyid, 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - """ # Are the arguments correctly formatted? @@ -778,7 +770,7 @@ def generate_and_save_rsa_key(keystore_directory, password, keystore_directory = check_directory(keystore_directory) # tuf.FormatError or tuf.CryptoError raised. - rsakey = tuf.rsa_key.generate(bits) + rsakey = tuf.keys.generate_rsa_key(bits) logger.info('Generated a new key: '+rsakey['keyid']) @@ -820,7 +812,6 @@ def check_directory(directory): The normalized absolutized path of 'directory'. - """ # Does 'directory' have the correct format? @@ -868,7 +859,6 @@ def get_target_keyids(metadata_directory): A dictionary containing the role information extracted from the metadata. Ex: {'targets':[keyid1, ...], 'targets/role1':[keyid], ...} - """ # Does 'metadata_directory' have the correct format? @@ -964,7 +954,6 @@ def build_config_file(config_file_directory, timeout, role_info): The normalized absolutized path of the saved configuration file. - """ # Do the arguments have the correct format? @@ -1054,7 +1043,6 @@ def build_root_file(config_filepath, root_keyids, metadata_directory, version): The path for the written root metadata file. - """ # Do the arguments have the correct format? @@ -1116,7 +1104,6 @@ def build_targets_file(target_paths, targets_keyids, metadata_directory, The path for the written targets metadata file. - """ # Do the arguments have the correct format? @@ -1207,7 +1194,6 @@ def build_release_file(release_keyids, metadata_directory, The path for the written release metadata file. - """ # Do the arguments have the correct format? @@ -1282,7 +1268,6 @@ def build_timestamp_file(timestamp_keyids, metadata_directory, The path for the written timestamp metadata file. - """ # Do the arguments have the correct format? @@ -1370,7 +1355,6 @@ def build_delegated_role_file(delegated_targets_directory, delegated_keyids, The path for the written targets metadata file. - """ # Do the arguments have the correct format? @@ -1432,7 +1416,6 @@ def find_delegated_role(roles, delegated_role): None, if the role with the given name does not exist, or its unique index in the list of roles. - """ # Check argument types. @@ -1487,7 +1470,6 @@ def accept_any_file(full_target_path): True. - """ return True @@ -1524,7 +1506,6 @@ def get_targets(files_directory, recursive_walk=False, followlinks=True, A list of absolute paths to target files in the given files_directory. - """ targets = [] @@ -1546,8 +1527,3 @@ def get_targets(files_directory, recursive_walk=False, followlinks=True, del dirnames[:] return targets - - - - - diff --git a/tuf/roledb.py b/tuf/roledb.py index 4f114dfc..68dae917 100755 --- a/tuf/roledb.py +++ b/tuf/roledb.py @@ -23,13 +23,20 @@ The role database is a dictionary conformant to 'tuf.formats.ROLEDICT_SCHEMA' and has the form: + {'rolename': {'keyids': ['34345df32093bd12...'], 'threshold': 1 - 'paths': ['path/to/role.txt']}} - + 'signatures': ['abcd3452...'], + 'paths': ['path/to/role.txt'], + 'path_hash_prefixes': ['ab34df13'], + 'delegations': {'keys': {}, 'roles': {}}} + + The 'name', 'paths', 'path_hash_prefixes', and 'delegations' dict keys are + optional. """ import logging +import copy import tuf import tuf.formats @@ -66,7 +73,6 @@ def create_roledb_from_root_metadata(root_metadata): None. - """ # Does 'root_metadata' have the correct object format? @@ -81,6 +87,17 @@ def create_roledb_from_root_metadata(root_metadata): # Iterate through the roles found in 'root_metadata' # and add them to '_roledb_dict'. Duplicates are avoided. for rolename, roleinfo in root_metadata['roles'].items(): + if rolename == 'root': + roleinfo['version'] = root_metadata['version'] + roleinfo['expires'] = root_metadata['expires'] + + roleinfo['signatures'] = [] + roleinfo['signing_keyids'] = [] + roleinfo['compressions'] = [''] + roleinfo['partial_loaded'] = False + if rolename.startswith('targets'): + roleinfo['delegations'] = {'keys': {}, 'roles': []} + try: add_role(rolename, roleinfo) # tuf.Error raised if the parent role of 'rolename' does not exist. @@ -104,10 +121,17 @@ def add_role(rolename, roleinfo, require_parent=True): roleinfo: An object representing the role associated with 'rolename', conformant to - ROLE_SCHEMA. 'roleinfo' has the form: + ROLEDB_SCHEMA. 'roleinfo' has the form: {'keyids': ['34345df32093bd12...'], - 'threshold': 1} + 'threshold': 1, + 'signatures': ['ab23dfc32'] + 'paths': ['path/to/target1', 'path/to/target2', ...], + 'path_hash_prefixes': ['a324fcd...', ...], + 'delegations': {'keys': } + The 'paths', 'path_hash_prefixes', and 'delegations' dict keys are + optional. + The 'target' role has an additional 'paths' key. Its value is a list of strings representing the path of the target file(s). @@ -128,7 +152,6 @@ def add_role(rolename, roleinfo, require_parent=True): None. - """ # Does 'rolename' have the correct object format? @@ -137,10 +160,10 @@ def add_role(rolename, roleinfo, require_parent=True): tuf.formats.ROLENAME_SCHEMA.check_match(rolename) # Does 'roleinfo' have the correct object format? - tuf.formats.ROLE_SCHEMA.check_match(roleinfo) + tuf.formats.ROLEDB_SCHEMA.check_match(roleinfo) # Does 'require_parent' have the correct format? - tuf.formats.TOGGLE_SCHEMA.check_match(require_parent) + tuf.formats.BOOLEAN_SCHEMA.check_match(require_parent) # Raises tuf.InvalidNameError. _validate_rolename(rolename) @@ -157,12 +180,71 @@ def add_role(rolename, roleinfo, require_parent=True): if parent_role not in _roledb_dict: raise tuf.Error('Parent role does not exist: '+parent_role) - _roledb_dict[rolename] = roleinfo + _roledb_dict[rolename] = copy.deepcopy(roleinfo) +def update_roleinfo(rolename, roleinfo): + """ + + + + rolename: + An object representing the role's name, conformant to 'ROLENAME_SCHEMA' + (e.g., 'root', 'release', 'timestamp'). + + roleinfo: + An object representing the role associated with 'rolename', conformant to + ROLEDB_SCHEMA. 'roleinfo' has the form: + {'name': 'role_name', + 'keyids': ['34345df32093bd12...'], + 'threshold': 1, + 'paths': ['path/to/target1', 'path/to/target2', ...], + 'path_hash_prefixes': ['a324fcd...', ...]} + + The 'name', 'paths', and 'path_hash_prefixes' dict keys are optional. + + The 'target' role has an additional 'paths' key. Its value is a list of + strings representing the path of the target file(s). + + + tuf.FormatError, if 'rolename' or 'roleinfo' does not have the correct + object format. + + tuf.UnknownRoleError, if 'rolename' cannot be found in the role database. + + tuf.InvalidNameError, if 'rolename' is improperly formatted. + + + The role database is modified. + + + None. + """ + + # Does 'rolename' have the correct object format? + # This check will ensure 'rolename' has the appropriate number of objects + # and object types, and that all dict keys are properly named. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + + # Does 'roleinfo' have the correct object format? + tuf.formats.ROLEDB_SCHEMA.check_match(roleinfo) + + # Raises tuf.InvalidNameError. + _validate_rolename(rolename) + + if rolename not in _roledb_dict: + raise tuf.UnknownRoleError('Role does not exist: '+rolename) + + _roledb_dict[rolename] = copy.deepcopy(roleinfo) + + + + + + def get_parent_rolename(rolename): """ @@ -187,7 +269,6 @@ def get_parent_rolename(rolename): A string representing the name of the parent role. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -228,7 +309,6 @@ def get_all_parent_roles(rolename): A list containing all the parent roles. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -279,7 +359,6 @@ def role_exists(rolename): Boolean. True if 'rolename' is found in the role database, False otherwise. - """ # Raise tuf.FormatError, tuf.InvalidNameError. @@ -318,7 +397,6 @@ def remove_role(rolename): None. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -358,7 +436,6 @@ def remove_delegated_roles(rolename): None. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -390,7 +467,6 @@ def get_rolenames(): A list of rolenames. - """ return _roledb_dict.keys() @@ -399,6 +475,47 @@ def get_rolenames(): +def get_roleinfo(rolename): + """ + + Return the roleinfo of 'rolename'. + + {'keyids': ['34345df32093bd12...'], + 'threshold': 1, + 'signatures': ['ab453bdf...', ...], + 'paths': ['path/to/target1', 'path/to/target2', ...], + 'path_hash_prefixes': ['a324fcd...', ...], + 'delegations': {'keys': {}, 'roles': []}} + + The 'signatures', 'paths', 'path_hash_prefixes', and 'delegations' dict keys + are optional. + + + rolename: + An object representing the role's name, conformant to 'ROLENAME_SCHEMA' + (e.g., 'root', 'release', 'timestamp'). + + + tuf.FormatError, if 'rolename' is improperly formatted. + + tuf.UnknownRoleError, if 'rolename' does not exist. + + + None. + + + The roleinfo of 'rolename'. + """ + + # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. + _check_rolename(rolename) + + return copy.deepcopy(_roledb_dict[rolename]) + + + + + def get_role_keyids(rolename): """ @@ -426,7 +543,6 @@ def get_role_keyids(rolename): A list of keyids. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -462,7 +578,6 @@ def get_role_threshold(rolename): A threshold integer value. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -498,7 +613,6 @@ def get_role_paths(rolename): A list of paths. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -540,8 +654,7 @@ def get_delegated_rolenames(rolename): A list of rolenames. Note that the rolenames are *NOT* sorted by order of - delegation! - + delegation. """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -578,7 +691,6 @@ def clear_roledb(): None. - """ _roledb_dict.clear() @@ -593,7 +705,6 @@ def _check_rolename(rolename): 'tuf.formats.ROLENAME_SCHEMA', tuf.UnknownRoleError if 'rolename' is not found in the role database, or tuf.InvalidNameError if 'rolename' is not formatted correctly. - """ # Does 'rolename' have the correct object format? @@ -616,7 +727,6 @@ def _validate_rolename(rolename): Raise tuf.InvalidNameError if 'rolename' is not formatted correctly. It is assumed 'rolename' has been checked against 'ROLENAME_SCHEMA' prior to calling this function. - """ if rolename == '': diff --git a/tuf/rsa_key.py b/tuf/rsa_key.py deleted file mode 100755 index 8d737391..00000000 --- a/tuf/rsa_key.py +++ /dev/null @@ -1,664 +0,0 @@ -""" - - rsa_key.py - - - Vladimir Diaz - - - March 9, 2012. Based on a previous version of this module by Geremy Condra. - - - See LICENSE for licensing information. - - - The goal of this module is to support public-key cryptography using the RSA - algorithm. The RSA-related functions provided include generate(), - create_signature(), and verify_signature(). The create_encrypted_pem() and - create_from_encrypted_pem() functions are optional, and may be used save a - generated RSA key to a file. The 'PyCrypto' package used by 'rsa_key.py' - generates the actual RSA keys and the functions listed above can be viewed - as an easy-to-use public interface. Additional functions contained here - include create_in_metadata_format() and create_from_metadata_format(). These - last two functions produce or use RSA keys compatible with the key structures - listed in TUF Metadata files. The generate() function returns a dictionary - containing all the information needed of RSA keys, such as public and private= - keys, keyIDs, and an idenfier. create_signature() and verify_signature() are - supplemental functions used for generating RSA signatures and verifying them. - https://en.wikipedia.org/wiki/RSA_(algorithm) - - Key IDs are used as identifiers for keys (e.g., RSA key). They are the - hexadecimal representation of the hash of key object (specifically, the key - object containing only the public key). Review 'rsa_key.py' and the - '_get_keyid()' function to see precisely how keyids are generated. One may - get the keyid of a key object by simply accessing the dictionary's 'keyid' - key (i.e., rsakey['keyid']). - - """ - - -# Required for hexadecimal conversions. Signatures are hexlified. -import binascii - -# Crypto.PublicKey (i.e., PyCrypto public-key cryptography) provides algorithms -# such as Digital Signature Algorithm (DSA) and the ElGamal encryption system. -# 'Crypto.PublicKey.RSA' is needed here to generate, sign, and verify RSA keys. -import Crypto.PublicKey.RSA - -# PyCrypto requires 'Crypto.Hash' hash objects to generate PKCS#1 PSS -# signatures (i.e., Crypto.Signature.PKCS1_PSS). -import Crypto.Hash.SHA256 - -# RSA's probabilistic signature scheme with appendix (RSASSA-PSS). -# PKCS#1 v1.5 is provided for compatability with existing applications, but -# RSASSA-PSS is encouraged for newer applications. RSASSA-PSS generates -# a random salt to ensure the signature generated is probabilistic rather than -# deterministic, like PKCS#1 v1.5. -# http://en.wikipedia.org/wiki/RSA-PSS#Schemes -# https://tools.ietf.org/html/rfc3447#section-8.1 -import Crypto.Signature.PKCS1_PSS - -import tuf - -# Digest objects needed to generate hashes. -import tuf.hash - -# Perform object format-checking. -import tuf.formats - - -_KEY_ID_HASH_ALGORITHM = 'sha256' - -# Recommended RSA key sizes: -# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 -# According to the document above, revised May 6, 2003, RSA keys of -# size 3072 provide security through 2031 and beyond. -_DEFAULT_RSA_KEY_BITS = 3072 - - -def generate(bits=_DEFAULT_RSA_KEY_BITS): - """ - - Generate public and private RSA keys, with modulus length 'bits'. - In addition, a keyid used as an identifier for RSA keys is generated. - The object returned conforms to 'tuf.formats.RSAKEY_SCHEMA' and as the form: - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - Although the crytography library called sets a 1024-bit minimum key size, - generate() enforces a minimum key size of 2048 bits. If 'bits' is - unspecified, a 3072-bit RSA key is generated, which is the key size - recommended by TUF. - - - bits: - The key size, or key length, of the RSA key. 'bits' must be 2048, or - greater, and a multiple of 256. - - - ValueError, if an exception occurs after calling the RSA key generation - routine. 'bits' must be a multiple of 256. The 'ValueError' exception is - raised by the key generation function of the cryptography library called. - - tuf.FormatError, if 'bits' does not contain the correct format. - - - The RSA keys are generated by calling PyCrypto's - Crypto.PublicKey.RSA.generate(). - - - A dictionary containing the RSA keys and other identifying information. - - """ - - - # Does 'bits' have the correct format? - # This check will ensure 'bits' conforms to 'tuf.formats.RSAKEYBITS_SCHEMA'. - # 'bits' must be an integer object, with a minimum value of 2048. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) - - # Begin building the RSA key dictionary. - rsakey_dict = {} - keytype = 'rsa' - - # Generate the public and private RSA keys. The PyCrypto module performs - # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 - # or not a multiple of 256, although a 2048-bit minimum is enforced by - # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). - rsa_key_object = Crypto.PublicKey.RSA.generate(bits) - - # Extract the public & private halves of the RSA key and generate their - # PEM-formatted representations. The dictionary returned contains the - # private and public RSA keys in PEM format, as strings. - private_key_pem = rsa_key_object.exportKey(format='PEM') - rsa_pubkey = rsa_key_object.publickey() - public_key_pem = rsa_pubkey.exportKey(format='PEM') - - # Generate the keyid for the RSA key. 'key_value' corresponds to the - # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key - # information is not included in the generation of the 'keyid' identifier. - key_value = {'public': public_key_pem, - 'private': ''} - keyid = _get_keyid(key_value) - - # Build the 'rsakey_dict' dictionary. Update 'key_value' with the RSA - # private key prior to adding 'key_value' to 'rsakey_dict'. - key_value['private'] = private_key_pem - - rsakey_dict['keytype'] = keytype - rsakey_dict['keyid'] = keyid - rsakey_dict['keyval'] = key_value - - return rsakey_dict - - - - - -def create_in_metadata_format(key_value, private=False): - """ - - Return a dictionary conformant to 'tuf.formats.KEY_SCHEMA'. - If 'private' is True, include the private key. The dictionary - returned has the form: - {'keytype': 'rsa', - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - or if 'private' is False: - - {'keytype': 'rsa', - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': ''}} - - The private and public keys are in PEM format. - - RSA keys are stored in Metadata files (e.g., root.txt) in the format - returned by this function. - - - key_value: - A dictionary containing a private and public RSA key. - 'key_value' is of the form: - - {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}}, - conformat to 'tuf.formats.KEYVAL_SCHEMA'. - - private: - Indicates if the private key should be included in the - returned dictionary. - - - tuf.FormatError, if 'key_value' does not conform to - 'tuf.formats.KEYVAL_SCHEMA'. - - - None. - - - An 'KEY_SCHEMA' dictionary. - - """ - - - # Does 'key_value' have the correct format? - # This check will ensure 'key_value' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.KEYVAL_SCHEMA.check_match(key_value) - - if private is True and key_value['private']: - return {'keytype': 'rsa', 'keyval': key_value} - else: - public_key_value = {'public': key_value['public'], 'private': ''} - return {'keytype': 'rsa', 'keyval': public_key_value} - - - - - -def create_from_metadata_format(key_metadata): - """ - - Construct an RSA key dictionary (i.e., tuf.formats.RSAKEY_SCHEMA) - from 'key_metadata'. The dict returned by this function has the exact - format as the dict returned by generate(). It is of the form: - - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - RSA key dictionaries in RSAKEY_SCHEMA format should be used by - modules storing a collection of keys, such as a keydb and keystore. - RSA keys as stored in metadata files use a different format, so this - function should be called if an RSA key is extracted from one of these - metadata files and needs converting. Generate() creates an entirely - new key and returns it in the format appropriate for 'keydb.py' and - 'keystore.py'. - - - key_metadata: - The RSA key dictionary as stored in Metadata files, conforming to - 'tuf.formats.KEY_SCHEMA'. It has the form: - - {'keytype': '...', - 'keyval': {'public': '...', - 'private': '...'}} - - - tuf.FormatError, if 'key_metadata' does not conform to - 'tuf.formats.KEY_SCHEMA'. - - - None. - - - A dictionary containing the RSA keys and other identifying information. - - """ - - - # Does 'key_metadata' have the correct format? - # This check will ensure 'key_metadata' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.KEY_SCHEMA.check_match(key_metadata) - - # Construct the dictionary to be returned. - rsakey_dict = {} - keytype = 'rsa' - key_value = key_metadata['keyval'] - - # Convert 'key_value' to 'tuf.formats.KEY_SCHEMA' and generate its hash - # The hash is in hexdigest form. - keyid = _get_keyid(key_value) - - # We now have all the required key values. Build 'rsakey_dict'. - rsakey_dict['keytype'] = keytype - rsakey_dict['keyid'] = keyid - rsakey_dict['keyval'] = key_value - - return rsakey_dict - - - - - -def _get_keyid(key_value): - """Return the keyid for 'key_value'.""" - - # 'keyid' will be generated from an object conformant to KEY_SCHEMA, - # which is the format Metadata files (e.g., root.txt) store keys. - # 'create_in_metadata_format()' returns the object needed by _get_keyid(). - rsakey_meta = create_in_metadata_format(key_value, private=False) - - # Convert the RSA key to JSON Canonical format suitable for adding - # to digest objects. - rsakey_update_data = tuf.formats.encode_canonical(rsakey_meta) - - # Create a digest object and call update(), using the JSON - # canonical format of 'rskey_meta' as the update data. - digest_object = tuf.hash.digest(_KEY_ID_HASH_ALGORITHM) - digest_object.update(rsakey_update_data) - - # 'keyid' becomes the hexadecimal representation of the hash. - keyid = digest_object.hexdigest() - - return keyid - - - - - -def create_signature(rsakey_dict, data): - """ - - Return a signature dictionary of the form: - {'keyid': keyid, - 'method': 'PyCrypto-PKCS#1 PPS', - 'sig': sig}. - - The signing process will use the private key - rsakey_dict['keyval']['private'] and 'data' to generate the signature. - - RFC3447 - RSASSA-PSS - http://www.ietf.org/rfc/rfc3447.txt - - - rsakey_dict: - A dictionary containing the RSA keys and other identifying information. - 'rsakey_dict' has the form: - - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - data: - Data object used by create_signature() to generate the signature. - - - TypeError, if a private key is not defined for 'rsakey_dict'. - - tuf.FormatError, if an incorrect format is found for the - 'rsakey_dict' object. - - - PyCrypto's 'Crypto.Signature.PKCS1_PSS' called to perform the actual - signing. - - - A signature dictionary conformat to 'tuf.format.SIGNATURE_SCHEMA'. - - """ - - - # Does 'rsakey_dict' have the correct format? - # This check will ensure 'rsakey_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) - - # Signing the 'data' object requires a private key. - # The 'PyCrypto-PKCS#1 PSS' (i.e., PyCrypto module) signing method is the - # only method currently supported. - signature = {} - private_key = rsakey_dict['keyval']['private'] - keyid = rsakey_dict['keyid'] - method = 'PyCrypto-PKCS#1 PSS' - sig = None - - # Verify the signature, but only if the private key has been set. The private - # key is a NULL string if unset. Although it may be clearer to explicit check - # that 'private_key' is not '', we can/should check for a value and not - # compare identities with the 'is' keyword. - if len(private_key): - # Calculate the SHA256 hash of 'data' and generate the hash's PKCS1-PSS - # signature. - try: - rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) - sha256_object = Crypto.Hash.SHA256.new(data) - pkcs1_pss_signer = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) - sig = pkcs1_pss_signer.sign(sha256_object) - except (ValueError, IndexError, TypeError), e: - message = 'An RSA signature could not be generated.' - raise tuf.CryptoError(message) - else: - raise TypeError('The required private key is not defined for "rsakey_dict".') - - # Build the signature dictionary to be returned. - # The hexadecimal representation of 'sig' is stored in the signature. - signature['keyid'] = keyid - signature['method'] = method - signature['sig'] = binascii.hexlify(sig) - - return signature - - - - - -def verify_signature(rsakey_dict, signature, data): - """ - - Determine whether the private key belonging to 'rsakey_dict' produced - 'signature'. verify_signature() will use the public key found in - 'rsakey_dict', the 'method' and 'sig' objects contained in 'signature', - and 'data' to complete the verification. Type-checking performed on both - 'rsakey_dict' and 'signature'. - - - rsakey_dict: - A dictionary containing the RSA keys and other identifying information. - 'rsakey_dict' has the form: - - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - signature: - The signature dictionary produced by tuf.rsa_key.create_signature(). - 'signature' has the form: - {'keyid': keyid, 'method': 'method', 'sig': sig}. Conformant to - 'tuf.formats.SIGNATURE_SCHEMA'. - - data: - Data object used by tuf.rsa_key.create_signature() to generate - 'signature'. 'data' is needed here to verify the signature. - - - tuf.UnknownMethodError. Raised if the signing method used by - 'signature' is not one supported by tuf.rsa_key.create_signature(). - - tuf.FormatError. Raised if either 'rsakey_dict' - or 'signature' do not match their respective tuf.formats schema. - 'rsakey_dict' must conform to 'tuf.formats.RSAKEY_SCHEMA'. - 'signature' must conform to 'tuf.formats.SIGNATURE_SCHEMA'. - - - Crypto.Signature.PKCS1_PSS.verify() called to do the actual verification. - - - Boolean. True if the signature is valid, False otherwise. - - """ - - - # Does 'rsakey_dict' have the correct format? - # This check will ensure 'rsakey_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) - - # Does 'signature' have the correct format? - tuf.formats.SIGNATURE_SCHEMA.check_match(signature) - - # Using the public key belonging to 'rsakey_dict' - # (i.e., rsakey_dict['keyval']['public']), verify whether 'signature' - # was produced by rsakey_dict's corresponding private key - # rsakey_dict['keyval']['private']. Before returning the Boolean result, - # ensure 'PyCrypto-PKCS#1 PSS' was used as the signing method. - method = signature['method'] - sig = signature['sig'] - public_key = rsakey_dict['keyval']['public'] - valid_signature = False - - if method == 'PyCrypto-PKCS#1 PSS': - try: - rsa_key_object = Crypto.PublicKey.RSA.importKey(public_key) - pkcs1_pss_verifier = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) - sha256_object = Crypto.Hash.SHA256.new(data) - - # The metadata stores signatures in hex. Unhexlify and verify the - # signature. - signature = binascii.unhexlify(sig) - valid_signature = pkcs1_pss_verifier.verify(sha256_object, signature) - except (ValueError, IndexError, TypeError), e: - message = 'The RSA signature could not be verified.' - raise tuf.CryptoError(message) - else: - raise tuf.UnknownMethodError(method) - - return valid_signature - - - - - -def create_encrypted_pem(rsakey_dict, passphrase): - """ - - Return a string in PEM format, where the private part of the RSA key is - encrypted. The private part of the RSA key is encrypted by the Triple - Data Encryption Algorithm (3DES) and Cipher-block chaining (CBC) for the - mode of operation. Password-Based Key Derivation Function 1 (PBKF1) + MD5 - is used to strengthen 'passphrase'. - - https://en.wikipedia.org/wiki/Triple_DES - https://en.wikipedia.org/wiki/PBKDF2 - - - rsakey_dict: - A dictionary containing the RSA keys and other identifying information. - 'rsakey_dict' has the form: - - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - passphrase: - The passphrase, or password, to encrypt the private part of the RSA - key. 'passphrase' is not used directly as the encryption key, a stronger - encryption key is derived from it. - - - TypeError, if a private key is not defined for 'rsakey_dict'. - - tuf.FormatError, if an incorrect format is found for 'rsakey_dict'. - - - PyCrypto's Crypto.PublicKey.RSA.exportKey() called to perform the actual - generation of the PEM-formatted output. - - - A string in PEM format, where the private RSA key is encrypted. - - """ - - # Does 'rsakey_dict' have the correct format? - # This check will ensure 'rsakey_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) - - # Does 'signature' have the correct format? - tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) - - # Extract the private key from 'rsakey_dict', which is stored in PEM format - # and unencrypted. The extracted key will be imported and converted to - # PyCrypto's RSA key object (i.e., Crypto.PublicKey.RSA).Use PyCrypto's - # exportKey method, with a passphrase specified, to create the string. - # PyCrypto uses PBKDF1+MD5 to strengthen 'passphrase', and 3DES with CBC mode - # for encryption. - private_key = rsakey_dict['keyval']['private'] - try: - rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) - rsakey_pem_encrypted = rsa_key_object.exportKey(format='PEM', - passphrase=passphrase) - except (ValueError, IndexError, TypeError), e: - message = 'An encrypted RSA key in PEM format could not be generated.' - raise tuf.CryptoError(message) - - return rsakey_pem_encrypted - - - - - -def create_from_encrypted_pem(encrypted_pem, passphrase): - """ - - Return an RSA key in 'tuf.formats.RSAKEY_SCHEMA' format, which has the - form: - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The RSAKEY_SCHEMA object is generated from a byte string in PEM format, - where the private part of the RSA key is encrypted. PyCrypto's importKey - method is used, where a passphrase is specified. PyCrypto uses PBKDF1+MD5 - to strengthen 'passphrase', and 3DES with CBC mode for encryption/decryption. - Alternatively, key data may be encrypted with AES-CTR-Mode and the passphrase - strengthened with PBKDF2+SHA256. See 'keystore.py'. - - - encrypted_pem: - A byte string in PEM format, where the private key is encrypted. It has - the form: - - '-----BEGIN RSA PRIVATE KEY-----\n - Proc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC ...' - - passphrase: - The passphrase, or password, to decrypt the private part of the RSA - key. 'passphrase' is not directly used as the encryption key, instead - it is used to derive a stronger symmetric key. - - - TypeError, if a private key is not defined for 'rsakey_dict'. - - tuf.FormatError, if an incorrect format is found for the - 'rsakey_dict' object. - - - PyCrypto's 'Crypto.PublicKey.RSA.importKey()' called to perform the actual - conversion from an encrypted RSA private key. - - - A dictionary in 'tuf.formats.RSAKEY_SCHEMA' format. - - """ - - # Does 'encryped_pem' have the correct format? - # This check will ensure 'encrypted_pem' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.PEMRSA_SCHEMA.check_match(encrypted_pem) - - # Does 'passphrase' have the correct format? - tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) - - keytype = 'rsa' - rsakey_dict = {} - - try: - rsa_key_object = Crypto.PublicKey.RSA.importKey(encrypted_pem, passphrase) - except (ValueError, IndexError, TypeError), e: - message = 'An RSA key object could not be generated from the encrypted '+\ - 'PEM string.' - # Raise 'tuf.CryptoError' instead of PyCrypto's exception to avoid - # revealing sensitive error, such as a decryption error due to an - # invalid passphrase. - raise tuf.CryptoError(message) - - # Extract the public & private halves of the RSA key and generate their - # PEM-formatted representations. The dictionary returned contains the - # private and public RSA keys in PEM format, as strings. - private_key_pem = rsa_key_object.exportKey(format='PEM') - rsa_pubkey = rsa_key_object.publickey() - public_key_pem = rsa_pubkey.exportKey(format='PEM') - - # Generate the keyid for the RSA key. 'key_value' corresponds to the - # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key - # information is not included in the generation of the 'keyid' identifier. - key_value = {'public': public_key_pem, - 'private': ''} - keyid = _get_keyid(key_value) - - # Build the 'rsakey_dict' dictionary. Update 'key_value' with the RSA - # private key prior to adding 'key_value' to 'rsakey_dict'. - key_value['private'] = private_key_pem - - rsakey_dict['keytype'] = keytype - rsakey_dict['keyid'] = keyid - rsakey_dict['keyval'] = key_value - - return rsakey_dict diff --git a/tuf/schema.py b/tuf/schema.py index acd50b63..278e1899 100755 --- a/tuf/schema.py +++ b/tuf/schema.py @@ -38,7 +38,6 @@ criteria. See 'tuf.formats.py' and the rest of this module for extensive examples. Anything related to the checking of TUF objects and their formats can be found in 'formats.py'. - """ @@ -55,7 +54,6 @@ class Schema: that are encodable in JSON. 'Schema' is the base class for the other classes defined in this module. All derived classes should implement check_match(). - """ def matches(self, object): @@ -64,7 +62,6 @@ def matches(self, object): Return True if 'object' matches this schema, False if it doesn't. If the caller wishes to signal an error on a failed match, check_match() should be called, which will raise a 'tuf.FormatError' exception. - """ try: @@ -82,7 +79,6 @@ def check_match(self, object): implement check_match(). If 'object' matches the schema, check_match() should simply return. If 'object' does not match the schema, 'tuf.FormatError' should be raised. - """ raise NotImplementedError() @@ -110,7 +106,6 @@ class Any(Schema): True >>> schema.matches([1, 'list']) True - """ def __init__(self): @@ -143,7 +138,6 @@ class String(Schema): True >>> schema.matches('Not hi') False - """ def __init__(self, string): @@ -187,7 +181,6 @@ class AnyString(Schema): True >>> schema.matches({}) False - """ def __init__(self): @@ -202,6 +195,48 @@ def check_match(self, object): +class LengthString(Schema): + """ + + Matches any string of a specified length. The argument object + must be a string. At instantiation, the string length is set + and any future comparisons are checked against this internal + string value length. + + Supported methods include + matches(): returns a Boolean result. + check_match(): raises 'tuf.FormatError' on a mismatch. + + + + >>> schema = LengthString(5) + >>> schema.matches('Hello') + True + >>> schema.matches('Hi') + False + """ + + def __init__(self, length): + if isinstance(length, bool) or not isinstance(length, (int, long)): + # We need to check for bool as a special case, since bool + # is for historical reasons a subtype of int. + raise tuf.FormatError('Got '+repr(length)+' instead of an integer.') + + self._string_length = length + + + def check_match(self, object): + if not isinstance(object, basestring): + raise tuf.FormatError('Expected a string but got '+repr(object)) + + if len(object) != self._string_length: + raise tuf.FormatError('Expected a string of length '+ + repr(self._string_length)) + + + + + class OneOf(Schema): """ @@ -229,7 +264,6 @@ class OneOf(Schema): True >>> schema.matches(['Hi']) False - """ def __init__(self, alternatives): @@ -275,7 +309,6 @@ class AllOf(Schema): False >>> schema.matches('a') True - """ def __init__(self, required_schemas): @@ -314,7 +347,6 @@ class Boolean(Schema): True >>> schema.matches(11) False - """ def __init__(self): @@ -367,7 +399,6 @@ class ListOf(Schema): True >>> schema.matches([3]*11) False - """ def __init__(self, schema, min_count=0, max_count=sys.maxint, list_name='list'): @@ -380,7 +411,6 @@ def __init__(self, schema, min_count=0, max_count=sys.maxint, list_name='list'): min_count: The minimum number of sub-schema in 'schema'. max_count: The maximum number of sub-schema in 'schema'. list_name: A string identifier for the ListOf object. - """ if not isinstance(schema, Schema): @@ -443,7 +473,6 @@ class Integer(Schema): True >>> Integer(lo=10, hi=30).matches(5) False - """ def __init__(self, lo= -sys.maxint, hi=sys.maxint): @@ -454,7 +483,6 @@ def __init__(self, lo= -sys.maxint, hi=sys.maxint): lo: The minimum value the int object argument can be. hi: The maximum value the int object argument can be. - """ self._lo = lo @@ -468,7 +496,7 @@ def check_match(self, object): raise tuf.FormatError('Got '+repr(object)+' instead of an integer.') elif not (self._lo <= object <= self._hi): - int_range = '['+repr(self._lo)+','+repr(self._hi)+'].' + int_range = '['+repr(self._lo)+', '+repr(self._hi)+'].' raise tuf.FormatError(repr(object)+' not in range '+int_range) @@ -502,7 +530,6 @@ class DictOf(Schema): False >>> schema.matches({'a': ['x', 'y'], 'e' : ['', ''], 'd' : ['a', 'b']}) False - """ def __init__(self, key_schema, value_schema): @@ -513,7 +540,6 @@ def __init__(self, key_schema, value_schema): key_schema: The dictionary's key. value_schema: The dictionary's value. - """ if not isinstance(key_schema, Schema): @@ -564,7 +590,6 @@ class Optional(Schema): False >>> schema.matches({'k1': 'X'}) True - """ def __init__(self, schema): @@ -604,7 +629,6 @@ class Object(Schema): False >>> schema.matches({'a':'ZYYY'}) False - """ def __init__(self, object_name='object', **required): @@ -616,7 +640,6 @@ def __init__(self, object_name='object', **required): object_name: A string identifier for the object argument. A variable number of keyword arguments is accepted. - """ # Ensure valid arguments. @@ -713,7 +736,6 @@ class Struct(Schema): False >>> schema.matches(['X', 3, 'A']) False - """ def __init__(self, sub_schemas, optional_schemas=[], allow_more=False, @@ -727,7 +749,6 @@ def __init__(self, sub_schemas, optional_schemas=[], allow_more=False, optional_schemas: The optional list of schemas. allow_more: Specifies that an optional list of types is allowed. struct_name: A string identifier for the Struct object. - """ # Ensure each item of the list contains the expected object type. @@ -792,7 +813,6 @@ class RegularExpression(Schema): False >>> schema.matches([33, 'Hello']) False - """ def __init__(self, pattern=None, modifiers=0, re_object=None, re_name=None): @@ -805,7 +825,6 @@ def __init__(self, pattern=None, modifiers=0, re_object=None, re_name=None): modifiers: Flags to use when compiling the pattern. re_object: A compiled regular expression object. re_name: Identifier for the regular expression object. - """ if not isinstance(pattern, basestring): diff --git a/tuf/sig.py b/tuf/sig.py index ce5b9f56..d8ef7f0c 100755 --- a/tuf/sig.py +++ b/tuf/sig.py @@ -33,7 +33,6 @@ that will determine if a role still has a sufficient number of valid keys. If a caller needs to update the signatures of a 'signable' object, there is also a function for that. - """ import tuf @@ -76,7 +75,6 @@ def get_signature_status(signable, role=None): A dictionary representing the status of the signatures in 'signable'. Conformant to tuf.formats.SIGNATURESTATUS_SCHEMA. - """ # Does 'signable' have the correct format? @@ -106,9 +104,6 @@ def get_signature_status(signable, role=None): signed = signable['signed'] signatures = signable['signatures'] - # 'signed' needed in canonical JSON format. - data = tuf.formats.encode_canonical(signed) - # Iterate through the signatures and enumerate the signature_status fields. # (i.e., good_sigs, bad_sigs, etc.). for signature in signatures: @@ -125,7 +120,7 @@ def get_signature_status(signable, role=None): # Identify key using an unknown key signing method. try: - valid_sig = tuf.rsa_key.verify_signature(key, signature, data) + valid_sig = tuf.keys.verify_signature(key, signature, signed) except tuf.UnknownMethodError: unknown_method_sigs.append(keyid) continue @@ -201,7 +196,6 @@ def verify(signable, role): Boolean. True if the number of good signatures >= the role's threshold, False otherwise. - """ # Retrieve the signature status. tuf.sig.get_signature_status() raises @@ -243,10 +237,8 @@ def may_need_new_keys(signature_status): Boolean. - """ - # Does 'signature_status' have the correct format? # This check will ensure 'signature_status' has the appropriate number # of objects and object types, and that all dict keys are properly named. @@ -281,11 +273,11 @@ def generate_rsa_signature(signed, rsakey_dict): signed: - The data used by 'tuf.rsa_key.create_signature()' to generate signatures. + The data used by 'tuf.keys.create_signature()' to generate signatures. It is stored in the 'signed' field of 'signable'. rsakey_dict: - The RSA key, a tuf.formats.RSAKEY_SCHEMA dictionary. + The RSA key, a 'tuf.formats.RSAKEY_SCHEMA' dictionary. Used here to produce 'keyid', 'method', and 'sig'. @@ -300,7 +292,6 @@ def generate_rsa_signature(signed, rsakey_dict): Signature dictionary conformant to tuf.formats.SIGNATURE_SCHEMA. Has the form: {'keyid': keyid, 'method': 'evp', 'sig': sig} - """ # We need 'signed' in canonical JSON format to generate @@ -309,6 +300,6 @@ def generate_rsa_signature(signed, rsakey_dict): # Generate the RSA signature. # Raises tuf.FormatError and TypeError. - signature = tuf.rsa_key.create_signature(rsakey_dict, signed) + signature = tuf.keys.create_signature(rsakey_dict, signed) return signature diff --git a/tuf/tests/repository_setup.py b/tuf/tests/repository_setup.py index 066ca61e..778621d8 100644 --- a/tuf/tests/repository_setup.py +++ b/tuf/tests/repository_setup.py @@ -14,7 +14,6 @@ To provide a quick repository structure to be used in conjunction with test modules like test_updater.py for instance. - """ import os @@ -24,7 +23,6 @@ import tempfile import tuf.formats -import tuf.rsa_key as rsa_key import tuf.repo.keystore as keystore import tuf.repo.signerlib as signerlib import tuf.repo.signercli as signercli @@ -276,7 +274,6 @@ def create_repositories(): A dictionary of all repositories, with the following keys: (main_repository, client_repository, server_repository) - """ # Ensure the keyids for the required roles are loaded. Role keyids are diff --git a/tuf/tests/unittest_toolbox.py b/tuf/tests/unittest_toolbox.py index ff23af49..785fffcb 100644 --- a/tuf/tests/unittest_toolbox.py +++ b/tuf/tests/unittest_toolbox.py @@ -15,7 +15,6 @@ Provides an array of various methods for unit testing. Use it instead of actual unittest module. This module builds on unittest module. Specifically, Modified_TestCase is a derived class from unittest.TestCase. - """ import os @@ -27,7 +26,7 @@ import string import ConfigParser -import tuf.rsa_key as rsa_key +import tuf.keys import tuf.repo.keystore as keystore # Modify the number of iterations (from the higher default count) so the unit @@ -100,7 +99,6 @@ def setUp(): random_string(length=7): Generate a 'length' long string of random characters. - """ # List of all top level roles. @@ -222,6 +220,7 @@ def make_temp_config_file(self, suffix='', directory=None, config_dict={}, expir dictionary in it using ConfigParser. It then returns the temp file path, dictionary tuple. """ + config = ConfigParser.RawConfigParser() if not config_dict: # Using the fact that empty sequences are false. @@ -288,7 +287,6 @@ def make_temp_directory_with_data_files(self, _current_dir=None, Returns: ('/tmp/tmp_dir_Test_random/', [targets/tmp_random1.txt, targets/tmp_random2.txt, targets/more_targets/tmp_random3.txt]) - """ if not _current_dir: @@ -385,7 +383,7 @@ def generate_rsakey(): 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} """ - rsakey = rsa_key.generate() + rsakey = tuf.keys.generate_rsa_key() keyid = rsakey['keyid'] Modified_TestCase.rsa_keyids.append(keyid) password = Modified_TestCase.random_string() diff --git a/tuf/time_ed25519.py b/tuf/time_ed25519.py index ded684a1..89e8e726 100755 --- a/tuf/time_ed25519.py +++ b/tuf/time_ed25519.py @@ -3,30 +3,36 @@ import timeit import tuf -from tuf.ed25519_key import * +from tuf.ed25519_keys import * use_pynacl = False if '--pynacl' in sys.argv: use_pynacl = True -print('Time generate()') -print(timeit.timeit('generate(use_pynacl)', - setup='from __main__ import generate, use_pynacl', +print('Time generate_public_and_private()') +print(timeit.timeit('generate_public_and_private(use_pynacl)', + setup='from __main__ import generate_public_and_private, \ + use_pynacl', number=1)) print('\nTime create_signature()') -print(timeit.timeit('create_signature(ed25519_key, data, use_pynacl)', - setup='from __main__ import generate, create_signature, \ +print(timeit.timeit('create_signature(public, private, data, use_pynacl)', + setup='from __main__ import generate_public_and_private, \ + create_signature, \ use_pynacl; \ - ed25519_key = generate(use_pynacl);\ + public, private = \ + generate_public_and_private(use_pynacl); \ data = "The quick brown fox jumps over the lazy dog"', number=1)) print('\nTime verify_signature()') -print(timeit.timeit('verify_signature(ed25519_key, signature, data, use_pynacl)', - setup='from __main__ import generate, create_signature, \ - verify_signature, use_pynacl;\ - ed25519_key = generate(use_pynacl);\ +print(timeit.timeit('verify_signature(public, method, signature, data, use_pynacl)', + setup='from __main__ import generate_public_and_private, \ + create_signature, \ + verify_signature, use_pynacl; \ + public, private = \ + generate_public_and_private(use_pynacl); \ data = "The quick brown fox jumps over the lazy dog";\ - signature = create_signature(ed25519_key, data, use_pynacl)', + signature, method = \ + create_signature(public, private, data, use_pynacl)', number=1)) diff --git a/tuf/util.py b/tuf/util.py index 1460e838..dc376cd4 100755 --- a/tuf/util.py +++ b/tuf/util.py @@ -340,7 +340,7 @@ def close_temp_file(self): -def get_file_details(filepath): +def get_file_details(filepath, hash_algorithms=['sha256']): """ To get file's length and hash information. The hash is computed using the @@ -351,6 +351,8 @@ def get_file_details(filepath): filepath: Absolute file path of a file. + hash_algorithms: + tuf.FormatError: If hash of the file does not match HASHDICT_SCHEMA. @@ -363,6 +365,10 @@ def get_file_details(filepath): # Making sure that the format of 'filepath' is a path string. # 'tuf.FormatError' is raised on incorrect format. tuf.formats.PATH_SCHEMA.check_match(filepath) + tuf.formats.HASHALGORITHMS_SCHEMA.check_match(hash_algorithms) + + # The returned file hashes of 'filepath'. + file_hashes = {} # Does the path exists? if not os.path.exists(filepath): @@ -373,14 +379,15 @@ def get_file_details(filepath): file_length = os.path.getsize(filepath) # Obtaining hash of the file. - digest_object = tuf.hash.digest_filename(filepath, algorithm='sha256') - file_hash = {'sha256' : digest_object.hexdigest()} + for algorithm in hash_algorithms: + digest_object = tuf.hash.digest_filename(filepath, algorithm) + file_hashes.update({algorithm: digest_object.hexdigest()}) # Performing a format check to ensure 'file_hash' corresponds HASHDICT_SCHEMA. # Raise 'tuf.FormatError' if there is a mismatch. - tuf.formats.HASHDICT_SCHEMA.check_match(file_hash) + tuf.formats.HASHDICT_SCHEMA.check_match(file_hashes) - return file_length, file_hash + return file_length, file_hashes @@ -547,7 +554,7 @@ def load_json_file(filepath): Deserialize a JSON object from a file containing the object. - data: + filepath: Absolute path of JSON file. @@ -579,5 +586,3 @@ def load_json_file(filepath): return json.load(fileobject) finally: fileobject.close() - -