fleet/tools/tuf

Releasing updates to Fleet's TUF repository

The releaser.sh script automates the building and releasing of fleetd and osquery updates on Fleet's TUF repository.

• The script was developed and tested on macOS Intel.
• It currently supports pushing new fleetd and osqueryd versions.
• By storing credentials encrypted in a USB flash drive and storing their decryption passphrase on 1Password we are enforcing a form of 2FA.
• First, new releases are pushed to https://updates-staging.fleetdm.com. After verification (smoke test), releases are pushed to production, by syncing https://updates-staging.fleetdm.com to https://updates.fleetdm.com.

1. Push to staging:

graph LR;
subgraph Workstation;
    releaser[releaser.sh];
    1password("<div><img src='../../website/assets/images/articles/tales-from-fleet-security-securing-1password-cover-1600x900@2x.jpg' /></div>1Password");
    usb("<div><img src='../../website/assets/images/articles/config-less-fleetd-agent-deployment-1600x900@2x.png' /></div>USB flash drive");
    repository[(./repository)];
end;
r2_staging("<div><img src='../../website/assets/images/app-icon-cloudflare-warp-60x60@2x.png' /></div>r2://updates-staging");
github("<div><img src='../../website/assets/images/github-mark-white-24x24@2x.png' /></div>Github Action\n(signing and notarization)");

usb--(1) copy encrypted signing keys-->releaser;
1password--(2) get passphrases to decrypt encrypted signing keys-->releaser;
1password--(3) get Github API token-->releaser;
r2_staging--(4) pull TUF repository-->releaser;
releaser--(5) build components (new updates)\n(osqueryd, orbit, Fleet Desktop)-->github;
github--(6) download built components-->releaser;
releaser--(7) push updates and signed metadata-->r2_staging;

2. Push to production:

graph LR;
r2_staging("<div><img src='../../website/assets/images/app-icon-cloudflare-warp-60x60@2x.png' /></div>r2://updates-staging");
r2("<div><img src='../../website/assets/images/app-icon-cloudflare-warp-60x60@2x.png' /></div>r2://updates (production)");

r2_staging--server side sync-->r2;

Permissions and configuration

Following is the checklist for all credentials and configuration needed to run the script.

Dependencies

• make
• git
• 1Password 8 application.
• Install and configure 1Password's op cli to connect to the application: https://developer.1password.com/docs/cli/get-started/
• rclone
• fleetctl: Either built from source or installed by npm.
• tuf: Download the release from https://github.com/theupdateframework/go-tuf/releases/tag/v0.5.2 and place the tuf executable in /usr/local/bin/tuf. You will need to make an exception in "Privacy & Security" because the executable is not signed.
• gh: brew install gh.

1Password

You need to create three passphrases on your private 1Password vault for encrypting the signing keys (more on signing keys below). Create three private "passwords" with the following names: UPDATES TARGETS, UPDATES SNAPSHOT and UPDATES TIMESTAMP. The resulting credentials will have the following "path" within 1Password (these paths will be provided to the releaser.sh script)

Private/UPDATES TARGETS/password
Private/UPDATES SNAPSHOT/password
Private/UPDATES TIMESTAMP/password

R2

The following is required to be able to run rclone cli commands that can read/write to R2 buckets.

1. You will need to request the infrastructure team to provide you with access key and secret key that can read+write to the R2 buckets.
2. Set the following settings on ~/.config/rclone/rclone.conf:

[r2]
type = s3
provider = Cloudflare
access_key_id = <ACCESS_KEY>
secret_access_key = <SECRET_ACCESS_KEY>
region = auto
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
acl = private

ACCESS_KEY, SECRET_ACCESS_KEY, and ACCOUNT_ID are provided by the infrastructure team.

To smoke test (read) access you can run:

rclone copy --verbose --checksum r2://updates-staging/timestamp.json .
cat timestamp.json

TUF signing keys

You can skip this step if you already have authorized keys to sign and publish updates.

To release updates to our TUF repository you need the root role (ask in Slack who has such root role) to sign your signing keys.

1. First, run the following script

tuf gen-key targets && echo
tuf gen-key snapshot && echo
tuf gen-key timestamp && echo

2. Store the '$TUF_DIRECTORY/keys' folder (that contains the encrypted keys) on a USB flash drive that you will ONLY use for releasing fleetd updates.
3. Share '$TUF_DIRECTORY/staged/root.json' with Fleet member with the 'root' role, who will sign with its root key and push it to the remote repository.
4. The human with the root role will run the following commands to sign the provided staged/root.json:

tuf sign
tuf snapshot
tuf timestamp
tuf commit

And push the newly signed root.json to the remote repository.

Encrypted keys in USB

For releasing fleetd you need to plug in the USB that contains encrypted signing keys. In this guide we assume the USB device will be mounted in /Volumes/FLEET-UPD/ and it ONLY contains a keys/ directory.

Github

Personal access token

A personal access token is required to download artifacts from Github Actions using the Github API.

1. Create a personal access token at https://github.com/settings/tokens. Check repo and workflow permissions for the token.
2. Store the token on 1Password as a "password" with name "Github Token" The resulting credential will have the following "path" within 1Password (this path will be provided to the script)

Private/Github Token/password

Github session

You need to log in to your Github account using the cli (gh).

gh auth login

It will be used to create a PR which is used to update the changelog and trigger the Github actions to build components.

Releasing

Following are samples of the script execution to release components to edge and stable.

When releasing fleetd you need to checkout the branch (e.g. main) you want to release.

NOTE: When releasing fleetd: If there are only orbit changes on a release we still have to release the desktop component with its version string bumped (even if there are no changes in it). This is due to the fact that we want users to see the new version in the tray icon, e.g. "Fleet Desktop v1.21.0". Technical debt: We could improve this process to reduce the complexity of releasing fleetd when there are no Fleet Desktop changes.

Releasing to edge

Releasing fleetd 1.23.0 to edge

1. Push to staging:

TUF_DIRECTORY=/Users/foobar/updates-staging.fleetdm.com \
COMPONENT=fleetd \
ACTION=release-to-edge \
VERSION=1.23.0 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-UPD/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TIMESTAMP/password" \
GITHUB_USERNAME=foobar \
GITHUB_TOKEN_1PASSWORD_PATH="Private/Github Token/password" \
./tools/tuf/releaser.sh

2. Smoke test release on staging.
3. Push to production:

ACTION=release-to-production \
COMPONENT=fleetd \
VERSION=1.23.0 \
./tools/tuf/releaser.sh

4. Create PR with changelog against main (it may fail with conflicts when using patch branches):

ACTION=create-fleetd-release-pr \
VERSION=1.23.0 \
./tools/tuf/releaser.sh

4. Smoke test release on production.

Releasing osquery 5.12.1 to edge

1. Push to staging:

TUF_DIRECTORY=/Users/foobar/updates-staging.fleetdm.com \
COMPONENT=osqueryd \
ACTION=release-to-edge \
VERSION=5.12.1 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-UPD/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TIMESTAMP/password" \
GITHUB_USERNAME=foobar \
GITHUB_TOKEN_1PASSWORD_PATH="Private/Github Token/password" \
./tools/tuf/releaser.sh

2. Smoke test release on staging.
3. Push to production:

ACTION=release-to-production ./tools/tuf/releaser.sh

4. Smoke test release on production.

Promoting from edge to stable

Promoting fleetd 1.23.0 from edge to stable

1. Push to staging:

TUF_DIRECTORY=/Users/foobar/updates-staging.fleetdm.com \
COMPONENT=fleetd \
ACTION=promote-edge-to-stable \
VERSION=1.23.0 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-UPD/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TIMESTAMP/password" \
./tools/tuf/releaser.sh

2. Smoke test release on staging.
3. Push to production:

ACTION=release-to-production ./tools/tuf/releaser.sh

4. Smoke test release on production.

Promoting osqueryd 5.12.1 from edge to stable

1. Push to staging:

TUF_DIRECTORY=/Users/foobar/updates-staging.fleetdm.com \
COMPONENT=osqueryd \
ACTION=promote-edge-to-stable \
VERSION=5.12.1 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-UPD/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TIMESTAMP/password" \
./tools/tuf/releaser.sh

2. Smoke test release on staging.
3. Push to production:

ACTION=release-to-production ./tools/tuf/releaser.sh

4. Smoke test release on production.
5. Update osquery schema and flags:

ACTION=update-osquery-schema \
VERSION=5.19.0 \
./tools/tuf/releaser.sh

Releasing swiftDialog to stable

macOS only component

1. Download swiftDialog from the desired run of https://github.com/fleetdm/fleet/actions/workflows/generate-swift-dialog-targets.yml.
2. Extract the downloaded .zip to /path/to/swiftDialog.app.tar.gz
3. Push to staging:

TUF_DIRECTORY=/Users/foobar/updates-staging.fleetdm.com \
ACTION=release-swiftDialog-to-stable \
VERSION=2.5.6 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-UPD/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TIMESTAMP/password" \
SWIFT_DIALOG_PATH=/path/to/swiftDialog.app.tar.gz \
./tools/tuf/releaser.sh

4. Push to production:

ACTION=release-to-production ./tools/tuf/releaser.sh

Releasing nudge to stable

releaser.sh doesn't support nudge yet. macOS only component

The nudge executable can be generated from a macOS host by running:

make nudge-app-tar-gz version=1.1.10.81462 out-path=.

fleetctl updates add --target /path/to/macos/nudge.app.tar.gz --platform macos --name nudge --version 1.1.10.81462 -t edge

Releasing Escrow Buddy to stable

releaser.sh doesn't support Escrow Buddy yet. macOS only component

The Escrow Buddy pkg installer can be generated by running:

make escrow-buddy-pkg version=1.0.0 out-path=.

fleetctl updates add --target /path/to/escrowBuddy.pkg --platform macos --name escrowBuddy --version 1.0.0 -t stable

Updating timestamp

1. Push to staging:

TUF_DIRECTORY=/Users/foobar/updates-staging.fleetdm.com \
ACTION=update-timestamp \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-UPD/keys \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TIMESTAMP/password" \
./tools/tuf/releaser.sh

2. Push to production:

ACTION=release-to-production ./tools/tuf/releaser.sh

Doing a patch release of fleetd

Patch releases follow the same process as releasing a minor version, except instead of checking out the main branch of Fleet locally, you check out a patch branch of Fleet, e.g.:

git checkout rc-minor-fleetd-v1.41.1

As always, the VERSION env var used when running releaser.sh should match the version of the fleetd release, e.g.

TUF_DIRECTORY=/Users/foobar/updates-staging.fleetdm.com \
COMPONENT=fleetd \
ACTION=release-to-edge \
VERSION=1.41.1 # <-- note the patch version \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-UPD/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/UPDATES TIMESTAMP/password" \
GITHUB_USERNAME=foobar \
GITHUB_TOKEN_1PASSWORD_PATH="Private/Github Token/password" \
./tools/tuf/releaser.sh

See https://github.com/fleetdm/fleet/blob/main/orbit/TUF.md to find the latest released version.

After following the rest of the "Releasing to edge" steps above, publish your release using the instructions in "Promoting from edge to stable" above as you would for a minor release, again remembering to set the VERSION accordingly.

Testing and improving the script

• You can specify GIT_REPOSITORY_DIRECTORY to set a separate path for the Fleet repository (it uses the current by default). This is sometimes necessary if the tooling the script uses is not present in the branch we are trying to release from.

git clone git@github.com:fleetdm/fleet.git <SOME_DIRECTORY>
GIT_REPOSITORY_DIRECTORY=<SOME_DIRECTORY>

• If the PR and orbit tag were already generated but you need to run the script again you can set SKIP_PR_AND_TAG_PUSH=1 to skip that part.

TODOs to improve releaser.sh

• Support releasing nudge and swiftDialog.

Troubleshooting

Invalid timestamp.json version

The following issue was solved by resigning the timestamp metadata fleetctl updates timestamp (executed three times to increase the version to 4175)

2022-08-23T13:44:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174"
2022-08-23T13:59:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174"