Elgato_dark/fleet

Fork 0

mirror of https://github.com/fleetdm/fleet synced 2026-04-21 13:37:30 +00:00

Lucas Manuel Rodriguez d9a7d4e5e8

Add missing setup step for root (#22699 )

2024-10-07 10:15:16 -05:00

14 KiB

Raw Blame History

Releasing updates to Fleet's TUF repository

The releaser.sh script automates the building and releasing of fleetd and osquery updates on Fleet's TUF repository.

The script was developed and tested on macOS Intel.

It currently supports pushing new fleetd and osqueryd versions.

By storing credentials encrypted in a USB flash drive and storing their decryption passphrase on 1Password we are enforcing a form of 2FA.

graph LR;
    subgraph Workstation;
        releaser[releaser.sh];
        1password("<div><img src='../../website/assets/images/articles/tales-from-fleet-security-securing-1password-cover-1600x900@2x.jpg' /></div>1Password");
        usb("<div><img src='../../website/assets/images/articles/config-less-fleetd-agent-deployment-1600x900@2x.png' /></div>USB flash drive");
        repository[(./repository)];
    end;
    s3("<div><img src='../../website/assets/images/icon-aws-60x36@2x.png' /></div>s3://fleet-tuf-repo");
    github("<div><img src='../../website/assets/images/github-mark-white-24x24@2x.png' /></div>Github Action\n(signing and notarization)");
    
    usb--(1) copy encrypted signing keys-->releaser;
    1password--(2) get passphrases to decrypt encrypted signing keys-->releaser;
    1password--(3) get Github API token-->releaser;
    s3--(4) pull TUF repository-->releaser;
    releaser--(5) build components (new updates)\n(osqueryd, orbit, Fleet Desktop)-->github;
    github--(6) download built components-->releaser;
    releaser--(7) push updates and signed metadata-->s3;

Permissions and configuration

Following is the checklist for all credentials and configuration needed to run the script.

Dependencies

make
git
1Password 8 application.
Install and configure 1Password's op cli to connect to the application: https://developer.1password.com/docs/cli/get-started/
aws cli :brew install awscli.
fleetctl: Either built from source or installed by npm.
tuf: Download the release from https://github.com/theupdateframework/go-tuf/releases/download/v0.7.0/tuf_0.7.0_darwin_amd64.tar.gz and place the tuf executable in /usr/local/bin/tuf. You will need to make an exception in "Privacy & Security" because the executable is not signed.
gh: brew install gh.

1Password

You need to create three passphrases on your private 1Password vault for encrypting the signing keys (more on signing keys below). Create three private "passwords" with the following names: TUF TARGETS, TUF SNAPSHOT and TUF TIMESTAMP. The resulting credentials will have the following "path" within 1Password (these paths will be provided to the releaser.sh script)

Private/TUF TARGETS/password
Private/TUF SNAPSHOT/password
Private/TUF TIMESTAMP/password

If you need to run updates that require the "root" role (like rotating root keys) you will need to create a private "password" with the following name: TUF ROOT. The resulting credential will have the following "path" within 1Password (these paths will be provided to the releaser.sh script)

Private/TUF ROOT/password

AWS

The following is required to be able to run aws cli commands.

You will need to request the infrastructure team to add the "TUFAdministrators" role to your Google account.
Configure AWS SSO with the following steps: https://github.com/fleetdm/confidential/tree/main/infrastructure/sso#how-to-use-sso. Set the profile name as tuf (the profile name will be provided to the releaser.sh script).
Test the access by running:

AWS_PROFILE=tuf aws sso login

TUF signing keys

You can skip this step if you already have authorized keys to sign and publish updates.

To release updates to our TUF repository you need the root role (ask in Slack who has such root role) to sign your signing keys. First, run the following script

AWS_PROFILE=tuf \
ACTION=generate-signing-keys \
TUF_DIRECTORY=/Users/foobar/tuf3.fleetctl.com \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/TUF TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/TUF SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/TUF TIMESTAMP/password" \
./tools/tuf/releaser.sh

The human with the root role will run the following commands to sign the provided staged/root.json:

tuf sign
tuf snapshot
tuf timestamp
tuf commit

And push the newly signed root.json to the remote repository.

Encrypted keys in USB

For releasing fleetd you need to plug in the USB that contains encrypted signing keys. In this guide we assume the USB device will be mounted in /Volumes/FLEET-TUF/ and it ONLY contains a keys/ directory.

Github

Personal access token

A personal access token is required to download artifacts from Github Actions using the Github API.

Create a fine-grained personal access token at https://github.com/settings/tokens?type=beta
Store the token on 1Password as a "password" with name "Github Token" The resulting credential will have the following "path" within 1Password (this path will be provided to the script)

Private/Github Token/password

Github session

You need to log in to your Github account using the cli (gh).

gh auth login

It will be used to create a PR which is used to update the changelog and trigger the Github actions to build components.

Samples

Following are samples of the script execution to release components to edge and stable.

When releasing fleetd you need to checkout the branch (e.g. main) you want to release.

NOTE: When releasing fleetd: If there are only orbit changes on a release we still have to release the desktop component with its version string bumped (even if there are no changes in it). This is due to the fact that we want users to see the new version in the tray icon, e.g. "Fleet Desktop v1.21.0". Technical debt: We could improve this process to reduce the complexity of releasing fleetd when there are no Fleet Desktop changes.

Releasing to `edge`

Releasing fleetd `1.23.0` to `edge`

AWS_PROFILE=tuf \
TUF_DIRECTORY=/Users/foobar/tuf.fleetctl.com \
COMPONENT=fleetd \
ACTION=release-to-edge \
VERSION=1.23.0 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-TUF/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/TUF TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/TUF SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/TUF TIMESTAMP/password" \
GITHUB_USERNAME=foobar \
GITHUB_TOKEN_1PASSWORD_PATH="Private/Github Token/password" \
PUSH_TO_REMOTE=1 \
./tools/tuf/releaser.sh

Releasing osquery `5.12.1` to `edge`

AWS_PROFILE=tuf \
TUF_DIRECTORY=/Users/foobar/tuf.fleetctl.com \
COMPONENT=osqueryd \
ACTION=release-to-edge \
VERSION=5.12.1 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-TUF/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/TUF TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/TUF SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/TUF TIMESTAMP/password" \
GITHUB_USERNAME=foobar \
GITHUB_TOKEN_1PASSWORD_PATH="Private/Github Token/password" \
PUSH_TO_REMOTE=1 \
./tools/tuf/releaser.sh

Promoting from `edge` to `stable`

Promoting fleetd `1.23.0` from `edge` to `stable`

AWS_PROFILE=tuf \
TUF_DIRECTORY=/Users/foobar/tuf.fleetctl.com \
COMPONENT=fleetd \
ACTION=promote-edge-to-stable \
VERSION=1.23.0 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-TUF/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/TUF TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/TUF SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/TUF TIMESTAMP/password" \
PUSH_TO_REMOTE=1 \
./tools/tuf/releaser.sh

Promoting osqueryd `5.12.1` from `edge` to `stable`

AWS_PROFILE=tuf \
TUF_DIRECTORY=/Users/foobar/tuf.fleetctl.com \
COMPONENT=osqueryd \
ACTION=promote-edge-to-stable \
VERSION=5.12.1 \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-TUF/keys \
TARGETS_PASSPHRASE_1PASSWORD_PATH="Private/TUF TARGETS/password" \
SNAPSHOT_PASSPHRASE_1PASSWORD_PATH="Private/TUF SNAPSHOT/password" \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/TUF TIMESTAMP/password" \
PUSH_TO_REMOTE=1 \
./tools/tuf/releaser.sh

Releasing `swiftDialog` to `stable`

releaser.sh doesn't support swiftDialog yet. macOS only component

The swiftDialog executable can be generated from a macOS host by running:

make swift-dialog-app-tar-gz version=2.2.1 build=4591 out-path=.

fleetctl updates add --target /path/to/macos/swiftDialog.app.tar.gz --platform macos --name swiftDialog --version 2.2.1 -t edge

Releasing `nudge` to `stable`

releaser.sh doesn't support nudge yet. macOS only component

The nudge executable can be generated from a macOS host by running:

make nudge-app-tar-gz version=1.1.10.81462 out-path=.

fleetctl updates add --target /path/to/macos/nudge.app.tar.gz --platform macos --name nudge --version 1.1.10.81462 -t edge

Releasing `Escrow Buddy` to `stable`

releaser.sh doesn't support Escrow Buddy yet. macOS only component

The Escrow Buddy pkg installer can be generated by running:

make escrow-buddy-pkg version=1.0.0 out-path=.

fleetctl updates add --target /path/to/escrowBuddy.pkg --platform macos --name escrowBuddy --version 1.0.0 -t stable

Updating timestamp

AWS_PROFILE=tuf \
TUF_DIRECTORY=/Users/foobar/tuf.fleetctl.com \
ACTION=update-timestamp \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-TUF/keys \
TIMESTAMP_PASSPHRASE_1PASSWORD_PATH="Private/TUF TIMESTAMP/password" \
PUSH_TO_REMOTE=1 \
./tools/tuf/releaser.sh

Rotating root key

Steps to perform and verify a rotation of a (soon to be or expired) root key.

Make sure to replace "foobar" with your username on all commands.

Pull remote repository to a local folder:

AWS_PROFILE=tuf \
TUF_DIRECTORY=/Users/foobar/tuf.fleetctl.com \
ACTION=pull-from-remote \
./tools/tuf/releaser.sh

Run a local server to serve the repository.

go run ./tools/file-server 8081 "/Users/foobar/tuf.fleetctl.com/repository" &

Run ngrok or similar to expose the server to a public URL.
Once you have the public URL, attempt to create a package and install on your devices:

fleetctl package --type={pkg|deb|rpm|msi} \
    --enable-scripts --fleet-desktop \
    --fleet-url=... \
    --enroll-secret=... 
    --update-url=https://your-ngrok-subdomain.ngrok.app

The following will rotate the root key and exit (no push):

AWS_PROFILE=tuf \
TUF_DIRECTORY=/Users/foobar/tuf.fleetctl.com \
ACTION=rotate-root-key \
KEYS_SOURCE_DIRECTORY=/Volumes/FLEET-TUF/keys \
ROOT_PASSPHRASE_1PASSWORD_PATH="Private/TUF ROOT/password" \
./tools/tuf/releaser.sh

Make sure the local repository is in a valid state (by running step (3) again). Another way to do this is by changing the TUF URL on your already running agents from https://tuf.fleetctl.com to the URL generated in step (2).
Push new root key to production:

AWS_PROFILE=tuf \
TUF_DIRECTORY=/Users/foobar/tuf.fleetctl.com \
ACTION=push-to-remote \
./tools/tuf/releaser.sh

Testing and improving the script

You can specify GIT_REPOSITORY_DIRECTORY to set a separate path for the Fleet repository (it uses the current by default). This is sometimes necessary if the tooling the script uses is not present in the branch we are trying to release from.

git clone git@github.com:fleetdm/fleet.git <SOME_DIRECTORY>
GIT_REPOSITORY_DIRECTORY=<SOME_DIRECTORY>

If the PR and orbit tag were already generated but you need to run the script again you can set SKIP_PR_AND_TAG_PUSH=1 to skip that part.
While developing you can run with PUSH_TO_REMOTE=0 to prevent pushing invalid metadata/components to the production repository.

TODOs to improve releaser.sh

Support releasing nudge and swiftDialog.

Troubleshooting

Removing Unused Targets

If you've inadvertently published a target that is no longer in use, follow these steps to remove it.

Before performing any actions on Fleet's TUF repository you must:
Make sure your local copy of the repository is up-to-date. See Syncing Fleet's TUF repository.
Create a local backup in case we mess up with the repository:
mkdir ~/tuf.fleetctl.com/backup
cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup

You'll need the go-tuf binary. The removal operations aren't integrated into fleetctl at the moment.
Use tuf remove to remove the target and update targets.json. Substitute desktop/windows/stable/desktop.exe with the target you intend to delete.

tuf remove desktop/windows/stable/desktop.exe

Snapshot, timestamp, and commit the changes.

tuf snapshot
tuf timestamp
tuf commit

Run the following command to generate a timestamp that expires in two weeks (otherwise the default expiration when using go-tuf commands is 1 day)

fleetctl updates timestamp

Confirm that the version of the local timestamp.json file is more recent than that of the remote server.
Verify the changes that will be synced by running a dry sync. Include the --delete flag as you're removing targets.

aws s3 sync ./repository s3://fleet-tuf-repo --delete --dryrun

diff the local targets.json file with its remote version.
To upload the changes, perform a sync without the --dryrun:

aws s3 sync ./repository s3://fleet-tuf-repo --delete

Invalid timestamp.json version

The following issue was solved by resigning the timestamp metadata fleetctl updates timestamp (executed three times to increase the version to 4175)

2022-08-23T13:44:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174"
2022-08-23T13:59:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174"

14 KiB Raw Blame History