From 3d420e71aca1568f56d5fc6ab2bfe51265eb6765 Mon Sep 17 00:00:00 2001 From: Lucas Manuel Rodriguez Date: Wed, 25 Oct 2023 12:13:25 -0300 Subject: [PATCH] Oncall: Update fleetd release guide (#14681) I added the commands that we use when releasing new versions of fleetd components to `edge` and when promoting `edge` to `stable`. --- orbit/docs/Releasing-Orbit.md | 274 -------------------- orbit/docs/TUF-Update-Guide.md | 459 ++++++++++++++++++--------------- 2 files changed, 246 insertions(+), 487 deletions(-) delete mode 100644 orbit/docs/Releasing-Orbit.md diff --git a/orbit/docs/Releasing-Orbit.md b/orbit/docs/Releasing-Orbit.md deleted file mode 100644 index 76516ea489..0000000000 --- a/orbit/docs/Releasing-Orbit.md +++ /dev/null @@ -1,274 +0,0 @@ -# Releasing Orbit - -What we call "Orbit" is actually a group of components: -1. Orbit executable: Orbit is the director of the orchestra. It manages itself all the other components. -2. Osquery executable/bundle. -3. "Fleet Desktop" application: Renders Fleet's tray icon on the user desktop session and provides transparency to the end-user about what Fleet collects from the device. - -# Auto-update - -Orbit runs an auto-updater routine that polls a [TUF](https://theupdateframework.io/) server for new -updates in any of the three -components mentioned above. Each component (also known as "target") can be updated independently. This document aims to -describe all the steps needed to release a new version of each target. - -## Methodology - -Our TUF server provides two channels, `edge` and `stable`. -- `stable` is what all users in production use. -- `edge` is used to verify updates before releasing to `stable`. - -At a high level, the following steps are used to release updates: - -1. The new update/s are first pushed to the `edge` channel. -2. Smoke testing is used to verify the updates are working as expected. -3. The new update/s are then pushed to the `stable` channel. -4. Same as (2), smoke testing is used to verify the updates are working as expected. - -# Actors - -In all the steps described herein, there are two actors: -- Team member "Updater" pushing the updates. This actor requires: - 1. Authorized signing keys for pushing new updates on the TUF repository. - 2. Write access to the TUF server (to update https://tuf.fleetctl.com). - 3. Write privileges to the [fleet](https://github.com/fleetdm/fleet) repository to create pull - requests and trigger Github Actions. -- Team member "Verifier" verifying/testing the pushed updates. - -The majority of the steps are run by the "Updater" team member, unless stated otherwise. - -# Updating Orbit - -## 1. Edge Release - -### Setup - -The Verifier will setup a CentOS, Ubuntu, Windows and macOS hosts with Orbit running from the `edge` channel: -```sh -fleetctl package --type=pkg --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --orbit-channel edge -fleetctl package --type=msi --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --orbit-channel edge -fleetctl package --type=deb --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --orbit-channel edge -fleetctl package --type=rpm --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --orbit-channel edge -``` - -> NOTE: The fleetctl version to use should be the latest released version, installed via `sudo npm install -g fleetctl`. - -### Steps - -Assuming version `vX.Y.Z` is being released. - -1. Run `make changelog-orbit` to generate the `orbit/CHANGELOG.md` changes for Orbit. -2. Edit `orbit/CHANGELOG.md` accordingly. -3. Checkout a new branch (we generally use `prepare-orbit-vX.Y.Z`), commit the changes and tag the repository: -```sh -git checkout -b prepare-orbit-vX.Y.Z -git add -u -git commit -m "Prepare changes for Orbit vX.Y.Z" -git tag orbit-vX.Y.Z -``` -4. Push the branch and the tag: -```sh -git push origin prepare-orbit-vX.Y.Z -git push origin --tags -``` -5. After pushing the branch, create a pull request. -6. The pushed tag will trigger a new build of the following Github Action: - [goreleaser-orbit.yaml](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-orbit.yaml). -7. If the above Github Action ran successfully then a new "DRAFT" release will be created for Orbit - vX.Y.Z: https://github.com/fleetdm/fleet/releases. -8. Download and extract the assets (one for each platform Orbit supports). -9. Push the downloaded+extracted assets to the `edge` channel on our TUF repository (https://tuf.fleetctl.com/): -```sh -# Having extracted the asset for Linux in `./orbit-linux` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target orbit \ - --platform linux \ - --name ./orbit-linux \ - --version X.Y.Z -t X.Y -t X -t edge - -# Having extracted the asset for Linux in `./orbit-darwin` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target orbit \ - --platform macos \ - --name ./orbit-darwin \ - --version X.Y.Z -t X.Y -t X -t edge - -# Having extracted the asset for Windows in `./orbit.exe` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target orbit \ - --platform windows \ - --name ./orbit.exe \ - --version X.Y.Z -t X.Y -t X -t edge -``` - -### Verification - -Verifier will make sure all the hosts have updated the target successfully. The update interval delay can be up to 15 minutes. -Verifier can run `SELECT * from orbit_info;` live query on the hosts, which will provide the orbit version (confirming the update was successful). - -Once orbit has auto-updated on all hosts, Verifier runs the usual smoke testing on the 4 OSs (e.g. refetching & live querying hosts, listing software, etc.). - -## 2. Stable Release - -### Setup - -Verifier runs the same setup as `edge`, but without setting the `--orbit-channel` flag (the default value is `stable`). - -### Steps - -Run the same `fleetctl updates add` command as the `edge` case with the same targets, but with `-t stable`. - -### Verification - -Verification is the same as with the `edge` case. - -# Updating Osquery - -## 1. Edge Release - -### Setup - -The Verifier will setup a CentOS, Ubuntu, Windows and macOS host with `osqueryd` that uses the `edge` channel: -```sh -fleetctl package --type=pkg --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --osqueryd-channel edge -fleetctl package --type=msi --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --osqueryd-channel edge -fleetctl package --type=deb --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --osqueryd-channel edge -fleetctl package --type=rpm --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --osqueryd-channel edge -``` - -> NOTE: The fleetctl version to use should be the latest released version, installed via `sudo npm install -g fleetctl`. - -### Steps - -Assuming version `vX.Y.Z` is being released. - -1. Checkout a branch and edit the `OSQUERY_VERSION` env variable in `.github/workflows/generate-osqueryd-targets.yml`. -2. Push and create a pull request. -3. Once the pull request is created a Github Action will be triggered - [generate-osqueryd-targets.yml](https://github.com/fleetdm/fleet/actions/workflows/generate-osqueryd-targets.yml). - It generates the osqueryd targets for macOS, Windows and Linux as artifacts. -4. Download the artifacts from the previous step and push them to the `edge` channel: -```sh -# Having extracted the asset for Linux in `./osqueryd` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target osqueryd \ - --platform linux \ - --name ./osqueryd \ - --version X.Y.Z -t X.Y -t X -t edge - -# Having extracted the asset for Linux in `./osqueryd.app.tar.gz` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target osqueryd \ - --platform macos-app \ - --name ./osqueryd.app.tar.gz \ - --version X.Y.Z -t X.Y -t X -t edge - -# Having extracted the asset for Windows in `./osqueryd.exe` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target osqueryd \ - --platform windows \ - --name ./osqueryd.exe \ - --version X.Y.Z -t X.Y -t X -t edge -``` - -### Verification - -Verifier will make sure all the hosts have updated the target successfully. The update interval delay can be up to 15 minutes. -Verifier can run `SELECT * from osquery_info;` live query on the hosts, which will provide the osquery version (confirming the update was successful). - -Once osqueryd has auto-updated on all hosts, Verifier runs the usual smoke testing on the 4 OSs (e.g. refetching & live querying hosts, listing software, etc.). - -## 2. Stable Release - -### Setup - -Verifier runs the same setup as `edge`, but without setting the `--osqueryd-channel` flag (the default value is `stable`). - -### Steps - -Run the same `fleetctl updates add` command as the `edge` case with the same targets, but with `-t stable`. - -### Verification - -Verification is the same as with the `edge` case. - -# Updating Fleet Desktop - -## 1. Edge Release - -### Setup - -The Verifier will setup a CentOS, Ubuntu, Windows and macOS host with `desktop` that uses the `edge` channel: -```sh -fleetctl package --type=pkg --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --desktop-channel edge -fleetctl package --type=msi --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --desktop-channel edge -fleetctl package --type=deb --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --desktop-channel edge -fleetctl package --type=rpm --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --desktop-channel edge -``` - -> NOTE: The fleetctl version to use should be the latest released version, installed via `sudo npm install -g fleetctl`. - -### Steps - -Assuming version `vX.Y.Z` is being released. - -1. Checkout a branch and edit the `FLEET_DESKTOP_VERSION` env variable in `.github/workflows/generate-desktop-targets.yml`. -2. Push and create a pull request. -3. Once the pull request is created a Github Action will be triggered - [generate-desktop-targets.yml](https://github.com/fleetdm/fleet/actions/workflows/generate-desktop-targets.yml). - It generates the desktop targets for macOS, Windows and Linux as artifacts. -4. Download the artifacts from the previous step and push them to the `edge` channel: -```sh -# Having extracted the asset for Linux in `./desktop.tar.gz` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target desktop \ - --platform linux \ - --name ./desktop.tar.gz \ - --version X.Y.Z -t X.Y -t X -t edge - -# Having extracted the asset for Linux in `./desktop.app.tar.gz` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target desktop \ - --platform macos \ - --name ./desktop.app.tar.gz \ - --version X.Y.Z -t X.Y -t X -t edge - -# Having extracted the asset for Windows in `./fleet-desktop.exe` -fleetctl updates add \ - --path $STAGING_TUF_PATH_LOCATION \ - --target desktop \ - --platform windows \ - --name ./fleet-desktop.exe \ - --version X.Y.Z -t X.Y -t X -t edge -``` - -### Verification - -Verifier will make sure all the hosts have updated the target successfully. The update interval delay can be up to 15 minutes. - -Currently, there's no direct way to verify the auto-update for the Fleet Desktop application. -One way to verify is to check for `INF exiting due to successful update` in the Orbit logs. - -Once the Fleet Desktop Application has auto-updated on all hosts, Verifier runs the usual smoke testing on the 4 OSs (e.g. refetching & live querying hosts, listing software, etc.). - -## 2. Stable Release - -### Setup - -Verifier runs the same setup as `edge`, but without setting the `--desktop-channel` flag (the default value is `stable`). - -### Steps - -Run the same `fleetctl updates add` command as the `edge` case with the same targets, but with `-t stable`. - -### Verification - -Verification is the same as with the `edge` case. diff --git a/orbit/docs/TUF-Update-Guide.md b/orbit/docs/TUF-Update-Guide.md index 1aaa929860..691990b92a 100644 --- a/orbit/docs/TUF-Update-Guide.md +++ b/orbit/docs/TUF-Update-Guide.md @@ -1,39 +1,260 @@ -# TUF Update Guide +# Pushing new releases to TUF This document is a walkthrough guide for: -- A Fleet member to become a publisher of updates for [Fleet's TUF service](tuf.fleetctl.com). -- A Fleet member to publish new updates to [Fleet's TUF service](tuf.fleetctl.com). +- A Fleet member to publish new updates to [Fleet's TUF service](tuf.fleetctl.com). See [Pushing updates](#pushing-updates). +- A Fleet member to delete targets from [Fleet's TUF service](tuf.fleetctl.com). See [Removing unused targets](#removing-unused-targets). +- A Fleet member to become a publisher of updates for [Fleet's TUF service](tuf.fleetctl.com). See [Becoming a new Fleet publisher](#becoming-a-new-fleet-publisher). -> The roles needed to push new updates are `targets`, `snapshot` and `timestamp`. See [Roles and Metadata](https://theupdateframework.io/metadata/). +## Security -## Become a New Fleet Publisher +- The TUF keys for `targets`, `snapshot` and `timestamp` should be stored on a USB stick (used solely for this purpose). Whenever you need to push updates to Fleet's TUF repository you can temporarily copy the encrypted keys to your workstation (under the `keys/` folder, more on this below). +- The keys should be stored encrypted with its passphrase stored in 1Password (on a private vault). -### Security +## Syncing Fleet's TUF repository -- TUF keys for `targets`, `snapshot` and `timestamp` should be stored on a USB stick (used solely for this purpose). -- The keys are stored encrypted with a passphrase stored in 1Password (on a private vault). +> The `fleetctl updates` commands assume the folders `keys/`, `staged/` and `repository/` exist on the current working directory. -### Sync Fleet's TUF repository +> IMPORTANT: When syncing the repository make sure to use `--exact-timestamps`. Otherwise `aws s3 sync` may not sync files that do not change in size, like `timestamp.json`. -The `fleetctl updates --path=` commands assume `keys/`, `staged/` and `repository/` are under - (default value for is the current directory `"."`). +- The `keys/` folder contains the encrypted private keys. +- The `staged/` folder contains uncommitted changes (usually empty because `fleetctl updates` commands automatically commit the changes). +- The `repository/` folder contains the full TUF repository. + +Following are the commands to initialize the repository on your workstation: +```sh +mkdir /path/to/tuf.fleetctl.com + +cd /path/to/tuf.fleetctl.com +mkdir -p ./repository +cp /Volumes/YOUR-USB-NAME/keys ./keys +mkdir -p ./staged + +aws s3 sync s3://fleet-tuf-repo ./repository --exact-timestamps +``` + +## Pushing updates + +> Before performing any actions on Fleet's TUF repository you must: +> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository). +> 2. Create a local backup in case we mess up with the repository: +> ```sh +> mkdir ~/tuf.fleetctl.com/backup +> cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup +> ``` + +### Releasing to the `edge` channel + +> Make sure to install fleetd components using the `edge` channels in the three supported OSs (this is useful to smoke test the update). + +Following is the list of components and each command for each operating system. + +The commands show here update the local repository. After you are done running the commands below for each component, see [Pushing releases to Fleet's TUF repository](#pushing-releases-to-fleets-tuf-repository) to push the updates to Fleet's TUF repository (https://tuf.fleetctl.com). + +#### orbit + +The `orbit` executables are downloaded from the [GoReleaser Orbit action](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-orbit.yaml). +Such action is triggered when git tagging a new orbit version with a tag of the form: `orbit-v1.15.0`. + +> The following commands assume you are pushing version `1.15.0`. ```sh -cd /Volumes/FLEET-TUF/repository - -cd /Volumes/FLEET-TUF -mkdir -p repository -mkdir -p keys -mkdir -p staged - -aws s3 sync s3://fleet-tuf-repo ./repository +# macOS +fleetctl updates add --target /path/to/downloaded/macos/orbit --platform macos --name orbit --version 1.15.0 -t edge +# Linux +fleetctl updates add --target /path/to/downloaded/linux/orbit --platform linux --name orbit --version 1.15.0 -t edge +# Windows +fleetctl updates add --target /path/to/downloaded/windows/orbit.exe --platform windows --name orbit --version 1.15.0 -t edge ``` +#### desktop + +The Fleet Desktop executables are downloaded from the [Generate Fleet Desktop targets for Orbit action](https://github.com/fleetdm/fleet/actions/workflows/generate-desktop-targets.yml). +Such action is triggered by submitting a PR with the [following version string](https://github.com/fleetdm/fleet/blob/4a6bf0d447a2080f994da1e2f36ce6d51db88109/.github/workflows/generate-desktop-targets.yml#L27) changed. + +> The following commands assume you are pushing version `1.15.0`. + +```sh +# macOS +fleetctl updates add --target /path/to/macos/downloaded/desktop.app.tar.gz --platform macos --name desktop --version 1.15.0 -t edge +# Linux +fleetctl updates add --target /path/to/linux/downloaded/desktop.tar.gz --platform linux --name desktop --version 1.15.0 -t edge +# Windows +fleetctl updates add --target /path/to/windows/downloaded/fleet-desktop.exe --platform windows --name desktop --version 1.15.0 -t edge +``` + +#### swiftDialog + +> macOS only component + +The `swiftDialog` executable can be generated from a macOS host by running: +```sh +make swift-dialog-app-tar-gz version=2.2.1 build=4591 out-path=. +``` + +```sh +fleetctl updates add --target /path/to/macos/swiftDialog.app.tar.gz --platform macos --name swiftDialog --version 2.2.1 -t edge +``` + +#### nudge + +> macOS only component + +The `nudge` executable can be generated from a macOS host by running: +```sh +make nudge-app-tar-gz version=1.1.10.81462 out-path=. +``` + +```sh +fleetctl updates add --target /path/to/macos/nudge.app.tar.gz --platform macos --name nudge --version 1.1.10.81462 -t edge +``` + +#### osqueryd + +Osquery executables are downloaded from the [Generate osqueryd targets for Fleetd action](https://github.com/fleetdm/fleet/blob/main/.github/workflows/generate-osqueryd-targets.yml). +Such action is triggered by submitting a PR with the [following version string](https://github.com/fleetdm/fleet/blob/7067ca586a4aa1a0377b387d4b4478a5958193ff/.github/workflows/generate-osqueryd-targets.yml#L27) changed. + +> The following commands assume you are pushing version `5.9.1`. + +```sh +# macOS +fleetctl updates add --target /path/to/downloaded/macos/osqueryd.app.tar.gz --platform macos-app --name osqueryd --version 5.9.1 -t edge +# Linux +fleetctl updates add --target /path/to/downloaded/linux/osqueryd --platform linux --name osqueryd --version 5.9.1 -t edge +# Windows +fleetctl updates add --target /path/to/downloaded/windows/osqueryd.exe --platform windows --name osqueryd --version 5.9.1 -t edge +``` + +### Promoting `edge` to the `stable` channel + +> Make sure to install fleetd components using the `stable` channels in the three supported OSs (this is useful to smoke test the update). + +Following is the list of components and each command for each operating system. + +The commands show here update the local repository. After you are done running the commands below for each component, see [Pushing releases to Fleet's TUF repository](#pushing-releases-to-fleets-tuf-repository) to push the updates to Fleet's TUF repository (https://tuf.fleetctl.com). + +#### orbit + +> The following commands assume you are pushing version `1.15.0`. + +```sh +# macOS +fleetctl updates add --target ./repository/targets/orbit/macos/edge/orbit --platform macos --name orbit --version 1.15.0 -t 1.15 -t 1 -t stable +# Linux +fleetctl updates add --target ./repository/targets/orbit/linux/edge/orbit --platform linux --name orbit --version 1.15.0 -t 1.15 -t 1 -t stable +# Windows +fleetctl updates add --target ./repository/targets/orbit/windows/edge/orbit.exe --platform windows --name orbit --version 1.15.0 -t 1.15 -t 1 -t stable +``` + +#### desktop + +> The following commands assume you are pushing version `1.15.0`. + +```sh +# macOS +fleetctl updates add --target ./repository/targets/desktop/macos/edge/desktop.app.tar.gz --platform macos --name desktop --version 1.15.0 -t 1.15 -t 1 -t stable +# Linux +fleetctl updates add --target ./repository/targets/desktop/linux/edge/desktop.tar.gz --platform linux --name desktop --version 1.15.0 -t 1.15 -t 1 -t stable +# Windows +fleetctl updates add --target ./repository/targets/desktop/windows/edge/fleet-desktop.exe --platform windows --name desktop --version 1.15.0 -t 1.15 -t 1 -t stable +``` + +#### swiftDialog + +```sh +# macOS +fleetctl updates add --target ./repository/targets/swiftDialog/macos/edge/swiftDialog.app.tar.gz --platform macos --name swiftDialog --version 2.2.1 -t stable +``` + +#### nudge + +```sh +# macOS +fleetctl updates add --target ./repository/targets/nudge/macos/edge/nudge.app.tar.gz --platform macos --name nudge --version 1.1.10.81462 -t stable +``` + +#### osqueryd + +> The following commands assume you are pushing version `5.9.1`. + +```sh +# macOS +fleetctl updates add --target ./repository/targets/osqueryd/macos-app/edge/osqueryd.app.tar.gz --platform macos-app --name osqueryd --version 5.9.1 -t 5.9 -t 5 -t stable +# Linux +fleetctl updates add --target ./repository/targets/osqueryd/linux/edge/osqueryd --platform linux --name osqueryd --version 5.9.1 -t 5.9 -t 5 -t stable +# Windows +fleetctl updates add --target ./repository/targets/osqueryd/windows/edge/osqueryd.exe --platform windows --name osqueryd --version 5.9.1 -t 5.9 -t 5 -t stable +``` + +#### Pushing releases to Fleet's TUF repository + +Once you are done with the changes on your local repository, you can use the following command to review the changes before pushing (`--dryrun` allows us to verify the upgrade before pushing): +```sh +AWS_PROFILE=tuf aws s3 sync ./repository s3://fleet-tuf-repo --dryrun +(dryrun) upload: repository/snapshot.json to s3://fleet-tuf-repo/snapshot.json +(dryrun) upload: repository/targets.json to s3://fleet-tuf-repo/targets.json +[...] +(dryrun) upload: repository/timestamp.json to s3://fleet-tuf-repo/timestamp.json +``` + +If all looks good, run the same command without the `--dryrun` flag. + +> NOTE: Some things to note after the changes are pushed: +> - Once pushed you might see some clients failing to upgrade due to some sha256 mismatches. These temporary failures are expected because it takes some time for caches to be invalidated (these errors should go away after a few minutes). +> - The auto-update routines in orbit run every one hour, so you might need to wait up to an hour to verify hosts are auto-updating properly. + +## Removing Unused Targets + +If you've inadvertently published a target that is no longer in use, follow these steps to remove it. + +> Before performing any actions on Fleet's TUF repository you must: +> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository). +> 2. Create a local backup in case we mess up with the repository: +> ```sh +> mkdir ~/tuf.fleetctl.com/backup +> cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup +> ``` + +1. You'll need the [`go-tuf`](https://github.com/theupdateframework/go-tuf) binary. The removal operations aren't integrated into `fleetctl` at the moment. +2. Use `tuf remove` to remove the target and update `targets.json`. Substitute `desktop/windows/stable/desktop.exe` with the target you intend to delete. +```sh +tuf remove desktop/windows/stable/desktop.exe +``` +3. Snapshot, timestamp, and commit the changes. +```sh +tuf snapshot +tuf timestamp +tuf commit +``` +4. Run the following command to generate a timestamp that expires in two weeks (otherwise the default expiration when using `go-tuf` commands is 1 day) +```sh +fleetctl updates timestamp +``` +5. Confirm that the version of the local `timestamp.json` file is more recent than that of the remote server. +6. Verify the changes that will be synced by running a dry sync. Include the `--delete` flag as you're removing targets. +```sh +aws s3 sync ./repository s3://fleet-tuf-repo --delete --dryrun +``` +7. `diff` the local `targets.json` file with its remote version. +8. To upload the changes, perform a sync without the `--dryrun`: +```sh +aws s3 sync ./repository s3://fleet-tuf-repo --delete +``` + +## Becoming a New Fleet Publisher + +> Before performing any actions on Fleet's TUF repository you must: +> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository). +> 2. Create a local backup in case we mess up with the repository: +> ```sh +> mkdir ~/tuf.fleetctl.com/backup +> cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup +> ``` + ### Generate targets+snapshot+timestamp keys -All commands shown in this guide are executed from `/Volumes/FLEET-TUF`: +All commands shown in this guide are executed from `/path/to/tuf.fleetctl.com`: ```sh -cd /Volumes/FLEET-TUF +cd /path/to/tuf.fleetctl.com ``` ```sh @@ -65,200 +286,12 @@ Essentially the following commands are executed to sign the new keys: - `tuf timestamp` - `tuf commit` -## Pushing new updates +## Misc issues -Following are tested steps to push new targets. +### Invalid timestamp.json version -### 1. Backup current TUF repository - -Just in case we break the remote TUF directory, let's do a local backup: -```sh -mkdir ~/tuf.fleetctl.com/backup -aws s3 sync s3://fleet-tuf-repo ~/tuf.fleetctl.com/backup -``` - -### 2. Make sure the local repository is up-to-date - -```sh -aws s3 sync s3://fleet-tuf-repo ./repository -``` - -### 3. Setup Orbit in Linux, Windows, macOS - -Install Orbit with the (to be updated) channels in the three supported OSs. - -E.g. if we need to push a new version of `osqueryd` to `edge`, then generate and install a package with: -```sh -fleetctl package ... --osqueryd-channel=edge ... -``` - -### 4. Setup Orbit in one host that points to our repository - -This allows us to verify that already running clients will upgrade successfully. - -Serve current unmodified repository: -```sh -cd repository && python3 -m http.server -``` - -Generate packages using the local TUF server (in my case on a macOS host): -```sh -fleetctl package --type=pkg --update-url=http://localhost:8000 ... -[...] -``` - -Install generated `fleet-osquery.pkg`. - -### 5. Actually pushing new updates - -In this example we are promoting orbit from `edge` to `stable`: -```sh -./fleetctl-macos updates add --target ./repository/targets/orbit/linux/edge/orbit --platform linux --name orbit --version 1.1.0 -t 1.1 -t 1 -t stable -[...] -./fleetctl-macos updates add --target ./repository/targets/orbit/windows/edge/orbit.exe --platform windows --name orbit --version 1.1.0 -t 1.1 -t 1 -t stable -[...] -./fleetctl-macos updates add --target ./repository/targets/orbit/macos/edge/orbit --platform macos --name orbit --version 1.1.0 -t 1.1 -t 1 -t stable -[...] -``` - -`--dryrun` allows us to verify the upgrade before pushing: -```sh -AWS_PROFILE=tuf aws s3 sync ./repository s3://fleet-tuf-repo --dryrun -(dryrun) upload: repository/snapshot.json to s3://fleet-tuf-repo/snapshot.json -(dryrun) upload: repository/targets.json to s3://fleet-tuf-repo/targets.json -(dryrun) upload: repository/targets/orbit/linux/1.1.0/orbit to s3://fleet-tuf-repo/targets/orbit/linux/1.1.0/orbit -(dryrun) upload: repository/targets/orbit/linux/1.1/orbit to s3://fleet-tuf-repo/targets/orbit/linux/1.1/orbit -(dryrun) upload: repository/targets/orbit/linux/1/orbit to s3://fleet-tuf-repo/targets/orbit/linux/1/orbit -(dryrun) upload: repository/targets/orbit/linux/stable/orbit to s3://fleet-tuf-repo/targets/orbit/linux/stable/orbit -(dryrun) upload: repository/targets/orbit/macos/1.1.0/orbit to s3://fleet-tuf-repo/targets/orbit/macos/1.1.0/orbit -(dryrun) upload: repository/targets/orbit/macos/1.1/orbit to s3://fleet-tuf-repo/targets/orbit/macos/1.1/orbit -(dryrun) upload: repository/targets/orbit/macos/1/orbit to s3://fleet-tuf-repo/targets/orbit/macos/1/orbit -(dryrun) upload: repository/targets/orbit/macos/stable/orbit to s3://fleet-tuf-repo/targets/orbit/macos/stable/orbit -(dryrun) upload: repository/targets/orbit/windows/1.1.0/orbit.exe to s3://fleet-tuf-repo/targets/orbit/windows/1.1.0/orbit.exe -(dryrun) upload: repository/targets/orbit/windows/1.1/orbit.exe to s3://fleet-tuf-repo/targets/orbit/windows/1.1/orbit.exe -(dryrun) upload: repository/targets/orbit/windows/1/orbit.exe to s3://fleet-tuf-repo/targets/orbit/windows/1/orbit.exe -(dryrun) upload: repository/targets/orbit/windows/stable/orbit.exe to s3://fleet-tuf-repo/targets/orbit/windows/stable/orbit.exe -(dryrun) upload: repository/timestamp.json to s3://fleet-tuf-repo/timestamp.json -``` - -In this other example we are updating osquery's `edge` channel: - -```sh -./fleetctl-macos updates add --target /Users/luk/Downloads/tuf-osqueryd/osqueryd.exe --platform windows --name osqueryd --version 5.5.1 -t edge -[...] -./fleetctl-macos updates add --target /Users/luk/Downloads/tuf-osqueryd/osqueryd.app.tar.gz --platform macos-app --name osqueryd --version 5.5.1 -t edge -[...] -./fleetctl-macos updates add --target /Users/luk/Downloads/tuf-osqueryd/osqueryd --platform linux --name osqueryd --version 5.5.1 -t edge -[...] -``` - -`--dryrun` allows us to verify the upgrade before pushing: -```sh -aws s3 sync ./repository s3://fleet-tuf-repo --profile tuf --dryrun -(dryrun) upload: repository/snapshot.json to s3://fleet-tuf-repo/snapshot.json -(dryrun) upload: repository/targets.json to s3://fleet-tuf-repo/targets.json -(dryrun) upload: repository/targets/osqueryd/linux/5.5.1/osqueryd to s3://fleet-tuf-repo/targets/osqueryd/linux/5.5.1/osqueryd -(dryrun) upload: repository/targets/osqueryd/linux/edge/osqueryd to s3://fleet-tuf-repo/targets/osqueryd/linux/edge/osqueryd -(dryrun) upload: repository/targets/osqueryd/macos-app/5.5.1/osqueryd.app.tar.gz to s3://fleet-tuf-repo/targets/osqueryd/macos-app/5.5.1/osqueryd.app.tar.gz -(dryrun) upload: repository/targets/osqueryd/macos-app/edge/osqueryd.app.tar.gz to s3://fleet-tuf-repo/targets/osqueryd/macos-app/edge/osqueryd.app.tar.gz -(dryrun) upload: repository/targets/osqueryd/windows/5.5.1/osqueryd.exe to s3://fleet-tuf-repo/targets/osqueryd/windows/5.5.1/osqueryd.exe -(dryrun) upload: repository/targets/osqueryd/windows/edge/osqueryd.exe to s3://fleet-tuf-repo/targets/osqueryd/windows/edge/osqueryd.exe -(dryrun) upload: repository/timestamp.json to s3://fleet-tuf-repo/timestamp.json -``` - -### 6. Verify the already running test host - -Verify host enrolled in step (4) upgraded to the new versions successfully. - -### 7. Verify generation of new packages - -```sh -fleetctl package --type=pkg --update-url=http://localhost:8000 ... -fleetctl package --type=msi --update-url=http://localhost:8000 ... -fleetctl package --type=deb --update-url=http://localhost:8000 ... -``` - -### 8. Push! - -Run the same command shown above, but without `--dryrun` -```sh -aws s3 sync ./repository s3://fleet-tuf-repo --profile tuf -``` - -### 9. Final Verification - -Now that the repository is pushed, verify that the hosts enrolled in step (3) update as expected. - -### Removing Unused Targets - -If you've inadvertently published a target that is no longer in use, follow these steps to remove it. - -### 1. Preparation - -1. Backup the remote TUF directory to your local machine. - -```sh -mkdir ~/tuf.fleetctl.com/backup -aws s3 sync s3://fleet-tuf-repo ~/tuf.fleetctl.com/backup -``` - -2. Ensure your local repository mirrors the current state. - -```sh -aws s3 sync s3://fleet-tuf-repo ./repository -``` - -3. You'll need the [`go-tuf`](https://github.com/theupdateframework/go-tuf) binary. The removal operations aren't integrated into `fleetctl` at the moment. - -### 2. Local Target Removal - -1. Use `tuf remove` to remove the target and update `targets.json`. Substitute `desktop/windows/stable/desktop.exe` with the target you intend to delete. - -```sh -tuf remove desktop/windows/stable/desktop.exe -``` - -2. Snapshot, timestamp, and commit the changes. - -```sh -tuf snapshot -tuf timestamp -tuf commit -``` - -### 3. Verification Before Publishing - -1. Confirm that the version of the local `timestamp.json` file is more recent than that of the remote server. - -2. Verify the changes that will be synced by running a dry sync. Include the `--delete` flag as you're removing targets. - -```sh -aws s3 sync ./repository s3://fleet-tuf-repo --delete --dryrun -``` - -3. `diff` the local `targets.json` file with its remote version. - -### 4. Publish and Confirm - -1. To upload the changes, perform a sync without the `--dryrun`. - -```sh -aws s3 sync ./repository s3://fleet-tuf-repo --delete -``` - -2. Check the changes on the remote repository. - -### Issues found - -#### Invalid timestamp.json version - -The following issue was solved by resigning the timestamp metadata `fleetctl updates timestamp` (executed three times to increase the version to 4175) +The following issue was solved by resigning the timestamp metadata `fleetctl updates timestamp` (executed three times to increase the version to `4175`) ```sh 2022-08-23T13:44:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174" 2022-08-23T13:59:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174" -``` - -## Notes - -- "Measure thrice cut once": Steps 3, 4, 5 and 6 allows us to verify the repository is in good shape before pushing to [Fleet's TUF service](tuf.fleetctl.com). -- Steps may look different if the upgrade is performed on a Linux or Windows host. +``` \ No newline at end of file