Elgato_dark/fleet
1
0
Fork 0
mirror of https://github.com/fleetdm/fleet synced 2026-05-23 00:49:03 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 106

Oncall: Update fleetd release guide (#14681)

Browse source 
I added the commands that we use when releasing new versions of fleetd
components to `edge` and when promoting `edge` to `stable`.
This commit is contained in:
Lucas Manuel Rodriguez 2023-10-25 12:13:25 -03:00 committed by GitHub
parent c12af96f7d
commit 3d420e71ac
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 246 additions and 487 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

274
orbit/docs/Releasing-Orbit.md
View file

@ -1,274 +0,0 @@
# Releasing Orbit
What we call "Orbit" is actually a group of components:
1. Orbit executable: Orbit is the director of the orchestra. It manages itself all the other components.
2. Osquery executable/bundle.
3. "Fleet Desktop" application: Renders Fleet's tray icon on the user desktop session and provides transparency to the end-user about what Fleet collects from the device.
# Auto-update
Orbit runs an auto-updater routine that polls a [TUF](https://theupdateframework.io/) server for new
updates in any of the three
components mentioned above. Each component (also known as "target") can be updated independently. This document aims to
describe all the steps needed to release a new version of each target.
## Methodology
Our TUF server provides two channels, `edge` and `stable`.
- `stable` is what all users in production use.
- `edge` is used to verify updates before releasing to `stable`.
At a high level, the following steps are used to release updates:
1. The new update/s are first pushed to the `edge` channel.
2. Smoke testing is used to verify the updates are working as expected.
3. The new update/s are then pushed to the `stable` channel.
4. Same as (2), smoke testing is used to verify the updates are working as expected.
# Actors
In all the steps described herein, there are two actors:
- Team member "Updater" pushing the updates. This actor requires:
1. Authorized signing keys for pushing new updates on the TUF repository.
2. Write access to the TUF server (to update https://tuf.fleetctl.com).
3. Write privileges to the [fleet](https://github.com/fleetdm/fleet) repository to create pull
requests and trigger Github Actions.
- Team member "Verifier" verifying/testing the pushed updates.
The majority of the steps are run by the "Updater" team member, unless stated otherwise.
# Updating Orbit
## 1. Edge Release
### Setup
The Verifier will setup a CentOS, Ubuntu, Windows and macOS hosts with Orbit running from the `edge` channel:
```sh
fleetctl package --type=pkg --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --orbit-channel edge
fleetctl package --type=msi --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --orbit-channel edge
fleetctl package --type=deb --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --orbit-channel edge
fleetctl package --type=rpm --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --orbit-channel edge
```
> NOTE: The fleetctl version to use should be the latest released version, installed via `sudo npm install -g fleetctl`.
### Steps
Assuming version `vX.Y.Z` is being released.
1. Run `make changelog-orbit` to generate the `orbit/CHANGELOG.md` changes for Orbit.
2. Edit `orbit/CHANGELOG.md` accordingly.
3. Checkout a new branch (we generally use `prepare-orbit-vX.Y.Z`), commit the changes and tag the repository:
```sh
git checkout -b prepare-orbit-vX.Y.Z
git add -u
git commit -m "Prepare changes for Orbit vX.Y.Z"
git tag orbit-vX.Y.Z
```
4. Push the branch and the tag:
```sh
git push origin prepare-orbit-vX.Y.Z
git push origin --tags
```
5. After pushing the branch, create a pull request.
6. The pushed tag will trigger a new build of the following Github Action:
[goreleaser-orbit.yaml](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-orbit.yaml).
7. If the above Github Action ran successfully then a new "DRAFT" release will be created for Orbit
vX.Y.Z: https://github.com/fleetdm/fleet/releases.
8. Download and extract the assets (one for each platform Orbit supports).
9. Push the downloaded+extracted assets to the `edge` channel on our TUF repository (https://tuf.fleetctl.com/):
```sh
# Having extracted the asset for Linux in `./orbit-linux`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target orbit \
--platform linux \
--name ./orbit-linux \
--version X.Y.Z -t X.Y -t X -t edge
# Having extracted the asset for Linux in `./orbit-darwin`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target orbit \
--platform macos \
--name ./orbit-darwin \
--version X.Y.Z -t X.Y -t X -t edge
# Having extracted the asset for Windows in `./orbit.exe`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target orbit \
--platform windows \
--name ./orbit.exe \
--version X.Y.Z -t X.Y -t X -t edge
```
### Verification
Verifier will make sure all the hosts have updated the target successfully. The update interval delay can be up to 15 minutes.
Verifier can run `SELECT * from orbit_info;` live query on the hosts, which will provide the orbit version (confirming the update was successful).
Once orbit has auto-updated on all hosts, Verifier runs the usual smoke testing on the 4 OSs (e.g. refetching & live querying hosts, listing software, etc.).
## 2. Stable Release
### Setup
Verifier runs the same setup as `edge`, but without setting the `--orbit-channel` flag (the default value is `stable`).
### Steps
Run the same `fleetctl updates add` command as the `edge` case with the same targets, but with `-t stable`.
### Verification
Verification is the same as with the `edge` case.
# Updating Osquery
## 1. Edge Release
### Setup
The Verifier will setup a CentOS, Ubuntu, Windows and macOS host with `osqueryd` that uses the `edge` channel:
```sh
fleetctl package --type=pkg --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --osqueryd-channel edge
fleetctl package --type=msi --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --osqueryd-channel edge
fleetctl package --type=deb --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --osqueryd-channel edge
fleetctl package --type=rpm --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --osqueryd-channel edge
```
> NOTE: The fleetctl version to use should be the latest released version, installed via `sudo npm install -g fleetctl`.
### Steps
Assuming version `vX.Y.Z` is being released.
1. Checkout a branch and edit the `OSQUERY_VERSION` env variable in `.github/workflows/generate-osqueryd-targets.yml`.
2. Push and create a pull request.
3. Once the pull request is created a Github Action will be triggered
[generate-osqueryd-targets.yml](https://github.com/fleetdm/fleet/actions/workflows/generate-osqueryd-targets.yml).
It generates the osqueryd targets for macOS, Windows and Linux as artifacts.
4. Download the artifacts from the previous step and push them to the `edge` channel:
```sh
# Having extracted the asset for Linux in `./osqueryd`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target osqueryd \
--platform linux \
--name ./osqueryd \
--version X.Y.Z -t X.Y -t X -t edge
# Having extracted the asset for Linux in `./osqueryd.app.tar.gz`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target osqueryd \
--platform macos-app \
--name ./osqueryd.app.tar.gz \
--version X.Y.Z -t X.Y -t X -t edge
# Having extracted the asset for Windows in `./osqueryd.exe`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target osqueryd \
--platform windows \
--name ./osqueryd.exe \
--version X.Y.Z -t X.Y -t X -t edge
```
### Verification
Verifier will make sure all the hosts have updated the target successfully. The update interval delay can be up to 15 minutes.
Verifier can run `SELECT * from osquery_info;` live query on the hosts, which will provide the osquery version (confirming the update was successful).
Once osqueryd has auto-updated on all hosts, Verifier runs the usual smoke testing on the 4 OSs (e.g. refetching & live querying hosts, listing software, etc.).
## 2. Stable Release
### Setup
Verifier runs the same setup as `edge`, but without setting the `--osqueryd-channel` flag (the default value is `stable`).
### Steps
Run the same `fleetctl updates add` command as the `edge` case with the same targets, but with `-t stable`.
### Verification
Verification is the same as with the `edge` case.
# Updating Fleet Desktop
## 1. Edge Release
### Setup
The Verifier will setup a CentOS, Ubuntu, Windows and macOS host with `desktop` that uses the `edge` channel:
```sh
fleetctl package --type=pkg --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --desktop-channel edge
fleetctl package --type=msi --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --desktop-channel edge
fleetctl package --type=deb --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --desktop-channel edge
fleetctl package --type=rpm --fleet-url=https://fleet.example.com --enroll-secret=<...> --fleet-desktop --desktop-channel edge
```
> NOTE: The fleetctl version to use should be the latest released version, installed via `sudo npm install -g fleetctl`.
### Steps
Assuming version `vX.Y.Z` is being released.
1. Checkout a branch and edit the `FLEET_DESKTOP_VERSION` env variable in `.github/workflows/generate-desktop-targets.yml`.
2. Push and create a pull request.
3. Once the pull request is created a Github Action will be triggered
[generate-desktop-targets.yml](https://github.com/fleetdm/fleet/actions/workflows/generate-desktop-targets.yml).
It generates the desktop targets for macOS, Windows and Linux as artifacts.
4. Download the artifacts from the previous step and push them to the `edge` channel:
```sh
# Having extracted the asset for Linux in `./desktop.tar.gz`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target desktop \
--platform linux \
--name ./desktop.tar.gz \
--version X.Y.Z -t X.Y -t X -t edge
# Having extracted the asset for Linux in `./desktop.app.tar.gz`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target desktop \
--platform macos \
--name ./desktop.app.tar.gz \
--version X.Y.Z -t X.Y -t X -t edge
# Having extracted the asset for Windows in `./fleet-desktop.exe`
fleetctl updates add \
--path $STAGING_TUF_PATH_LOCATION \
--target desktop \
--platform windows \
--name ./fleet-desktop.exe \
--version X.Y.Z -t X.Y -t X -t edge
```
### Verification
Verifier will make sure all the hosts have updated the target successfully. The update interval delay can be up to 15 minutes.
Currently, there's no direct way to verify the auto-update for the Fleet Desktop application.
One way to verify is to check for `INF exiting due to successful update` in the Orbit logs.
Once the Fleet Desktop Application has auto-updated on all hosts, Verifier runs the usual smoke testing on the 4 OSs (e.g. refetching & live querying hosts, listing software, etc.).
## 2. Stable Release
### Setup
Verifier runs the same setup as `edge`, but without setting the `--desktop-channel` flag (the default value is `stable`).
### Steps
Run the same `fleetctl updates add` command as the `edge` case with the same targets, but with `-t stable`.
### Verification
Verification is the same as with the `edge` case.

459
orbit/docs/TUF-Update-Guide.md
View file

@ -1,39 +1,260 @@
# TUF Update Guide
# Pushing new releases to TUF
This document is a walkthrough guide for:
- A Fleet member to become a publisher of updates for [Fleet's TUF service](tuf.fleetctl.com).
- A Fleet member to publish new updates to [Fleet's TUF service](tuf.fleetctl.com).
- A Fleet member to publish new updates to [Fleet's TUF service](tuf.fleetctl.com). See [Pushing updates](#pushing-updates).
- A Fleet member to delete targets from [Fleet's TUF service](tuf.fleetctl.com). See [Removing unused targets](#removing-unused-targets).
- A Fleet member to become a publisher of updates for [Fleet's TUF service](tuf.fleetctl.com). See [Becoming a new Fleet publisher](#becoming-a-new-fleet-publisher).
> The roles needed to push new updates are `targets`, `snapshot` and `timestamp`. See [Roles and Metadata](https://theupdateframework.io/metadata/).
## Security
## Become a New Fleet Publisher
- The TUF keys for `targets`, `snapshot` and `timestamp` should be stored on a USB stick (used solely for this purpose). Whenever you need to push updates to Fleet's TUF repository you can temporarily copy the encrypted keys to your workstation (under the `keys/` folder, more on this below).
- The keys should be stored encrypted with its passphrase stored in 1Password (on a private vault).
### Security
## Syncing Fleet's TUF repository
- TUF keys for `targets`, `snapshot` and `timestamp` should be stored on a USB stick (used solely for this purpose).
- The keys are stored encrypted with a passphrase stored in 1Password (on a private vault).
> The `fleetctl updates` commands assume the folders `keys/`, `staged/` and `repository/` exist on the current working directory.
### Sync Fleet's TUF repository
> IMPORTANT: When syncing the repository make sure to use `--exact-timestamps`. Otherwise `aws s3 sync` may not sync files that do not change in size, like `timestamp.json`.
The `fleetctl updates --path=<SOME_PATH>` commands assume `keys/`, `staged/` and `repository/` are under
<SOME_PATH> (default value for <SOME_PATH> is the current directory `"."`).
- The `keys/` folder contains the encrypted private keys.
- The `staged/` folder contains uncommitted changes (usually empty because `fleetctl updates` commands automatically commit the changes).
- The `repository/` folder contains the full TUF repository.
Following are the commands to initialize the repository on your workstation:
```sh
mkdir /path/to/tuf.fleetctl.com
cd /path/to/tuf.fleetctl.com
mkdir -p ./repository
cp /Volumes/YOUR-USB-NAME/keys ./keys
mkdir -p ./staged
aws s3 sync s3://fleet-tuf-repo ./repository --exact-timestamps
```
## Pushing updates
> Before performing any actions on Fleet's TUF repository you must:
> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository).
> 2. Create a local backup in case we mess up with the repository:
> ```sh
> mkdir ~/tuf.fleetctl.com/backup
> cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup
> ```
### Releasing to the `edge` channel
> Make sure to install fleetd components using the `edge` channels in the three supported OSs (this is useful to smoke test the update).
Following is the list of components and each command for each operating system.
The commands show here update the local repository. After you are done running the commands below for each component, see [Pushing releases to Fleet's TUF repository](#pushing-releases-to-fleets-tuf-repository) to push the updates to Fleet's TUF repository (https://tuf.fleetctl.com).
#### orbit
The `orbit` executables are downloaded from the [GoReleaser Orbit action](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-orbit.yaml).
Such action is triggered when git tagging a new orbit version with a tag of the form: `orbit-v1.15.0`.
> The following commands assume you are pushing version `1.15.0`.
```sh
cd /Volumes/FLEET-TUF/repository
cd /Volumes/FLEET-TUF
mkdir -p repository
mkdir -p keys
mkdir -p staged
aws s3 sync s3://fleet-tuf-repo ./repository
# macOS
fleetctl updates add --target /path/to/downloaded/macos/orbit --platform macos --name orbit --version 1.15.0 -t edge
# Linux
fleetctl updates add --target /path/to/downloaded/linux/orbit --platform linux --name orbit --version 1.15.0 -t edge
# Windows
fleetctl updates add --target /path/to/downloaded/windows/orbit.exe --platform windows --name orbit --version 1.15.0 -t edge
```
#### desktop
The Fleet Desktop executables are downloaded from the [Generate Fleet Desktop targets for Orbit action](https://github.com/fleetdm/fleet/actions/workflows/generate-desktop-targets.yml).
Such action is triggered by submitting a PR with the [following version string](https://github.com/fleetdm/fleet/blob/4a6bf0d447a2080f994da1e2f36ce6d51db88109/.github/workflows/generate-desktop-targets.yml#L27) changed.
> The following commands assume you are pushing version `1.15.0`.
```sh
# macOS
fleetctl updates add --target /path/to/macos/downloaded/desktop.app.tar.gz --platform macos --name desktop --version 1.15.0 -t edge
# Linux
fleetctl updates add --target /path/to/linux/downloaded/desktop.tar.gz --platform linux --name desktop --version 1.15.0 -t edge
# Windows
fleetctl updates add --target /path/to/windows/downloaded/fleet-desktop.exe --platform windows --name desktop --version 1.15.0 -t edge
```
#### swiftDialog
> macOS only component
The `swiftDialog` executable can be generated from a macOS host by running:
```sh
make swift-dialog-app-tar-gz version=2.2.1 build=4591 out-path=.
```
```sh
fleetctl updates add --target /path/to/macos/swiftDialog.app.tar.gz --platform macos --name swiftDialog --version 2.2.1 -t edge
```
#### nudge
> macOS only component
The `nudge` executable can be generated from a macOS host by running:
```sh
make nudge-app-tar-gz version=1.1.10.81462 out-path=.
```
```sh
fleetctl updates add --target /path/to/macos/nudge.app.tar.gz --platform macos --name nudge --version 1.1.10.81462 -t edge
```
#### osqueryd
Osquery executables are downloaded from the [Generate osqueryd targets for Fleetd action](https://github.com/fleetdm/fleet/blob/main/.github/workflows/generate-osqueryd-targets.yml).
Such action is triggered by submitting a PR with the [following version string](https://github.com/fleetdm/fleet/blob/7067ca586a4aa1a0377b387d4b4478a5958193ff/.github/workflows/generate-osqueryd-targets.yml#L27) changed.
> The following commands assume you are pushing version `5.9.1`.
```sh
# macOS
fleetctl updates add --target /path/to/downloaded/macos/osqueryd.app.tar.gz --platform macos-app --name osqueryd --version 5.9.1 -t edge
# Linux
fleetctl updates add --target /path/to/downloaded/linux/osqueryd --platform linux --name osqueryd --version 5.9.1 -t edge
# Windows
fleetctl updates add --target /path/to/downloaded/windows/osqueryd.exe --platform windows --name osqueryd --version 5.9.1 -t edge
```
### Promoting `edge` to the `stable` channel
> Make sure to install fleetd components using the `stable` channels in the three supported OSs (this is useful to smoke test the update).
Following is the list of components and each command for each operating system.
The commands show here update the local repository. After you are done running the commands below for each component, see [Pushing releases to Fleet's TUF repository](#pushing-releases-to-fleets-tuf-repository) to push the updates to Fleet's TUF repository (https://tuf.fleetctl.com).
#### orbit
> The following commands assume you are pushing version `1.15.0`.
```sh
# macOS
fleetctl updates add --target ./repository/targets/orbit/macos/edge/orbit --platform macos --name orbit --version 1.15.0 -t 1.15 -t 1 -t stable
# Linux
fleetctl updates add --target ./repository/targets/orbit/linux/edge/orbit --platform linux --name orbit --version 1.15.0 -t 1.15 -t 1 -t stable
# Windows
fleetctl updates add --target ./repository/targets/orbit/windows/edge/orbit.exe --platform windows --name orbit --version 1.15.0 -t 1.15 -t 1 -t stable
```
#### desktop
> The following commands assume you are pushing version `1.15.0`.
```sh
# macOS
fleetctl updates add --target ./repository/targets/desktop/macos/edge/desktop.app.tar.gz --platform macos --name desktop --version 1.15.0 -t 1.15 -t 1 -t stable
# Linux
fleetctl updates add --target ./repository/targets/desktop/linux/edge/desktop.tar.gz --platform linux --name desktop --version 1.15.0 -t 1.15 -t 1 -t stable
# Windows
fleetctl updates add --target ./repository/targets/desktop/windows/edge/fleet-desktop.exe --platform windows --name desktop --version 1.15.0 -t 1.15 -t 1 -t stable
```
#### swiftDialog
```sh
# macOS
fleetctl updates add --target ./repository/targets/swiftDialog/macos/edge/swiftDialog.app.tar.gz --platform macos --name swiftDialog --version 2.2.1 -t stable
```
#### nudge
```sh
# macOS
fleetctl updates add --target ./repository/targets/nudge/macos/edge/nudge.app.tar.gz --platform macos --name nudge --version 1.1.10.81462 -t stable
```
#### osqueryd
> The following commands assume you are pushing version `5.9.1`.
```sh
# macOS
fleetctl updates add --target ./repository/targets/osqueryd/macos-app/edge/osqueryd.app.tar.gz --platform macos-app --name osqueryd --version 5.9.1 -t 5.9 -t 5 -t stable
# Linux
fleetctl updates add --target ./repository/targets/osqueryd/linux/edge/osqueryd --platform linux --name osqueryd --version 5.9.1 -t 5.9 -t 5 -t stable
# Windows
fleetctl updates add --target ./repository/targets/osqueryd/windows/edge/osqueryd.exe --platform windows --name osqueryd --version 5.9.1 -t 5.9 -t 5 -t stable
```
#### Pushing releases to Fleet's TUF repository
Once you are done with the changes on your local repository, you can use the following command to review the changes before pushing (`--dryrun` allows us to verify the upgrade before pushing):
```sh
AWS_PROFILE=tuf aws s3 sync ./repository s3://fleet-tuf-repo --dryrun
(dryrun) upload: repository/snapshot.json to s3://fleet-tuf-repo/snapshot.json
(dryrun) upload: repository/targets.json to s3://fleet-tuf-repo/targets.json
[...]
(dryrun) upload: repository/timestamp.json to s3://fleet-tuf-repo/timestamp.json
```
If all looks good, run the same command without the `--dryrun` flag.
> NOTE: Some things to note after the changes are pushed:
> - Once pushed you might see some clients failing to upgrade due to some sha256 mismatches. These temporary failures are expected because it takes some time for caches to be invalidated (these errors should go away after a few minutes).
> - The auto-update routines in orbit run every one hour, so you might need to wait up to an hour to verify hosts are auto-updating properly.
## Removing Unused Targets
If you've inadvertently published a target that is no longer in use, follow these steps to remove it.
> Before performing any actions on Fleet's TUF repository you must:
> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository).
> 2. Create a local backup in case we mess up with the repository:
> ```sh
> mkdir ~/tuf.fleetctl.com/backup
> cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup
> ```
1. You'll need the [`go-tuf`](https://github.com/theupdateframework/go-tuf) binary. The removal operations aren't integrated into `fleetctl` at the moment.
2. Use `tuf remove` to remove the target and update `targets.json`. Substitute `desktop/windows/stable/desktop.exe` with the target you intend to delete.
```sh
tuf remove desktop/windows/stable/desktop.exe
```
3. Snapshot, timestamp, and commit the changes.
```sh
tuf snapshot
tuf timestamp
tuf commit
```
4. Run the following command to generate a timestamp that expires in two weeks (otherwise the default expiration when using `go-tuf` commands is 1 day)
```sh
fleetctl updates timestamp
```
5. Confirm that the version of the local `timestamp.json` file is more recent than that of the remote server.
6. Verify the changes that will be synced by running a dry sync. Include the `--delete` flag as you're removing targets.
```sh
aws s3 sync ./repository s3://fleet-tuf-repo --delete --dryrun
```
7. `diff` the local `targets.json` file with its remote version.
8. To upload the changes, perform a sync without the `--dryrun`:
```sh
aws s3 sync ./repository s3://fleet-tuf-repo --delete
```
## Becoming a New Fleet Publisher
> Before performing any actions on Fleet's TUF repository you must:
> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository).
> 2. Create a local backup in case we mess up with the repository:
> ```sh
> mkdir ~/tuf.fleetctl.com/backup
> cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup
> ```
### Generate targets+snapshot+timestamp keys
All commands shown in this guide are executed from `/Volumes/FLEET-TUF`:
All commands shown in this guide are executed from `/path/to/tuf.fleetctl.com`:
```sh
cd /Volumes/FLEET-TUF
cd /path/to/tuf.fleetctl.com
```
```sh
@ -65,200 +286,12 @@ Essentially the following commands are executed to sign the new keys:
- `tuf timestamp`
- `tuf commit`
## Pushing new updates
## Misc issues
Following are tested steps to push new targets.
### Invalid timestamp.json version
### 1. Backup current TUF repository
Just in case we break the remote TUF directory, let's do a local backup:
```sh
mkdir ~/tuf.fleetctl.com/backup
aws s3 sync s3://fleet-tuf-repo ~/tuf.fleetctl.com/backup
```
### 2. Make sure the local repository is up-to-date
```sh
aws s3 sync s3://fleet-tuf-repo ./repository
```
### 3. Setup Orbit in Linux, Windows, macOS
Install Orbit with the (to be updated) channels in the three supported OSs.
E.g. if we need to push a new version of `osqueryd` to `edge`, then generate and install a package with:
```sh
fleetctl package ... --osqueryd-channel=edge ...
```
### 4. Setup Orbit in one host that points to our repository
This allows us to verify that already running clients will upgrade successfully.
Serve current unmodified repository:
```sh
cd repository && python3 -m http.server
```
Generate packages using the local TUF server (in my case on a macOS host):
```sh
fleetctl package --type=pkg --update-url=http://localhost:8000 ...
[...]
```
Install generated `fleet-osquery.pkg`.
### 5. Actually pushing new updates
In this example we are promoting orbit from `edge` to `stable`:
```sh
./fleetctl-macos updates add --target ./repository/targets/orbit/linux/edge/orbit --platform linux --name orbit --version 1.1.0 -t 1.1 -t 1 -t stable
[...]
./fleetctl-macos updates add --target ./repository/targets/orbit/windows/edge/orbit.exe --platform windows --name orbit --version 1.1.0 -t 1.1 -t 1 -t stable
[...]
./fleetctl-macos updates add --target ./repository/targets/orbit/macos/edge/orbit --platform macos --name orbit --version 1.1.0 -t 1.1 -t 1 -t stable
[...]
```
`--dryrun` allows us to verify the upgrade before pushing:
```sh
AWS_PROFILE=tuf aws s3 sync ./repository s3://fleet-tuf-repo --dryrun
(dryrun) upload: repository/snapshot.json to s3://fleet-tuf-repo/snapshot.json
(dryrun) upload: repository/targets.json to s3://fleet-tuf-repo/targets.json
(dryrun) upload: repository/targets/orbit/linux/1.1.0/orbit to s3://fleet-tuf-repo/targets/orbit/linux/1.1.0/orbit
(dryrun) upload: repository/targets/orbit/linux/1.1/orbit to s3://fleet-tuf-repo/targets/orbit/linux/1.1/orbit
(dryrun) upload: repository/targets/orbit/linux/1/orbit to s3://fleet-tuf-repo/targets/orbit/linux/1/orbit
(dryrun) upload: repository/targets/orbit/linux/stable/orbit to s3://fleet-tuf-repo/targets/orbit/linux/stable/orbit
(dryrun) upload: repository/targets/orbit/macos/1.1.0/orbit to s3://fleet-tuf-repo/targets/orbit/macos/1.1.0/orbit
(dryrun) upload: repository/targets/orbit/macos/1.1/orbit to s3://fleet-tuf-repo/targets/orbit/macos/1.1/orbit
(dryrun) upload: repository/targets/orbit/macos/1/orbit to s3://fleet-tuf-repo/targets/orbit/macos/1/orbit
(dryrun) upload: repository/targets/orbit/macos/stable/orbit to s3://fleet-tuf-repo/targets/orbit/macos/stable/orbit
(dryrun) upload: repository/targets/orbit/windows/1.1.0/orbit.exe to s3://fleet-tuf-repo/targets/orbit/windows/1.1.0/orbit.exe
(dryrun) upload: repository/targets/orbit/windows/1.1/orbit.exe to s3://fleet-tuf-repo/targets/orbit/windows/1.1/orbit.exe
(dryrun) upload: repository/targets/orbit/windows/1/orbit.exe to s3://fleet-tuf-repo/targets/orbit/windows/1/orbit.exe
(dryrun) upload: repository/targets/orbit/windows/stable/orbit.exe to s3://fleet-tuf-repo/targets/orbit/windows/stable/orbit.exe
(dryrun) upload: repository/timestamp.json to s3://fleet-tuf-repo/timestamp.json
```
In this other example we are updating osquery's `edge` channel:
```sh
./fleetctl-macos updates add --target /Users/luk/Downloads/tuf-osqueryd/osqueryd.exe --platform windows --name osqueryd --version 5.5.1 -t edge
[...]
./fleetctl-macos updates add --target /Users/luk/Downloads/tuf-osqueryd/osqueryd.app.tar.gz --platform macos-app --name osqueryd --version 5.5.1 -t edge
[...]
./fleetctl-macos updates add --target /Users/luk/Downloads/tuf-osqueryd/osqueryd --platform linux --name osqueryd --version 5.5.1 -t edge
[...]
```
`--dryrun` allows us to verify the upgrade before pushing:
```sh
aws s3 sync ./repository s3://fleet-tuf-repo --profile tuf --dryrun
(dryrun) upload: repository/snapshot.json to s3://fleet-tuf-repo/snapshot.json
(dryrun) upload: repository/targets.json to s3://fleet-tuf-repo/targets.json
(dryrun) upload: repository/targets/osqueryd/linux/5.5.1/osqueryd to s3://fleet-tuf-repo/targets/osqueryd/linux/5.5.1/osqueryd
(dryrun) upload: repository/targets/osqueryd/linux/edge/osqueryd to s3://fleet-tuf-repo/targets/osqueryd/linux/edge/osqueryd
(dryrun) upload: repository/targets/osqueryd/macos-app/5.5.1/osqueryd.app.tar.gz to s3://fleet-tuf-repo/targets/osqueryd/macos-app/5.5.1/osqueryd.app.tar.gz
(dryrun) upload: repository/targets/osqueryd/macos-app/edge/osqueryd.app.tar.gz to s3://fleet-tuf-repo/targets/osqueryd/macos-app/edge/osqueryd.app.tar.gz
(dryrun) upload: repository/targets/osqueryd/windows/5.5.1/osqueryd.exe to s3://fleet-tuf-repo/targets/osqueryd/windows/5.5.1/osqueryd.exe
(dryrun) upload: repository/targets/osqueryd/windows/edge/osqueryd.exe to s3://fleet-tuf-repo/targets/osqueryd/windows/edge/osqueryd.exe
(dryrun) upload: repository/timestamp.json to s3://fleet-tuf-repo/timestamp.json
```
### 6. Verify the already running test host
Verify host enrolled in step (4) upgraded to the new versions successfully.
### 7. Verify generation of new packages
```sh
fleetctl package --type=pkg --update-url=http://localhost:8000 ...
fleetctl package --type=msi --update-url=http://localhost:8000 ...
fleetctl package --type=deb --update-url=http://localhost:8000 ...
```
### 8. Push!
Run the same command shown above, but without `--dryrun`
```sh
aws s3 sync ./repository s3://fleet-tuf-repo --profile tuf
```
### 9. Final Verification
Now that the repository is pushed, verify that the hosts enrolled in step (3) update as expected.
### Removing Unused Targets
If you've inadvertently published a target that is no longer in use, follow these steps to remove it.
### 1. Preparation
1. Backup the remote TUF directory to your local machine.
```sh
mkdir ~/tuf.fleetctl.com/backup
aws s3 sync s3://fleet-tuf-repo ~/tuf.fleetctl.com/backup
```
2. Ensure your local repository mirrors the current state.
```sh
aws s3 sync s3://fleet-tuf-repo ./repository
```
3. You'll need the [`go-tuf`](https://github.com/theupdateframework/go-tuf) binary. The removal operations aren't integrated into `fleetctl` at the moment.
### 2. Local Target Removal
1. Use `tuf remove` to remove the target and update `targets.json`. Substitute `desktop/windows/stable/desktop.exe` with the target you intend to delete.
```sh
tuf remove desktop/windows/stable/desktop.exe
```
2. Snapshot, timestamp, and commit the changes.
```sh
tuf snapshot
tuf timestamp
tuf commit
```
### 3. Verification Before Publishing
1. Confirm that the version of the local `timestamp.json` file is more recent than that of the remote server.
2. Verify the changes that will be synced by running a dry sync. Include the `--delete` flag as you're removing targets.
```sh
aws s3 sync ./repository s3://fleet-tuf-repo --delete --dryrun
```
3. `diff` the local `targets.json` file with its remote version.
### 4. Publish and Confirm
1. To upload the changes, perform a sync without the `--dryrun`.
```sh
aws s3 sync ./repository s3://fleet-tuf-repo --delete
```
2. Check the changes on the remote repository.
### Issues found
#### Invalid timestamp.json version
The following issue was solved by resigning the timestamp metadata `fleetctl updates timestamp` (executed three times to increase the version to 4175)
The following issue was solved by resigning the timestamp metadata `fleetctl updates timestamp` (executed three times to increase the version to `4175`)
```sh
2022-08-23T13:44:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174"
2022-08-23T13:59:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174"
```
## Notes
- "Measure thrice cut once": Steps 3, 4, 5 and 6 allows us to verify the repository is in good shape before pushing to [Fleet's TUF service](tuf.fleetctl.com).
- Steps may look different if the upgrade is performed on a Linux or Windows host.
```