fleet/orbit/docs/TUF-Update-Guide.md

# Pushing new releases to TUF

This document is a walkthrough guide for:
- A Fleet member to publish new updates to [Fleet's TUF service](tuf.fleetctl.com). See [Pushing updates](#pushing-updates).
- A Fleet member to delete targets from [Fleet's TUF service](tuf.fleetctl.com). See [Removing unused targets](#removing-unused-targets).
- A Fleet member to become a publisher of updates for [Fleet's TUF service](tuf.fleetctl.com). See [Becoming a new Fleet publisher](#becoming-a-new-fleet-publisher).

Video walkthrough related to this process for additional [context](https://drive.google.com/file/d/1c_iukFEMne12Cxx9WVTt_j1Wp0sC1kQU/view?usp=drive_link).

## Security

- The TUF keys for `targets`, `snapshot` and `timestamp` should be stored on a USB stick (used solely for this purpose). Whenever you need to push updates to Fleet's TUF repository you can temporarily copy the encrypted keys to your workstation (under the `keys/` folder, more on this below).
- The keys should be stored encrypted with its passphrase stored in 1Password (on a private vault).
- Every `fleetctl updates` command will prompt for the passphrases to decrypt the encrypted keys. You can input the passphrases every time or can alternatively set the following environment variables: `FLEET_TIMESTAMP_PASSPHRASE`, `FLEET_SNAPSHOT_PASSPHRASE` and `FLEET_TARGETS_PASSPHRASE`. Make sure to not leave traces of the passphrases (scripts, history and/or environment) when you are done.

## Syncing Fleet's TUF repository

> The `fleetctl updates` commands assume the folders `keys/`, `staged/` and `repository/` exist on the current working directory.

> IMPORTANT: When syncing the repository make sure to use `--exact-timestamps`. Otherwise `aws s3 sync` may not sync files that do not change in size, like `timestamp.json`.

- The `keys/` folder contains the encrypted private keys.
- The `staged/` folder contains uncommitted changes (usually empty because `fleetctl updates` commands automatically commit the changes).
- The `repository/` folder contains the full TUF repository.

Following are the commands to initialize the repository on your workstation:
```sh
mkdir /path/to/tuf.fleetctl.com

cd /path/to/tuf.fleetctl.com
mkdir -p ./repository
cp /Volumes/YOUR-USB-NAME/keys ./keys
mkdir -p ./staged

aws s3 sync s3://fleet-tuf-repo ./repository --exact-timestamps
```

## Pushing updates

> Before performing any actions on Fleet's TUF repository you must:
> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository).
> 2. Create a local backup in case we mess up with the repository:
>    ```sh
>    mkdir ~/tuf.fleetctl.com/backup
>    cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup
>    ```

### Releasing to the `edge` channel

> Make sure to install fleetd components using the `edge` channels in the three supported OSs (this is useful to smoke test the update).

Following is the list of components and each command for each operating system.

The commands show here update the local repository. After you are done running the commands below for each component, see [Pushing releases to Fleet's TUF repository](#pushing-releases-to-fleets-tuf-repository) to push the updates to Fleet's TUF repository (https://tuf.fleetctl.com).

#### orbit

The `orbit` executables are downloaded from the [GoReleaser Orbit action](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-orbit.yaml).
Such action is triggered when git tagging a new orbit version with a tag of the form: `orbit-v1.15.0`.
    
> The following commands assume you are pushing version `1.15.0`.

```sh
# macOS
fleetctl updates add --target /path/to/downloaded/macos/orbit --platform macos --name orbit --version 1.15.0 -t edge
# Linux
fleetctl updates add --target /path/to/downloaded/linux/orbit --platform linux --name orbit --version 1.15.0 -t edge
# Windows
fleetctl updates add --target /path/to/downloaded/windows/orbit.exe --platform windows --name orbit --version 1.15.0 -t edge
```

#### desktop 

The Fleet Desktop executables are downloaded from the [Generate Fleet Desktop targets for Orbit action](https://github.com/fleetdm/fleet/actions/workflows/generate-desktop-targets.yml).
Such action is triggered by submitting a PR with the [following version string](https://github.com/fleetdm/fleet/blob/4a6bf0d447a2080f994da1e2f36ce6d51db88109/.github/workflows/generate-desktop-targets.yml#L27) changed.

> The following commands assume you are pushing version `1.15.0`.

```sh
# macOS
fleetctl updates add --target /path/to/macos/downloaded/desktop.app.tar.gz --platform macos --name desktop --version 1.15.0 -t edge
# Linux
fleetctl updates add --target /path/to/linux/downloaded/desktop.tar.gz --platform linux --name desktop --version 1.15.0 -t edge
# Windows
fleetctl updates add --target /path/to/windows/downloaded/fleet-desktop.exe --platform windows --name desktop --version 1.15.0 -t edge
```

#### swiftDialog

> macOS only component

The `swiftDialog` executable can be generated from a macOS host by running:
```sh
make swift-dialog-app-tar-gz version=2.2.1 build=4591 out-path=.
```

```sh
fleetctl updates add --target /path/to/macos/swiftDialog.app.tar.gz --platform macos --name swiftDialog --version 2.2.1 -t edge
```

#### nudge

> macOS only component

The `nudge` executable can be generated from a macOS host by running:
```sh
make nudge-app-tar-gz version=1.1.10.81462 out-path=.
```

```sh
fleetctl updates add --target /path/to/macos/nudge.app.tar.gz --platform macos --name nudge --version 1.1.10.81462 -t edge
```

#### osqueryd

Osquery executables are downloaded from the [Generate osqueryd targets for Fleetd action](https://github.com/fleetdm/fleet/blob/main/.github/workflows/generate-osqueryd-targets.yml).
Such action is triggered by submitting a PR with the [following version string](https://github.com/fleetdm/fleet/blob/7067ca586a4aa1a0377b387d4b4478a5958193ff/.github/workflows/generate-osqueryd-targets.yml#L27) changed.

> The following commands assume you are pushing version `5.9.1`.

```sh
# macOS
fleetctl updates add --target /path/to/downloaded/macos/osqueryd.app.tar.gz --platform macos-app --name osqueryd --version 5.9.1 -t edge
# Linux
fleetctl updates add --target /path/to/downloaded/linux/osqueryd --platform linux --name osqueryd --version 5.9.1 -t edge
# Windows
fleetctl updates add --target /path/to/downloaded/windows/osqueryd.exe --platform windows --name osqueryd --version 5.9.1 -t edge
```

#### Push updates

Once all components are updated in your local repository we need to push the changes to the remote repository.
See [Pushing releases to Fleet's TUF repository](#pushing-releases-to-fleets-tuf-repository).

### Promoting `edge` to the `stable` channel

> Make sure to install fleetd components using the `stable` channels in the three supported OSs (this is useful to smoke test the update).

Following is the list of components and each command for each operating system.

The commands show here update the local repository. After you are done running the commands below for each component, see [Pushing releases to Fleet's TUF repository](#pushing-releases-to-fleets-tuf-repository) to push the updates to Fleet's TUF repository (https://tuf.fleetctl.com).

#### orbit

> The following command assumes you are pushing version `1.15.0`:
```sh
/fleet/repo/tools/tuf/promote_edge_to_stable.sh orbit 1.15.0
```

#### desktop

> The following command assumes you are pushing version `1.15.0`:
```sh
/fleet/repo/tools/tuf/promote_edge_to_stable.sh desktop 1.15.0
```

#### swiftDialog

> The following command assumes you are pushing version `2.2.1`:
```sh
/fleet/repo/tools/tuf/promote_edge_to_stable.sh swiftDialog 2.2.1
```

#### nudge

> The following command assumes you are pushing version `1.1.10.81462`:
```sh
/fleet/repo/tools/tuf/promote_edge_to_stable.sh nudge 1.1.10.81462
```

#### osqueryd

> The following command assumes you are pushing version `5.9.1`.
```sh
/fleet/repo/tools/tuf/promote_edge_to_stable.sh osqueryd 5.9.1
```

#### Push updates

Once all components are updated in your local repository we need to push the changes to the remote repository.
See [Pushing releases to Fleet's TUF repository](#pushing-releases-to-fleets-tuf-repository).

### Pushing releases to Fleet's TUF repository

Once you are done with the changes on your local repository, you can use the following command to review the changes before pushing (`--dryrun` allows us to verify the upgrade before pushing):
```sh
AWS_PROFILE=tuf aws s3 sync ./repository s3://fleet-tuf-repo --dryrun
(dryrun) upload: repository/snapshot.json to s3://fleet-tuf-repo/snapshot.json
(dryrun) upload: repository/targets.json to s3://fleet-tuf-repo/targets.json
[...]
(dryrun) upload: repository/timestamp.json to s3://fleet-tuf-repo/timestamp.json
```

If all looks good, run the same command without the `--dryrun` flag.

> NOTE: Some things to note after the changes are pushed:
>   - Once pushed you might see some clients failing to upgrade due to some sha256 mismatches. These temporary failures are expected because it takes some time for caches to be invalidated (these errors should go away after a few minutes).
>   - The auto-update routines in orbit runs every 15 minutes, so you might need to wait up to 15 minutes to verify online hosts are auto-updating properly.

## Removing Unused Targets

If you've inadvertently published a target that is no longer in use, follow these steps to remove it.

> Before performing any actions on Fleet's TUF repository you must:
> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository).
> 2. Create a local backup in case we mess up with the repository:
>    ```sh
>    mkdir ~/tuf.fleetctl.com/backup
>    cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup
>    ```

1. You'll need the [`go-tuf`](https://github.com/theupdateframework/go-tuf) binary. The removal operations aren't integrated into `fleetctl` at the moment.
2. Use `tuf remove` to remove the target and update `targets.json`. Substitute `desktop/windows/stable/desktop.exe` with the target you intend to delete.
```sh
tuf remove desktop/windows/stable/desktop.exe
```
3. Snapshot, timestamp, and commit the changes.
```sh
tuf snapshot
tuf timestamp
tuf commit
```
4. Run the following command to generate a timestamp that expires in two weeks (otherwise the default expiration when using `go-tuf` commands is 1 day) 
```sh
fleetctl updates timestamp
```
5. Confirm that the version of the local `timestamp.json` file is more recent than that of the remote server.
6. Verify the changes that will be synced by running a dry sync. Include the `--delete` flag as you're removing targets.
```sh
aws s3 sync ./repository s3://fleet-tuf-repo --delete --dryrun
```
7. `diff` the local `targets.json` file with its remote version.
8. To upload the changes, perform a sync without the `--dryrun`:
```sh
aws s3 sync ./repository s3://fleet-tuf-repo --delete
```

## Becoming a New Fleet Publisher

> Before performing any actions on Fleet's TUF repository you must:
> 1. Make sure your local copy of the repository is up-to-date. See [Syncing Fleet's TUF repository](#syncing-fleets-tuf-repository).
> 2. Create a local backup in case we mess up with the repository:
>    ```sh
>    mkdir ~/tuf.fleetctl.com/backup
>    cp -r ~/tuf.fleetctl.com ~/tuf.fleetctl.com-backup
>    ```

### Generate targets+snapshot+timestamp keys

All commands shown in this guide are executed from `/path/to/tuf.fleetctl.com`:
```sh
cd /path/to/tuf.fleetctl.com
```

```sh
tuf gen-key targets
Enter targets keys passphrase:
Repeat targets keys passphrase:
Generated targets key with ID ae943cb8be8a849b37c66ed46bdd7e905ba3118c0c051a6ee3cd30625855a076
```
```sh
tuf gen-key snapshot
Enter snapshot keys passphrase:
Repeat snapshot keys passphrase:
Generated snapshot key with ID 1a4d9beb826d1ff4e036d757cfcd6e36d0f041e58d25f99ef3a20ae3f8dd71e3
```
```sh
tuf gen-key timestamp
Enter timestamp keys passphrase:
Repeat timestamp keys passphrase:
Generated timestamp key with ID d940df08b59b12c30f95622a05cc40164b78a11dd7d408395ee4f79773331b30
```

Share `staged/root.json` with Fleet member with the `root` role, who will sign with its root key and push to the repository.

### Root role signs the `staged/root.json`

Essentially the following commands are executed to sign the new keys:
- `tuf sign`
- `tuf snapshot`
- `tuf timestamp`
- `tuf commit`

## Misc issues

### Invalid timestamp.json version

The following issue was solved by resigning the timestamp metadata `fleetctl updates timestamp` (executed three times to increase the version to `4175`)
```sh
2022-08-23T13:44:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174"
2022-08-23T13:59:48-03:00 INF update failed error="update metadata: update metadata: tuf: failed to decode timestamp.json: version 4172 is lower than current version 4174"
```