Improve Orbit auto-update test guide (#10591)

Before we can automate (https://github.com/fleetdm/fleet/issues/10334), we need to have good docs on how to perform the testing steps manually.
2026-05-23 08:58:41 +00:00 · 2023-03-24 14:38:39 -03:00 · 2023-03-24 14:38:39 -03:00 · 99c0eef4ef
commit 99c0eef4ef
parent e367c77246
2 changed files with 94 additions and 51 deletions
--- a/tools/tuf/test/Orbit-auto-update-test-guide.md
+++ b/tools/tuf/test/Orbit-auto-update-test-guide.md
@ -5,7 +5,7 @@
 To test Orbit we must have a setup for the three OSs where we support Orbit: macOS, Linux and Windows.

 This guide assumes:
- A macOS host OS, where we'll run most of the commands, TUF server, Orbit and the Fleet server.
+- A macOS Intel workstation (host), where we'll run most of the commands, TUF server, Orbit and the Fleet server.
 - Two VMWare VMs, with Windows 10 and Ubuntu 22.04, where we'll run Orbit.
 - The two guest OSs will connect to the host OS via the `host.docker.internal` hostname.
 To do this, you can add an entry like `192.168.103.1	host.docker.internal` to the `hosts` file in the VMs
@ -17,8 +17,9 @@ To do this, you can add an entry like `192.168.103.1	host.docker.internal` to th
 Head over to https://github.com/fleetdm/fleet/releases and grab the git tag of the last releases for Fleet and Orbit.

 At the time of writing:
- Last Orbit release: `orbit-v1.2.0`
- Last Fleet release: `fleet-v4.21.0`
+- Last Orbit release: `orbit-v1.2.0`.
+- Last Fleet release: `fleet-v4.21.0`.
+- Last osquery version: `5.8.1`. (In the steps below we will use `5.6.0` and `5.7.0` because we need to test auto-updating two times.)

 ## Run Fleet

@ -64,6 +65,7 @@ GENERATE_MSI=1 \
 ENROLL_SECRET=$ENROLL_SECRET \
 FLEET_DESKTOP=1 \
 FLEET_CERTIFICATE=1 \
+OSQUERY_VERSION=5.6.0 \
 ./tools/tuf/test/main.sh
 ```

@ -93,13 +95,13 @@ cp fleet-osquery_42.0.0_amd64.deb ~/shared-ubuntu
  sudo dpkg --install fleet-osquery_42.0.0_amd64.deb
  ```

-5. Verify three hosts have enrolled (by running `./build/fleetctl get hosts` or using the browser).
+5. Verify the three hosts have enrolled (by running `./build/fleetctl get hosts` or using the browser).

 6. Verify the three Fleet Desktop instances are working, by clicking the "My device" menu item on the three OSs.

-## New Orbit release
+## New releases

-1. Now let's "release" new Orbit + Fleet Desktop version (via auto-update) by using latest `main`.
+1. Now let's "release" a new version of the three fleetd components (orbit, Fleet Desktop and osquery) via auto-update by using latest `main`:

 ```sh
 git checkout main
@ -109,14 +111,11 @@ git checkout main

 ```sh
 # Compile a new version of Orbit for Windows:
-GOOS=windows GOARCH=amd64 go build -o orbit-windows.exe ./orbit/cmd/orbit
+GOOS=windows GOARCH=amd64 go build -ldflags="-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Version=43" -o orbit-windows.exe ./orbit/cmd/orbit
 # Push the compiled Orbit as a new version
 ./tools/tuf/test/push_target.sh windows orbit orbit-windows.exe 43
 ```

-Wait for ~1m for all Windows hosts to auto-update Orbit.
-Verify the Windows Fleet Desktop instances are working, by visiting "My device".
-
 ```sh
 # Compile a new version of fleet-desktop for Windows:
 FLEET_DESKTOP_VERBOSE=1 FLEET_DESKTOP_VERSION=43.0.0 make desktop-windows
@ -124,18 +123,23 @@ FLEET_DESKTOP_VERBOSE=1 FLEET_DESKTOP_VERSION=43.0.0 make desktop-windows
 ./tools/tuf/test/push_target.sh windows desktop fleet-desktop.exe 43
 ```

+```sh
+# Download version from our main TUF:
+curl https://tuf.fleetctl.com/targets/osqueryd/windows/5.7.0/osqueryd.exe --output osqueryd.exe
+
+# Push the osqueryd target as a new version
+./tools/tuf/test/push_target.sh windows osqueryd osqueryd.exe 5.7.0
+```
+
 ### Linux

 ```sh
 # Compile a new version of Orbit for Linux:
-GOOS=linux GOARCH=amd64 go build -o orbit-linux ./orbit/cmd/orbit
+GOOS=linux GOARCH=amd64 go build -ldflags="-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Version=43" -o orbit-linux ./orbit/cmd/orbit
 # Push the compiled Orbit as a new version
 ./tools/tuf/test/push_target.sh linux orbit orbit-linux 43
 ```

-Wait for ~1m for all Linux hosts to auto-update Orbit.
-Verify the Linux Fleet Desktop instances are working, by visiting "My device", and hit "Refresh" in the "My device" page.
-
 ```sh
 # Compile a new version of fleet-desktop for Linux:
 FLEET_DESKTOP_VERBOSE=1 FLEET_DESKTOP_VERSION=43.0.0 make desktop-linux
@ -143,18 +147,23 @@ FLEET_DESKTOP_VERBOSE=1 FLEET_DESKTOP_VERSION=43.0.0 make desktop-linux
 ./tools/tuf/test/push_target.sh linux desktop desktop.tar.gz 43
 ```

+```sh
+# Download version from our main TUF:
+curl https://tuf.fleetctl.com/targets/osqueryd/linux/5.7.0/osqueryd --output osqueryd-linux
+
+# Push the osqueryd target as a new version
+./tools/tuf/test/push_target.sh linux osqueryd osqueryd-linux 5.7.0
+```
+
 ### macOS

 ```sh
 # Compile a new version of Orbit for macOS:
-GOOS=darwin GOARCH=amd64 go build -o orbit-darwin ./orbit/cmd/orbit
+GOOS=darwin GOARCH=amd64 go build -ldflags="-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Version=43" -o orbit-darwin ./orbit/cmd/orbit
 # Push the compiled Orbit as a new version
 ./tools/tuf/test/push_target.sh macos orbit orbit-darwin 43
 ```

-Wait for ~1m for all macOS hosts to auto-update Orbit.
-Verify the macOS Fleet Desktop instances are working, by visiting "My device", and hit "Refresh" in the "My device" page.
-
 ```sh
 # Compile a new version of fleet-desktop for macOS:
 FLEET_DESKTOP_VERBOSE=1 FLEET_DESKTOP_VERSION=43.0.0 make desktop-app-tar-gz
@ -162,11 +171,51 @@ FLEET_DESKTOP_VERBOSE=1 FLEET_DESKTOP_VERSION=43.0.0 make desktop-app-tar-gz
 ./tools/tuf/test/push_target.sh macos desktop desktop.app.tar.gz 43
 ```

+```sh
+# Download version from our main TUF.
+curl https://tuf.fleetctl.com/targets/osqueryd/macos-app/5.7.0/osqueryd.app.tar.gz --output osqueryd.app.tar.gz
+
+# Push the osqueryd target as a new version
+./tools/tuf/test/push_target.sh macos-app osqueryd osqueryd.app.tar.gz 5.7.0
+```
+
 2. Wait for ~1m for all hosts to fully auto-update.

-3. Verify all hosts now show "Fleet Desktop v43.0.0" on the Fleet Desktop menu.
+3. Run the following live query on all hosts: `SELECT * FROM orbit_info;`. The query should return `version=43`.

-4. Verify the three Fleet Desktop instances are working, by visiting "My device", and hit "Refresh" in the "My device" page.
+4. Verify all hosts now show "Fleet Desktop v43.0.0" on the Fleet Desktop menu.
+
+5. Verify the Fleet Desktop instances are working, by visiting "My device", and hit "Refresh" in the "My device" page.
+
+6. Run the following live query on all hosts: `SELECT * FROM osquery_info;`. The query should return `version=5.7.0`.
+
+## Test Orbit N + 1
+
+We need to test that the `main` (to-be-released) version of Orbit has not broken the auto-update mechanism. So we'll test upgrading to a new hypothetical future version "44".
+Run the above tests again on the three OSs but by compiling orbit and Fleet Desktop differently:
+
+- When compiling Orbit, change the following, from:
+  ```sh
+  -ldflags="-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Version=43"
+  ```
+  to
+  ```sh
+  -ldflags="-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Version=44"
+  ```
+- When compiling Fleet Desktop, change the following, from:
+  ```sh
+  FLEET_DESKTOP_VERSION=43.0.0
+  ```
+  to
+  ```sh
+  FLEET_DESKTOP_VERSION=44.0.0
+  ```
+- When pushing osquery, change `5.7.0` to `5.8.1`.
+
+
+1. Run the following live query on the hosts: `SELECT * FROM orbit_info;`. The query should now return `version=44`.
+2. Run the following live query on the hosts: `SELECT * FROM osquery_info;`. The query should now return `version=5.8.1`.
+3. Verify all hosts now show "Fleet Desktop v44.0.0" on the Fleet Desktop menu.

 ## New Fleet release

--- a/tools/tuf/test/create_repository.sh
+++ b/tools/tuf/test/create_repository.sh
@ -13,7 +13,6 @@ set -e
 # FLEET_TIMESTAMP_PASSPHRASE: Timestamp role passphrase.
 # SYSTEMS: Space separated list of systems to support in the TUF repository. Default value is: "macos windows linux"
 # MACOS_USE_PREBUILT_DESKTOP_APP_TAR_GZ: Set variable to use a pre-built desktop.app.tar.gz. Useful when running on non-macOS host.
-# MACOS_USE_PREBUILT_OSQUERYD_APP_TAR_GZ: Set variable to use a pre-built osqueryd.app.tar.gz. Useful when running on non-macOS host.

 if [[ -z "$TUF_PATH" ]]; then
    echo "Must set the TUF_PATH environment variable."
@ -24,45 +23,40 @@ if [[ -d "$TUF_PATH" ]]; then
    exit 0
 fi

-OSQUERY_MACOS_APP_BUNDLE_VERSION=5.7.0
 NUDGE_MACOS_APP_BUNDLE_VERSION=1.1.10.81462
 SYSTEMS=${SYSTEMS:-macos linux windows}

+if [[ -z "$OSQUERY_VERSION" ]]; then
+    OSQUERY_VERSION=5.8.1
+fi
+
 mkdir -p $TUF_PATH/tmp

 ./build/fleetctl updates init --path $TUF_PATH

 for system in $SYSTEMS; do

-    if [[ $system == "macos" ]]; then
-        if [[ -z "$MACOS_USE_PREBUILT_OSQUERYD_APP_TAR_GZ" ]]; then
-            # Generate and add osqueryd .app bundle for macos-app.
-            make osqueryd-app-tar-gz version=$OSQUERY_MACOS_APP_BUNDLE_VERSION out-path=.
-        fi
-        ./build/fleetctl updates add \
-            --path $TUF_PATH \
-            --target osqueryd.app.tar.gz \
-            --platform macos-app \
-            --name osqueryd \
-            --version 42.0.0 -t 42.0 -t 42 -t stable
-        rm osqueryd.app.tar.gz
-    else
-        # Use latest stable version of osqueryd from our TUF server.
-        osqueryd="osqueryd"
-        if [[ $system == "windows" ]]; then
-            osqueryd="$osqueryd.exe"
-        fi
-        osqueryd_path="$TUF_PATH/tmp/$osqueryd"
-        curl https://tuf.fleetctl.com/targets/osqueryd/$system/stable/$osqueryd --output $osqueryd_path
-
-        ./build/fleetctl updates add \
-            --path $TUF_PATH \
-            --target $osqueryd_path \
-            --platform $system \
-            --name osqueryd \
-            --version 42.0.0 -t 42.0 -t 42 -t stable
-        rm $osqueryd_path
+    # Use latest stable version of osqueryd from our TUF server.
+    osqueryd="osqueryd"
+    osqueryd_system="$system"
+    if [[ $system == "windows" ]]; then
+        osqueryd="$osqueryd.exe"
+    elif [[ $system == "macos" ]]; then
+        osqueryd="$osqueryd.app.tar.gz"
+        osqueryd_system="macos-app"
    fi
+    osqueryd_path="$TUF_PATH/tmp/$osqueryd"
+    curl https://tuf.fleetctl.com/targets/osqueryd/$osqueryd_system/$OSQUERY_VERSION/$osqueryd --output $osqueryd_path
+
+    major=$(echo "$OSQUERY_VERSION" | cut -d "." -f 1)
+    min=$(echo "$OSQUERY_VERSION" | cut -d "." -f 2)
+    ./build/fleetctl updates add \
+        --path $TUF_PATH \
+        --target $osqueryd_path \
+        --platform $osqueryd_system \
+        --name osqueryd \
+        --version $OSQUERY_VERSION -t $major.$min -t $major -t stable
+    rm $osqueryd_path

    goose_value="$system"
    if [[ $system == "macos" ]]; then
@ -74,7 +68,7 @@ for system in $SYSTEMS; do
    fi

    # Compile the latest version of orbit from source.
-    GOOS=$goose_value GOARCH=amd64 go build -o $orbit_target ./orbit/cmd/orbit
+    GOOS=$goose_value GOARCH=amd64 go build -ldflags="-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Version=42" -o $orbit_target ./orbit/cmd/orbit

    # If macOS and CODESIGN_IDENTITY is defined, sign the executable.
    if [[ $system == "macos" && -n "$CODESIGN_IDENTITY" ]]; then