fleet/.github/ISSUE_TEMPLATE/release-qa.md

name	about	title	labels	assignees
Release QA	Checklist of required tests prior to release	Release QA:	#g-mdm,#g-orchestration,#g-software,#g-security-compliance,:release	xpkoala,andreykizimenko,chrstphr84,Brajim20

Goal: easy-to-follow test steps for checking a release manually

Important reference data

1. fleetctl preview setup
2. Permissions documentation
3. Fleet free vs premium documentation

Smoke Tests

Smoke tests are limited to core functionality and serve as a pre-release final review. If smoke tests are failing, a release cannot proceed.

Fleet core:

Fleet version (Head to the "My account" page in the Fleet UI or run fleetctl version):

Web browser (e.g. Chrome 88.0.4324):

Prerequisites

1. Local instance is running and up to date with the target release branch
2. In your browser, clear local storage using devtools.

Orchestration

Test name	Step instructions	Expected result	pass/fail
$Name	{what a tester should do}	{what a tester should see when they do that}	pass/fail
Update flow	remove all fleet processes/agents/etc using fleetctl preview reset for a clean slate run fleetctl preview with no tag for latest stable create a host/report to later confirm upgrade with STOP fleet-preview-server instances in containers/apps on Docker run fleetctl preview with appropriate testing tag Navigate through all new UI flows and confirm dashboard, hosts, controls, queries, policies, and settings pages are working as expected.	All previously created hosts/queries are verified to still exist	pass/fail
Login flow	navigate to the login page and attempt to login with both valid and invalid credentials to verify some combination of expected results. navigate to the login page and attempt to login with both valid and invalid sso credentials to verify expected results.	text fields prompt when blank correct error message is "authentication failed" forget password link prompts for email valid credentials result in a successful login. valid sso credentials result in a successful login	pass/fail
Packs flow	Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs).	Packs successfully run on host machines after migrations New Packs can be created Packs can be edited and deleted Packs results information is logged	pass/fail
Log destination flow	Verify log destination for software, reports, policies, and packs.	Software, report, policy, and packs logs are successfully sent to external log destinations Software, report, policy, and packs logs are successfully sent to Filesystem log destinations	pass/fail
IdP Provisioning (SCIM)	Verify host vitals sync	Configure and verify provisioning with the following IdPs: Okta Entra Hydrant/Google Enroll hosts with EUA & IdP Provisioning enabled MacOS Windows Ubuntu iOS/iPadOS Android	pass/fail
GitOps and generate-gitops	Verify `fleetctl generate-gitops` and `GitOps` functionality	Generate-gitops from a version-matched fleetctl successfully outputs YAML from a brand new Fleet server (net of auto-populated fleets etc.). Running GitOps either using the gitops.sh script directly (from the fleet-gitops repo) or by using the GitOps GitHub or GitLab workflow (attempting via one of these three is sufficient) succeeds.	pass/fail
Fleet Free	Verify that product group features behave correctly on Fleet Free	Run basic checks for the product group area while using a Fleet Free license. Features documented as Free work normally Packs Gitops Premium features are correctly restricted or hidden IdP information No UI, API, or workflow errors occur when using Free-only functionality Reference: https://fleetdm.com/pricing	pass/fail
UI / UX	Verify visual consistency and layout integrity across product group areas	Perform a quick visual scan of the UI and confirm: No layout or alignment issues (misaligned, overlapping, or clipped elements) Fonts, colors, and icons render correctly and match the design system UI components render correctly (buttons, inputs, tables) No obvious visual regressions or broken UI states	pass/fail

MDM

Test name	Step instructions	Expected result	pass/fail
$Name	{what a tester should do}	{what a tester should see when they do that}	pass/fail
MDM enrollment flow	Verify MDM enrollments, run MDM commands	Erase an ADE-eligible macOS host and verify able to complete automated enrollment flow. With Windows MDM turned On, enroll a Windows host and verify MDM is turned On for the host. Erase an Auto-Pilot enabled Windows host and complete automated enrollment flow. Verify able to run MDM commands on both macOS and Windows hosts from the CLI.	pass/fail
MDM migration flow	Verify MDM migration for ADE and non-ADE hosts	Turn off MDM on an ADE-eligible macOS host and verify that the native, "Device Enrollment" macOS notification appears. On the My device page, follow the "Turn on MDM" instructions and verify that MDM is turned on. Turn off MDM on a non ADE-eligible macOS host. Verify Windows host migrates from 3rd party MDM to Fleet when automatic migration is turned on.	pass/fail
OS settings	Verify OS settings functionality	Verify Profiles upload/download/delete (macOS & Windows). Verify Profiles are delivered to host and applied.	pass/fail
Setup experience	Verify macOS Setup experience	Configure End user authentication. Upload a Bootstrap package. Add software (FMA, VPP, & Custom pkg) Add a script Enroll an ADE-eligible macOS host and verify successful authentication. Verify Bootstrap package is delivered. Verify SwiftDialogue window displays. Verify software installs and script runs.	pass/fail
iOS/iPadOS	Verify enrollment, profiles, & software installs	Verify ADE enrollment. Verify BYOD OTA enrollment. Verify BYOD Account-driven user enrollment (AppleID). Verify Profiles are delivered to host and applied. Verify VPP apps install & display correctly in Activity feed. Verify Turn Off MDM for BYOD & ADE hosts.	pass/fail
Android	Verify enrollment, profiles, & software installs	Verify BYOD enrollment. Verify Profiles are delivered to host and applied. Verify apps install. Verify certificate delivery Verify Unenroll.	pass/fail
Token & Certificate Renewals	APNs cert and ABM token renewal workflow	Renew APNs Certificate. Renew ABM Token. Ensure ADE hosts can enroll.	pass/fail
Fleet Free	Verify that product group features behave correctly on Fleet Free	Run basic checks for the product group area while using a Fleet Free license. Features documented as Free work normally Host enrollment Apple, Windows, Android MDM Configuration profile delivery APNs Certificate renewal Premium features are correctly restricted or hidden Setup experience No UI, API, or workflow errors occur when using Free-only functionality Reference: https://fleetdm.com/pricing	pass/fail
UI / UX	Verify visual consistency and layout integrity across product group areas	Perform a quick visual scan of the UI and confirm: No layout or alignment issues (misaligned, overlapping, or clipped elements) Fonts, colors, and icons render correctly and match the design system UI components render correctly (buttons, inputs, tables) No obvious visual regressions or broken UI states	pass/fail

Software

Test name	Step instructions	Expected result	pass/fail
$Name	{what a tester should do}	{what a tester should see when they do that}	pass/fail
Report flow	Create, edit, run, and delete reports.	permissions regarding creating/editing/deleting reports are up to date with documentation syntax errors result in error messaging queries can be run manually	pass/fail
Host Flow	Verify a new host can be added and removed following modal instructions using your own device.	Host is added via command line Host serial number and date added are accurate Host is not visible after it is deleted Warning and informational modals show when expected and make sense	pass/fail
My device page	Verify the end user's my device page loads successfully.	Clicking the Fleet desktop item, then "My device" successfully loads the my device page. The "My device" page is populated correctly and as expected. Styling and padding appears correct.	pass/fail
Scripts	Verify script library and execution	Verify able to run a script on all host types from CLI. Verify scripts library upload/download/delete. From Host details (macOS, Windows, & Linux) run a script that should PASS, verify. From Host details (macOS, Windows, & Linux) run a script that should FAIL, verify. Verify UI loading state and statuses for scripts. Disable scripts globally and verify unable to run. Verify scripts display correctly in Activity feed.	pass/fail
Software	Verify software library and install / download	Verify software library upload/download/delete. From Host details (macOS, Windows, & Linux) run an install that should PASS, verify. From My Device (macOS, Windows, & Linux) software tab should have self-service items available, verify. Verify UI loading state and statuses for installing software. Verify software installs display correctly in Activity feed.	pass/fail
Fleet Free	Verify that product group features behave correctly on Fleet Free	Run basic checks for the product group area while using a Fleet Free license. Features documented as Free work normally Host details page Reports (Add, edit, live report) Software inventory Scripts (Add, delete, run) My device page (Mac, Windows, Linux) Premium features are correctly restricted or hidden Add software No UI, API, or workflow errors occur when using Free-only functionality Reference: https://fleetdm.com/pricing	pass/fail
UI / UX	Verify visual consistency and layout integrity across product group areas	Perform a quick visual scan of the UI and confirm: No layout or alignment issues (misaligned, overlapping, or clipped elements) Fonts, colors, and icons render correctly and match the design system UI components render correctly (buttons, inputs, tables) No obvious visual regressions or broken UI states	pass/fail

Security & Compliance

Test name	Step instructions	Expected result	pass/fail
$Name	{what a tester should do}	{what a tester should see when they do that}	pass/fail
Disk encryption	Verify disk encryption functionality	Verify able to configure Disk encryption (macOS, Windows, & Linux). Verify host enrolled with Disk encryption enforced successfully encrypts.	pass/fail
Vulnerabilities	Verify that software vulnerabilities are correctly populated	Verify that known vulnerable software items display expected CVEs and severity information in the Software tab Verify that individual vulnerabilities can be previewed and open the correct NVD page when selected Verify that vulnerable software appears under "My device > Software" for affected hosts with expected CVEs	pass/fail
Certificate Authorities	Verify setup and certificate delivery	Configure and verify that certificates deploy to hosts with the following CAs: DigiCert NDES SmallStep	pass/fail
OS updates	Verify OS updates flow	Configure OS updates (macOS & Windows). Verify enforce minimumOS occurs during enrollment (macOS 14+).	pass/fail
Lock & Wipe	Verify hosts can be locked & wiped	Verify locking a host from the Fleet UI (macOS, Windows, & Linux) Verify unlocking a host from the Fleet UI (macOS, Windows, & Linux) Verify wiping a host from the Fleet UI (macOS, Windows, & Linux) Verify wiping and locking hosts using fleetctl (macOS, Windows, & Linux)	pass/fail
Fleet Free	Verify that product group features behave correctly on Fleet Free	Run basic checks for the product group area while using a Fleet Free license. Features documented as Free work normally Vulnerability detection Individual CVE page Premium features are correctly restricted or hidden Disk encryption OS Updates Lock / Wipe Certificate authorities No UI, API, or workflow errors occur when using Free-only functionality Reference: https://fleetdm.com/pricing	pass/fail
UI / UX	Verify visual consistency and layout integrity across product group areas	Perform a quick visual scan of the UI and confirm: No layout or alignment issues (misaligned, overlapping, or clipped elements) Fonts, colors, and icons render correctly and match the design system UI components render correctly (buttons, inputs, tables) No obvious visual regressions or broken UI states	pass/fail

All Product Groups

Test name	Step instructions	Expected result	Pass/Fail
$Name	{what a tester should do}	{what a tester should see when they do that}	pass/fail
Release blockers	Verify there are no outstanding release blocking tickets.	Check this filter to view all open ~release blocker tickets. If any are found raise an alarm in the #help-engineering and #g-mdm (or #g-endpoint-ops) channels.	pass/fail
Load tests - minor releases only unless otherwise specified	Verify all load test metrics are within acceptable range on final build of RC.	Check this Google doc to review load test key metrics and checks. After all expected changes have been merged to the RC branch, two load tests will need to be run - a new instance with no data, and a migrated instance. For the new instance with no data, set up a load test environment using the RC branch and allow it at least 24hrs of run time. For the migrated instance, set up a load test environment on the previous minor release branch. Once the environment has been set up and stabilized, follow the instructions in Deploying code changes to fleet to migrate to the RC branch. Monitor the metrics post-migration to determine if any performance issues arise. Record metrics in this spreadsheet for the two load test runs.	pass/fail
Migration Test	Verify Fleet can migrate to the next version with no issues.	Using this github action Using the most recent stable version of Fleet and main, click Run workflow Enter the Docker tag of Fleet starting version, e.g. v4.64.2 Enter the Docker tag of Fleet version to upgrade to, e.g. rc-minor-fleet-v4.65.0 Click Run workflow Action should complete successfully	pass/fail
Cloud migration tests	Verify Fleet can migrate when using real world data.	Using this github action Enter fleetdm/fleet:rc-minor-fleet-<version> for The image to test Select all for Where will we deploy? Action should complete successfully and the total time for each instance shouldn't be drastically different from previous releases	pass/fail

Notes

Issues found new to this version:

Issues found that reproduce in last stable version:

What has not been tested:

Include any notes on whether issues should block release or not as needed:

fleetd agent:

Includes updates to:

• Orbit: True / False
• Desktop: True / False
• Chrome extension: True / False

List versions changes for any component updates below:

• Orbit v1.xx.x > v1.xx.x
• Desktop v1.xx.x > v1.xx.x
• Chrome extension v1.xx.x > v1.xx.x

Testing gates for new fleetd release

Goal: Ensure new fleetd is tested and promoted from local > edge > stable channels

1. Build a new fleetd from the release candidate branch as needed for Orbit, Desktop, and Chrome Extension (e.g. rc-minor-fleet-v4.80.0). IMPORTANT: Do not build fleetd from main as it is a moving target and new fleetd changes from future releases might be already merged.

Test name	Step instructions	Expected result	pass/fail
$Name	{what a tester should do}	{what a tester should see when they do that}	pass/fail
`fleetd` local testing	1. Following [Testing TUF]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/README.md)) instructions create binaries for Mac, Windows, and Ubuntu using your local TUF repository and install on macOS, Linux, and Windows hosts. IMPORTANT: Reminder to use an RC branch and not `main`	1. Confirm the hosts install with the updated version and are working correctly. 2. Confirm any new features and/or bug fixes associated with this release are working as intended.	pass/fail
`fleetd` auto-update tests	1. Conduct the [`fleetd` auto-update n+1 test]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/Fleetd-auto-update-test-guide.md)) 2. QA certifies new release by commenting in issue.	1. Agent successfully auto-updates. 2. Issue is certified by QA.	pass/fail
`fleetd` tests	1. Set up a host in your instance to receive updates from the `edge` channels. 2. Work with engineer leading the release to push changes to the `edge` channel.	1. Confirm the hosts running on the edge channel receive the update and are working correctly. 2. Confirm any new features and/or bug fixes associated with this release are working as intended.	pass/fail

New fleetd pushed to edge

Goal: Ensure fleetd version pushed to edge is working with the current released version of fleet.

1. Fleet server is running the latest released version available on Fleet Releases page.
2. Set Agent options to use edge in the Fleet server configuration. For example:
   update_channels:
osqueryd: edge
orbit: edge
desktop: edge
   

Test name	Step instructions	Expected result	pass/fail
$Name	{what a tester should do}	{what a tester should see when they do that}	pass/fail
Report flow	Run reports.	1. Queries can be run manually	pass/fail
Host Flow	Verify a new host can be added using your own device.	1. Hosts can enroll and report correct version of `fleetd` (orbit, osquery, desktop). 2. Refetching host vitals completes and returns updated information.	pass/fail
My device page	Verify the end user's my device page loads successfully.	1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page. 2. The "My device" page is populated correctly and as expected. 3. Styling and padding appears correct.	pass/fail
Scripts	Verify script execution	1. Verify able to run a script on all host types from CLI. 2. From Host details (macOS, Windows, & Linux) run a script that should PASS, verify. 3. From Host details (macOS, Windows, & Linux) run a script that should FAIL, verify. 4. Verify script results display correctly in Activity feed.	pass/fail
Software	Verify software install / download	1. From Host details (macOS, Windows, & Linux) run an install that should PASS, verify. 2. From My Device (macOS, Windows, & Linux) software tab should have self-service items available, verify. 3. Verify software installs display correctly in Activity feed.	pass/fail
OS settings	Verify OS settings functionality	1. Verify able to configure Disk encryption (macOS, Windows, & Linux). 2. Verify host enrolled with Disk encryption enforced successfully encrypts.	pass/fail
Packs flow	Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs).	1. Packs successfully run on host machines after migrations 2. New Packs can be created. 3. Packs can be edited and deleted 4. Packs results information is logged	pass/fail
Fleet Free	Verify that Fleet Desktop works on Fleet Free.	After repointing a Fleet Desktop install at a server running Fleet Free: Clicking the Fleet desktop item, then "My device" successfully loads the my device page. The "My device" page is populated correctly and as expected. Styling and padding appears correct.	pass/fail
Auto-updates disabled	Verify that fleetd works on when the installer package is built with `--disable-updates`.	Generate package with fleetctl package [...] --updates-disabled Install packages on macOS, Windows, and Linux Smoke test orbit and Fleet Desktop functionality, and osquery tables.	pass/fail

Notes

Issues found new to this version:

Issues found that reproduce in last stable version:

What has not been tested:

Include any notes on whether issues should block release or not as needed: