mirror of
https://github.com/fleetdm/fleet
synced 2026-04-21 21:47:20 +00:00
241d63895d
Changes: 1. Adding Fleet free checks for each product group 2. Adding UI/UX checks for each product group 3. Expanding the IdP coverage and moving it to orchestration (postmortem action item: https://github.com/fleetdm/fleet/issues/39684) 4. Moving Certificates to S&C 5. Adjusting assignee list
24 KiB
24 KiB
| name | about | title | labels | assignees |
|---|---|---|---|---|
| Release QA | Checklist of required tests prior to release | Release QA: | #g-mdm,#g-orchestration,#g-software,#g-security-compliance,:release | xpkoala,andreykizimenko,chrstphr84,Brajim20 |
Goal: easy-to-follow test steps for checking a release manually
Important reference data
- fleetctl preview setup
- permissions documentation
- premium tests require license key (needs renewal)
fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjQwOTk1MjAwLCJzdWIiOiJkZXZlbG9wbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6ImJhc2ljIiwiaWF0IjoxNjIyNDI2NTg2fQ.WmZ0kG4seW3IrNvULCHUPBSfFdqj38A_eiXdV_DFunMHechjHbkwtfkf1J6JQJoDyqn8raXpgbdhafDwv3rmDw - premium tests require license key (active - Expires Sunday, January 1, 2023 12:00:00 AM)
fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjcyNTMxMjAwLCJzdWIiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6InByZW1pdW0iLCJpYXQiOjE2NDI1MjIxODF9.EGHQjIzM73YyMbnCruswzg360DEYCsDi9uz48YcDwQHq90BabGT5PIXRiculw79emGj5sk2aKgccTd2hU5J7Jw
Database migration tests
- Create a custom issue tagged
:help-customersin the confidential repo to run cloud migration tests targeted off of the RC branch. Tests will be run off of these environments. - Once tests are complete, if migration duration for any environment takes more than 5 seconds, check logs to determine whether any single migration took more than 5 seconds, or if the entire process took more than 15 seconds. If either is the case and there is not already a progress indicator for the migration that updates at least every ten seconds, file an unreleased bug triaged to the team that created the migration to audit the migration and evaluate if progress updates or performance improvements are needed.
Smoke Tests
Smoke tests are limited to core functionality and serve as a pre-release final review. If smoke tests are failing, a release cannot proceed.
Fleet core:
Fleet version (Head to the "My account" page in the Fleet UI or run fleetctl version):
Web browser (e.g. Chrome 88.0.4324):
Prerequisites
fleetctl previewis set up and running the desired test version using--tagparameters.- Unless you are explicitly testing older browser versions, browser is up to date.
- Certificate & flagfile are in place to create new host.
- In your browser, clear local storage using devtools.
Orchestration
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Update flow |
| All previously created hosts/queries are verified to still exist | pass/fail |
| Login flow |
|
| pass/fail |
| Packs flow | Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). |
| pass/fail |
| Log destination flow | Verify log destination for software, reports, policies, and packs. |
| pass/fail |
| IdP Provisioning (SCIM) | Verify host vitals sync |
| pass/fail |
| GitOps and generate-gitops | Verify `fleetctl generate-gitops` and `GitOps` functionality |
| pass/fail |
| Fleet Free | Verify that product group features behave correctly on Fleet Free |
Run basic checks for the product group area while using a Fleet Free license.
Reference: https://fleetdm.com/pricing | pass/fail |
| UI / UX | Verify visual consistency and layout integrity across product group areas |
Perform a quick visual scan of the UI and confirm:
| pass/fail |
MDM
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| MDM enrollment flow | Verify MDM enrollments, run MDM commands |
| pass/fail |
| MDM migration flow | Verify MDM migration for ADE and non-ADE hosts |
| pass/fail |
| OS settings | Verify OS settings functionality |
| pass/fail |
| Setup experience | Verify macOS Setup experience |
| pass/fail |
| iOS/iPadOS | Verify enrollment, profiles, & software installs |
| pass/fail |
| Android | Verify enrollment, profiles, & software installs |
| pass/fail |
| Token & Certificate Renewals | APNs cert and ABM token renewal workflow |
| pass/fail |
| Fleet Free | Verify that product group features behave correctly on Fleet Free |
Run basic checks for the product group area while using a Fleet Free license.
Reference: https://fleetdm.com/pricing | pass/fail |
| UI / UX | Verify visual consistency and layout integrity across product group areas |
Perform a quick visual scan of the UI and confirm:
| pass/fail |
Software
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Report flow | Create, edit, run, and delete reports. |
| pass/fail |
| Host Flow | Verify a new host can be added and removed following modal instructions using your own device. |
| pass/fail |
| My device page | Verify the end user's my device page loads successfully. |
| pass/fail |
| Scripts | Verify script library and execution |
| pass/fail |
| Software | Verify software library and install / download |
| pass/fail |
| Migration Test | Verify Fleet can migrate to the next version with no issues. |
Using the github action https://github.com/fleetdm/fleet/actions/workflows/db-upgrade-test.yml
| pass/fail |
| Fleet Free | Verify that product group features behave correctly on Fleet Free |
Run basic checks for the product group area while using a Fleet Free license.
Reference: https://fleetdm.com/pricing | pass/fail |
| UI / UX | Verify visual consistency and layout integrity across product group areas |
Perform a quick visual scan of the UI and confirm:
| pass/fail |
Security & Compliance
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Disk encryption | Verify disk encryption functionality |
| pass/fail |
| Vulnerabilities | Verify that software vulnerabilities are correctly populated |
| pass/fail |
| Certificate Authorities | Verify setup and certificate delivery |
| pass/fail |
| OS updates | Verify OS updates flow |
| pass/fail |
| Lock & Wipe | Verify hosts can be locked & wiped |
| pass/fail |
| Fleet Free | Verify that product group features behave correctly on Fleet Free |
Run basic checks for the product group area while using a Fleet Free license.
Reference: https://fleetdm.com/pricing | pass/fail |
| UI / UX | Verify visual consistency and layout integrity across product group areas |
Perform a quick visual scan of the UI and confirm:
| pass/fail |
All Product Groups
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Release blockers | Verify there are no outstanding release blocking tickets. |
| pass/fail |
| Load tests - minor releases only unless otherwise specified | Verify all load test metrics are within acceptable range on final build of RC. |
| pass/fail |
Notes
Issues found new to this version:
Issues found that reproduce in last stable version:
What has not been tested:
Include any notes on whether issues should block release or not as needed:
fleetd agent:
Includes updates to:
- Orbit: True / False
- Desktop: True / False
- Chrome extension: True / False
List versions changes for any component updates below:
- Orbit
v1.xx.x>v1.xx.x - Desktop
v1.xx.x>v1.xx.x - Chrome extension
v1.xx.x>v1.xx.x
Testing gates for new fleetd release
Goal: Ensure new fleetd is tested and promoted from local > edge > stable channels
- Build a new
fleetdfrom the release candidate branch as needed for Orbit, Desktop, and Chrome Extension (e.g.rc-minor-fleet-v4.80.0). IMPORTANT: Do not build fleetd frommainas it is a moving target and new fleetd changes from future releases might be already merged.
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| `fleetd` local testing |
1. Following [Testing TUF]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/README.md)) instructions create binaries for Mac, Windows, and Ubuntu using your local TUF repository and install on macOS, Linux, and Windows hosts. IMPORTANT: Reminder to use an RC branch and not `main` |
1. Confirm the hosts install with the updated version and are working correctly. 2. Confirm any new features and/or bug fixes associated with this release are working as intended. |
pass/fail |
| `fleetd` auto-update tests |
1. Conduct the [`fleetd` auto-update n+1 test]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/Fleetd-auto-update-test-guide.md)) 2. QA certifies new release by commenting in issue. |
1. Agent successfully auto-updates. 2. Issue is certified by QA. |
pass/fail |
| `fleetd` tests |
1. Set up a host in your instance to receive updates from the `edge` channels. 2. Work with engineer leading the release to push changes to the `edge` channel. |
1. Confirm the hosts running on the edge channel receive the update and are working correctly. 2. Confirm any new features and/or bug fixes associated with this release are working as intended. |
pass/fail |
New fleetd pushed to edge
Goal: Ensure fleetd version pushed to edge is working with the current released version of fleet.
- Fleet server is running the latest released version available on Fleet Releases page.
- Set Agent options to use edge in the Fleet server configuration. For example:
update_channels:
osqueryd: edge
orbit: edge
desktop: edge
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Report flow | Run reports. | 1. Queries can be run manually | pass/fail |
| Host Flow | Verify a new host can be added using your own device. |
1. Hosts can enroll and report correct version of `fleetd` (orbit, osquery, desktop). 2. Refetching host vitals completes and returns updated information. | pass/fail |
| My device page | Verify the end user's my device page loads successfully. |
1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page. 2. The "My device" page is populated correctly and as expected. 3. Styling and padding appears correct. | pass/fail |
| Scripts | Verify script execution |
1. Verify able to run a script on all host types from CLI. 2. From Host details (macOS, Windows, & Linux) run a script that should PASS, verify. 3. From Host details (macOS, Windows, & Linux) run a script that should FAIL, verify. 4. Verify script results display correctly in Activity feed. | pass/fail |
| Software | Verify software install / download |
1. From Host details (macOS, Windows, & Linux) run an install that should PASS, verify. 2. From My Device (macOS, Windows, & Linux) software tab should have self-service items available, verify. 3. Verify software installs display correctly in Activity feed. | pass/fail |
| OS settings | Verify OS settings functionality |
1. Verify able to configure Disk encryption (macOS, Windows, & Linux). 2. Verify host enrolled with Disk encryption enforced successfully encrypts. | pass/fail |
| Packs flow | Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). |
1. Packs successfully run on host machines after migrations 2. New Packs can be created. 3. Packs can be edited and deleted 4. Packs results information is logged | pass/fail |
| Fleet Free | Verify that Fleet Desktop works on Fleet Free. |
After repointing a Fleet Desktop install at a server running Fleet Free:
| pass/fail |
| Auto-updates disabled | Verify that fleetd works on when the installer package is built with `--disable-updates`. |
| pass/fail |
Notes
Issues found new to this version:
Issues found that reproduce in last stable version:
What has not been tested:
Include any notes on whether issues should block release or not as needed: