Create smoke-tests.md (#2237)

Add issue template for per-release smoke tests. Increases visibility.
This commit is contained in:
Renee Jackson 2021-09-27 14:02:38 -03:00 committed by GitHub
parent 5653f1e868
commit b63cf9d125
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23

67
.github/ISSUE_TEMPLATE/smoke-tests.md vendored Normal file
View file

@ -0,0 +1,67 @@
---
name: Release QA
about: Checklist of required tests prior to release
title: ''
labels: ''
assignees: ''
---
# Goal: easy-to-follow test steps for sanity checking a release manually
**Fleet version** (Head to the "My account" page in the Fleet UI or run `fleetctl version`):
**Web browser** _(e.g. Chrome 88.0.4324)_:
# Important reference data
1. [fleetctl preview setup](https://fleetdm.com/get-started)
2. [permissions documentation](https://fleetdm.com/docs/using-fleet/permissions)
# Smoke Tests
Smoke tests are limited to core functionality and serve as a sanity test. If smoke tests are failing, a release cannot proceed.
## Prerequisites
1. `fleetctrl preview` is set up and running the desired test version using `--tag` parameters.
2. Unless you are explicitly testing older browser versions, browser is up to date.
3. Certificate & flagfile are in place to create new host.
## Instructions
<table>
<tr><th>Test name</th><th>Step instructions</th><th>Expected result</th><th>pass/fail</td></tr>
<tr><td>$Name</td><td>{what a tester should do}</td><td>{what a tester should see when they do that}</td><td>pass/fail</td></tr>
<tr><td>Update flow</td><td>
1. Prereq: previous version is set up with data.
2. Update fleet to testing version
</td><td>No data should be lost in update</td><td>pass/fail</td></tr>
<tr><td>Login flow</td><td>
1. navigate to the login page and attempt to login with both valid and invalid credentials to verify some combination of expected results.
2. Login with SSO
</td><td>
1. text fields prompt when blank
2. correct error message is "authentication failed"
3. forget password link prompts for email
4. valid credentials result in a successful login. </td><td>pass/fail</td></tr>
<tr><td>Query flow</td><td>Create, edit, run, and delete queries. </td><td>
1. permissions regarding creating/editing/deleting queries are up to date with documentation
2. syntax errors result in error messaging
3. queries can be run manually
</td><td>pass/fail</td></tr>
<tr><td>Host Flow</td><td>Verify a new host can be added and removed following modal instructions using your own device.</td><td>
1. Host is added via command line
2. Host serial number and date added are accurate
3. Host is not visible after it is deleted
4. Warning and informational modals show when expected and make sense
</td><td>pass/fail</td></tr>
</table>
# Notes
* {any notes}