mirror of
https://github.com/fleetdm/fleet
synced 2026-05-21 16:08:47 +00:00
6bd9cc8a44
for #19930 # Checklist for submitter - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [X] Added/updated tests - [X] If database migrations are included, checked table schema to confirm autoupdate - [X] Manual QA for all new/changed functionality # Details This PR adds a new feature to the existing monitoring add-on. The add-on will now send an SNS alert whenever a scheduled job like "vulnerabilities" or "apple_mdm_apns_pusher" exits early due to errors. The alert contains the job type and the set of errors (there can be multiple, since jobs can have multiple sub-jobs). By default the SNS topic for this new alert is the same as the one for the existing cron system alerts, but it can be configured to use a separate topic (e.g. dogfood instance will post to a separate slack channel). The actual changes are: **On the server side:** - Add errors field to cron_stats table (json DEFAULT NULL) - Added errors var to `Schedule` struct to collect errors from jobs - In `RunAllJobs`, collect err from job into new errors var - Update `Schedule.updateStats`and `CronStats.UpdateCronStats`to accept errors argument - If provided, update errors field of cron_stats table **On the monitor side:** - Add new SQL query to look for all completed schedules since last run with non-null errors - send SNS with job ID, name, errors # Testing New automated testing was added for the functional code that gathers and stores errors from cron runs in the database. To test the actual Lambda, I added a row in my `cron_stats` table with errors, then compiled and ran the Lambda executable locally, pointing it to my local mysql and localstack instances: ``` 2024/12/03 14:43:54 main.go:258: Lambda execution environment not found. Falling back to local execution. 2024/12/03 14:43:54 main.go:133: Connected to database! 2024/12/03 14:43:54 main.go:161: Row vulnerabilities last updated at 2024-11-27 03:30:03 +0000 UTC 2024/12/03 14:43:54 main.go:163: *** 1h hasn't updated in more than vulnerabilities, alerting! (status completed) 2024/12/03 14:43:54 main.go:70: Sending SNS Message 2024/12/03 14:43:54 main.go:74: Sending 'Environment: dev Message: Fleet cron 'vulnerabilities' hasn't updated in more than 1h. Last status was 'completed' at 2024-11-27 03:30:03 +0000 UTC.' to 'arn:aws:sns:us-east-1:000000000000:topic1' 2024/12/03 14:43:54 main.go:82: { MessageId: "260864ff-4cc9-4951-acea-cef883b2de5f" } 2024/12/03 14:43:54 main.go:198: *** mdm_apple_profile_manager job had errors, alerting! (errors {"something": "wrong"}) 2024/12/03 14:43:54 main.go:70: Sending SNS Message 2024/12/03 14:43:54 main.go:74: Sending 'Environment: dev Message: Fleet cron 'mdm_apple_profile_manager' (last updated 2024-12-03 20:34:14 +0000 UTC) raised errors during its run: {"something": "wrong"}.' to 'arn:aws:sns:us-east-1:000000000000:topic1' 2024/12/03 14:43:54 main.go:82: { MessageId: "5cd085ef-89f6-42c1-8470-d80a22b295f8" |
||
|---|---|---|
| .. | ||
| .keep | ||
| 8903c-table-text-wrapping | ||
| 19930-alert-on-cron-errors | ||
| 21855-paginate-queries | ||
| 22444-gitops-script-missing-path | ||
| 22523-cve-500 | ||
| 22723-jetbrains-eap-versions | ||
| 22884-license-error-fleetctl | ||
| 23011-dynamic-column-sort-type | ||
| 23238-use-secrets-in-scripts-profiles | ||
| 23309-mastodon-in-email-templates | ||
| 23448-maintainer-policy-automations | ||
| 23490-null-script-software-on-policies | ||
| 23579-cve-2024-10004-false-positive | ||
| 23611-Update-CIS | ||
| 23800-host-online-pctage | ||
| 23803-leading-zeros-bug | ||
| 23886-remove-associations-on-policy-delete | ||
| 24006-host-query-report-team-id | ||
| 24025-add-label-team-bug | ||
| 24033-helm-customization-options | ||
| 24120-sw-title-id | ||
| 24166-script-line-endings | ||
| 24268-update-fleetctl-deps | ||
| 24286-vulncheck | ||
| 24334-policy-truncation | ||
| 24386-fleet-legacy-query-pack | ||
| 24456-include-linux-encryption-data-only-when-enabled | ||
| 24459-chart-serrvice-annotation | ||
| 24549-validate-script-profle-secrets | ||
| 24725-no-private-ip-found | ||
| 24795-add-helpful-tooltip-setup-experience | ||