mirror of https://github.com/fleetdm/fleet synced 2026-05-24 09:28:54 +00:00

History

Scott Gress 4334017b38 Add Vulnerabilities exposure dataset (#44124 ) <!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> Related issue: For #43769 # Details Adds methods to collect data for the `cve` dataset. As with all sets this is collected at hourly granularity, but unlike the `uptime` set, the `cve` set uses the "snapshot" strategy so that we record at most one change (the most recent) per hour. For this first iteration, we are _recording_ data for all CVEs (i.e., which hosts were exposed to which CVEs at a given time), but we are only _reporting_ a subset of CVEs for the dashboard chart. See [this comment](https://github.com/fleetdm/fleet/pull/44124#discussion_r3155554405) for more info. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT ` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually - [X] Spot-checked the CVEs chosen by the `trackedCVESoftwareMatchers` and didn't find any outside of the expected - [X] With [front-end PR](https://github.com/fleetdm/fleet/pull/44261), generated chart: <img width="706" height="421" alt="image" src="https://github.com/user-attachments/assets/539d9877-6573-4406-a159-1d2a711a045f" /> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit New Features * Host vulnerability (CVE) chart added to the dashboard; CVE chart data collection is now active. * Critical CVE tracking surfaces high-severity vulnerabilities. * Improvements * CVE chart refreshes every 3 hours (was daily) for more timely insights. * Snapshot collection reconciles and closes prior data during empty runs to keep charts accurate. * CVE queries may produce zero datapoints when no tracked CVEs exist, without affecting other metrics. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-04-29 09:30:31 -05:00
..
main.go	Add Vulnerabilities exposure dataset (#44124 )	2026-04-29 09:30:31 -05:00
README.md	Dashboard charts backend (#43910 )	2026-04-23 12:43:23 -05:00

Add Vulnerabilities exposure dataset (#44124 )

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #43769

# Details

Adds methods to collect data for the `cve` dataset. As with all sets
this is collected at hourly granularity, but unlike the `uptime` set,
the `cve` set uses the "snapshot" strategy so that we record at most one
change (the most recent) per hour.

For this first iteration, we are _recording_ data for all CVEs (i.e.,
which hosts were exposed to which CVEs at a given time), but we are only
_reporting_ a subset of CVEs for the dashboard chart. See [this
comment](https://github.com/fleetdm/fleet/pull/44124#discussion_r3155554405)
for more info.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.

## Testing

- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [X] QA'd all new/changed functionality manually
- [X] Spot-checked the CVEs chosen by the `trackedCVESoftwareMatchers`
and didn't find any outside of the expected
- [X] With [front-end PR](https://github.com/fleetdm/fleet/pull/44261),
generated chart:
<img width="706" height="421" alt="image"
src="https://github.com/user-attachments/assets/539d9877-6573-4406-a159-1d2a711a045f"
/>



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Host vulnerability (CVE) chart added to the dashboard; CVE chart data
collection is now active.
  * Critical CVE tracking surfaces high-severity vulnerabilities.

* **Improvements**
* CVE chart refreshes every 3 hours (was daily) for more timely
insights.
* Snapshot collection reconciles and closes prior data during empty runs
to keep charts accurate.
* CVE queries may produce zero datapoints when no tracked CVEs exist,
without affecting other metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-04-29 09:30:31 -05:00

main.go

Add Vulnerabilities exposure dataset (#44124 )

2026-04-29 09:30:31 -05:00

README.md

Dashboard charts backend (#43910 )

2026-04-23 12:43:23 -05:00

README.md

charts-collect

Fetches live data from a Fleet instance via the REST API and writes chart rows into a local database. Designed to run hourly via cron.

What it collects

Uptime — fetches currently online hosts and ORs them into the current hour's host_hourly_data_blobs row (dataset='uptime').
CVE — fetches per-host vulnerabilities, inverts into per-CVE host bitmaps, and reconciles into host_scd_data (dataset='cve'). Unchanged CVEs keep their open row; changed bitmaps close the prior-day row and open a new one for today; intra-day changes overwrite today's row via ODKU.

Usage

go run ./tools/charts-collect \
  --fleet-url https://dogfood.fleetdm.com \
  --fleet-token <token>

go run ./tools/charts-collect \
  --fleet-url https://dogfood.fleetdm.com \
  --fleet-token <token> \
  --mysql-dsn "fleet:fleet@tcp(localhost:3306)/fleet"

Flags and env vars

Flag	Env	Description
`--fleet-url`	`FLEET_URL`	Fleet server URL (required)
`--fleet-token`	`FLEET_TOKEN`	Fleet API token (required)
`--mysql-dsn`	`MYSQL_DSN`	Full MySQL DSN

If --mysql-dsn / MYSQL_DSN is not set, the DSN is assembled from the same env vars used by the fleet server (so the same values can be reused, e.g. via Render fromService):

FLEET_MYSQL_ADDRESS
FLEET_MYSQL_USERNAME
FLEET_MYSQL_PASSWORD
FLEET_MYSQL_DATABASE

Notes

SCD encoding constants (9999-12-31 open sentinel, batch size) mirror server/chart/internal/mysql/scd.go. Keep in sync when either side changes.
Errors in one collector (uptime/cve) are logged but do not block the other.