Dashboard charts backend (#43910)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #42812
# Details
This PR implements a new bounded context, `chart`, with a single
endpoint `/charts`. The context encompasses a framework for recording
and querying and aggregating historical data for Fleet hosts, and
returning that data via the API for the purpose of charting.
This initial iteration has a full implementation of a dataset called
"uptime" which captures which hosts were online hour-by-hour (online
meaning, having been "seen" at some point during that hour). It has a
partial implementation of a "cve" dataset which will capture which hosts
were vulnerable to which CVEs during a given day.
### Data storage
Data is stored in an SCD (slowly-changing dimension) format in the
`host_scd_data` table, where the main "value" in a row is stored in the
`host_bitmap` column, which is a `mediumblob` where each bit encodes a
host ID (bit one represents host ID 1, bit 1444 represents host ID 1444,
etc.). The set of bits set on a row represents that hosts for which that
dataset is "on" during a given time period represented by the
`valid_from` (inclusive) and `valid_to` (exclusive) dates, where a
`valid_to` can have the special "sentinel" value 9999-12-31T00:00:00.000
meaning that the row is still "open" (the value represents everything
from `valid_from` to the present). Additionally an `entity_id` column
can be used for datasets with multiple dimensions, e.g. CVE exposure or
software usage which would have entity IDs representing CVEs or software
items respectively.
### Data collection
Data is collected via a cron job that runs every 10 minutes. Each
dataset has its own `Collect` method which will sample the data for the
given moment. For example the "uptime" dataset gathers the set of hosts
that are online at the moment, and the "cve" dataset will gather the set
of hosts that are vulnerable to each CVE at that moment. The sample can
then be recorded using one of two strategies:
* `accumulate`: bitwise OR the sample with any data already recorded for
the current hour, or add a new pre-closed row for that hour.
* `snapshot`: if there is no open row, create one with the sample and
`valid_to set` to the sentinel. Otherwise:
* If the sample has the same value as the current open row, do nothing
* If the sample has a different value and the current open row's
`valid_from` is within the same hour, update the current row's value
* If the sample has a different value and the current open row's
`valid_from` is not within the same hour, close the current open row and
start a new one with `valid_from` = the start of the current hour
### Data retrieval
1. Gets the set of host IDs to retrieve data for. This starts with the
set of host IDs in the requested fleet (or all the hosts a user has
access to if no `fleet_id` param was passed to the `/charts` endpoint),
and further whittled down by any filter options supplied with the
request (labels, platforms, etc.).
2. Finds all `host_scd_data` rows for the requested dataset and date
range (i.e. all rows whose `valid_from` is < the date range end and
`valid_to` is > the date range start).
3. Calculates the date ranges of the "buckets" to return datapoints for.
For the uptime chart we default to 3-hour buckets, so we want 8 buckets
per day.
4. Iterates over each bucket and finds the row or rows from
host_scd_data that cover that bucket range. For datasets using the
"accumulate" strategy, the values for those rows are ORed together. For
"snapshot"s, we take the one active at the bucket end time to represent
the bucket (e.g. "which hosts had a given CVE at the end of the day")
### Tools
This PR includes two dev tools that don't require deep review:
* **chart-backfill** - used to backfill data to various datasets for
testing
* **charts-collect** - used to collect data from a live server via the
API and put into a local hosts_scd_data table
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- With [front-end branch](https://github.com/fleetdm/fleet/pull/43878)
<img width="712" height="434" alt="image"
src="https://github.com/user-attachments/assets/b2ccce49-b5fd-4076-b47f-0eea6a53260c"
/>
## Database migrations
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added charting bounded context: HTTP API for metrics (uptime, CVE),
dataset registry, hosted dataset collection, background
collection/cleanup with opt-out env.
* New utilities: host bitmap operations and string-list/uint-list
parsers.
* New CLI tools to collect and backfill chart data.
* **Database**
* Migration and schema to store host time-series SCD chart data.
* **Tests**
* Extensive unit and integration tests for service, storage, caching,
cron, and utilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-23 17:43:23 +00:00
|
|
|
package api
|
|
|
|
|
|
|
|
|
|
import (
|
|
|
|
|
"context"
|
|
|
|
|
"time"
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
// SampleStrategy describes how a dataset's samples combine within a bucket and
|
|
|
|
|
// whether rows can collapse across buckets when the bitmap is unchanged.
|
|
|
|
|
type SampleStrategy string
|
|
|
|
|
|
|
|
|
|
const (
|
|
|
|
|
// SampleStrategyAccumulate means each sample is a partial observation.
|
|
|
|
|
// Writes: every row is born closed (valid_to set at insert time to bucketEnd).
|
|
|
|
|
// Within-bucket samples OR-merge into the existing row via ODKU; a sample in
|
|
|
|
|
// a new bucket just creates a new row with a new valid_from. No explicit
|
|
|
|
|
// close step, no cross-bucket collapse.
|
|
|
|
|
// Reads: bucket value = OR of every row whose interval overlaps the bucket
|
|
|
|
|
// ("hosts observed at any point during the bucket").
|
|
|
|
|
// Used for datasets like uptime and software usage.
|
|
|
|
|
// @todo: implement job to collapse identical consecutive rows
|
|
|
|
|
// to optimize storage and query performance.
|
|
|
|
|
SampleStrategyAccumulate SampleStrategy = "accumulate"
|
|
|
|
|
|
|
|
|
|
// SampleStrategySnapshot means each sample is the full state of a single moment.
|
|
|
|
|
// Writes: rows are always keyed to 1h boundaries (so row transitions align
|
|
|
|
|
// to hour marks regardless of tz). Within a 1h write-bucket, the latest
|
|
|
|
|
// sample's bitmap overwrites via ODKU — last sample wins. Across buckets,
|
|
|
|
|
// unchanged state keeps the row open (valid_to = sentinel); a changed sample
|
|
|
|
|
// closes the prior row at the new hour boundary and opens a new one.
|
|
|
|
|
// Reads: bucket value = OR across entities of each entity's row active at
|
|
|
|
|
// bucketEnd ("state as of the end of the bucket"). An entity whose row was
|
|
|
|
|
// closed mid-bucket with no replacement is absent at bucketEnd.
|
|
|
|
|
// Used for datasets like CVE and software inventory.
|
|
|
|
|
SampleStrategySnapshot SampleStrategy = "snapshot"
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
// Dataset defines the interface for a chartable dataset.
|
|
|
|
|
type Dataset interface {
|
|
|
|
|
// Name returns the dataset identifier used in the DB and API path.
|
|
|
|
|
Name() string
|
|
|
|
|
|
Add Vulnerabilities exposure dataset (#44124)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #43769
# Details
Adds methods to collect data for the `cve` dataset. As with all sets
this is collected at hourly granularity, but unlike the `uptime` set,
the `cve` set uses the "snapshot" strategy so that we record at most one
change (the most recent) per hour.
For this first iteration, we are _recording_ data for all CVEs (i.e.,
which hosts were exposed to which CVEs at a given time), but we are only
_reporting_ a subset of CVEs for the dashboard chart. See [this
comment](https://github.com/fleetdm/fleet/pull/44124#discussion_r3155554405)
for more info.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- [X] Spot-checked the CVEs chosen by the `trackedCVESoftwareMatchers`
and didn't find any outside of the expected
- [X] With [front-end PR](https://github.com/fleetdm/fleet/pull/44261),
generated chart:
<img width="706" height="421" alt="image"
src="https://github.com/user-attachments/assets/539d9877-6573-4406-a159-1d2a711a045f"
/>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Host vulnerability (CVE) chart added to the dashboard; CVE chart data
collection is now active.
* Critical CVE tracking surfaces high-severity vulnerabilities.
* **Improvements**
* CVE chart refreshes every 3 hours (was daily) for more timely
insights.
* Snapshot collection reconciles and closes prior data during empty runs
to keep charts accurate.
* CVE queries may produce zero datapoints when no tracked CVEs exist,
without affecting other metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-29 14:30:31 +00:00
|
|
|
// DefaultResolutionHours returns the default display granularity in hours.
|
|
|
|
|
// Used when the caller doesn't specify RequestOpts.Resolution. Unrelated
|
|
|
|
|
// to write-side granularity — all collectors write at 1h regardless of
|
|
|
|
|
// display resolution; see SampleStrategy for details.
|
Dashboard charts backend (#43910)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #42812
# Details
This PR implements a new bounded context, `chart`, with a single
endpoint `/charts`. The context encompasses a framework for recording
and querying and aggregating historical data for Fleet hosts, and
returning that data via the API for the purpose of charting.
This initial iteration has a full implementation of a dataset called
"uptime" which captures which hosts were online hour-by-hour (online
meaning, having been "seen" at some point during that hour). It has a
partial implementation of a "cve" dataset which will capture which hosts
were vulnerable to which CVEs during a given day.
### Data storage
Data is stored in an SCD (slowly-changing dimension) format in the
`host_scd_data` table, where the main "value" in a row is stored in the
`host_bitmap` column, which is a `mediumblob` where each bit encodes a
host ID (bit one represents host ID 1, bit 1444 represents host ID 1444,
etc.). The set of bits set on a row represents that hosts for which that
dataset is "on" during a given time period represented by the
`valid_from` (inclusive) and `valid_to` (exclusive) dates, where a
`valid_to` can have the special "sentinel" value 9999-12-31T00:00:00.000
meaning that the row is still "open" (the value represents everything
from `valid_from` to the present). Additionally an `entity_id` column
can be used for datasets with multiple dimensions, e.g. CVE exposure or
software usage which would have entity IDs representing CVEs or software
items respectively.
### Data collection
Data is collected via a cron job that runs every 10 minutes. Each
dataset has its own `Collect` method which will sample the data for the
given moment. For example the "uptime" dataset gathers the set of hosts
that are online at the moment, and the "cve" dataset will gather the set
of hosts that are vulnerable to each CVE at that moment. The sample can
then be recorded using one of two strategies:
* `accumulate`: bitwise OR the sample with any data already recorded for
the current hour, or add a new pre-closed row for that hour.
* `snapshot`: if there is no open row, create one with the sample and
`valid_to set` to the sentinel. Otherwise:
* If the sample has the same value as the current open row, do nothing
* If the sample has a different value and the current open row's
`valid_from` is within the same hour, update the current row's value
* If the sample has a different value and the current open row's
`valid_from` is not within the same hour, close the current open row and
start a new one with `valid_from` = the start of the current hour
### Data retrieval
1. Gets the set of host IDs to retrieve data for. This starts with the
set of host IDs in the requested fleet (or all the hosts a user has
access to if no `fleet_id` param was passed to the `/charts` endpoint),
and further whittled down by any filter options supplied with the
request (labels, platforms, etc.).
2. Finds all `host_scd_data` rows for the requested dataset and date
range (i.e. all rows whose `valid_from` is < the date range end and
`valid_to` is > the date range start).
3. Calculates the date ranges of the "buckets" to return datapoints for.
For the uptime chart we default to 3-hour buckets, so we want 8 buckets
per day.
4. Iterates over each bucket and finds the row or rows from
host_scd_data that cover that bucket range. For datasets using the
"accumulate" strategy, the values for those rows are ORed together. For
"snapshot"s, we take the one active at the bucket end time to represent
the bucket (e.g. "which hosts had a given CVE at the end of the day")
### Tools
This PR includes two dev tools that don't require deep review:
* **chart-backfill** - used to backfill data to various datasets for
testing
* **charts-collect** - used to collect data from a live server via the
API and put into a local hosts_scd_data table
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- With [front-end branch](https://github.com/fleetdm/fleet/pull/43878)
<img width="712" height="434" alt="image"
src="https://github.com/user-attachments/assets/b2ccce49-b5fd-4076-b47f-0eea6a53260c"
/>
## Database migrations
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added charting bounded context: HTTP API for metrics (uptime, CVE),
dataset registry, hosted dataset collection, background
collection/cleanup with opt-out env.
* New utilities: host bitmap operations and string-list/uint-list
parsers.
* New CLI tools to collect and backfill chart data.
* **Database**
* Migration and schema to store host time-series SCD chart data.
* **Tests**
* Extensive unit and integration tests for service, storage, caching,
cron, and utilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-23 17:43:23 +00:00
|
|
|
DefaultResolutionHours() int
|
|
|
|
|
|
|
|
|
|
// SampleStrategy returns how samples combine within and across buckets.
|
|
|
|
|
SampleStrategy() SampleStrategy
|
|
|
|
|
|
|
|
|
|
// Collect is called by the cron job to populate data in bulk.
|
|
|
|
|
Collect(ctx context.Context, store DatasetStore, now time.Time) error
|
|
|
|
|
|
|
|
|
|
// DefaultVisualization returns the default visualization type (e.g. "line", "heatmap").
|
|
|
|
|
DefaultVisualization() string
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// DatasetStore is the narrow interface that datasets need for their Collect
|
|
|
|
|
// method. It is satisfied by the chart internal Datastore, keeping dataset
|
|
|
|
|
// implementations decoupled from internals.
|
|
|
|
|
type DatasetStore interface {
|
|
|
|
|
// FindRecentlySeenHostIDs returns host IDs that have reported since the
|
|
|
|
|
// given cutoff. Used by datasets like uptime that derive their sample from
|
|
|
|
|
// recent host activity.
|
|
|
|
|
FindRecentlySeenHostIDs(ctx context.Context, since time.Time) ([]uint, error)
|
|
|
|
|
|
Add Vulnerabilities exposure dataset (#44124)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #43769
# Details
Adds methods to collect data for the `cve` dataset. As with all sets
this is collected at hourly granularity, but unlike the `uptime` set,
the `cve` set uses the "snapshot" strategy so that we record at most one
change (the most recent) per hour.
For this first iteration, we are _recording_ data for all CVEs (i.e.,
which hosts were exposed to which CVEs at a given time), but we are only
_reporting_ a subset of CVEs for the dashboard chart. See [this
comment](https://github.com/fleetdm/fleet/pull/44124#discussion_r3155554405)
for more info.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- [X] Spot-checked the CVEs chosen by the `trackedCVESoftwareMatchers`
and didn't find any outside of the expected
- [X] With [front-end PR](https://github.com/fleetdm/fleet/pull/44261),
generated chart:
<img width="706" height="421" alt="image"
src="https://github.com/user-attachments/assets/539d9877-6573-4406-a159-1d2a711a045f"
/>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Host vulnerability (CVE) chart added to the dashboard; CVE chart data
collection is now active.
* Critical CVE tracking surfaces high-severity vulnerabilities.
* **Improvements**
* CVE chart refreshes every 3 hours (was daily) for more timely
insights.
* Snapshot collection reconciles and closes prior data during empty runs
to keep charts accurate.
* CVE queries may produce zero datapoints when no tracked CVEs exist,
without affecting other metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-29 14:30:31 +00:00
|
|
|
// AffectedHostIDsByCVE returns, for every CVE currently affecting any host,
|
|
|
|
|
// the slice of host IDs impacted by it. Unresolved-only is implicit in the
|
|
|
|
|
// underlying joins: a host's software/OS row transitions when it upgrades
|
|
|
|
|
// past the vulnerable version, so the join naturally stops matching.
|
|
|
|
|
AffectedHostIDsByCVE(ctx context.Context) (map[string][]uint, error)
|
|
|
|
|
|
Dashboard charts backend (#43910)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #42812
# Details
This PR implements a new bounded context, `chart`, with a single
endpoint `/charts`. The context encompasses a framework for recording
and querying and aggregating historical data for Fleet hosts, and
returning that data via the API for the purpose of charting.
This initial iteration has a full implementation of a dataset called
"uptime" which captures which hosts were online hour-by-hour (online
meaning, having been "seen" at some point during that hour). It has a
partial implementation of a "cve" dataset which will capture which hosts
were vulnerable to which CVEs during a given day.
### Data storage
Data is stored in an SCD (slowly-changing dimension) format in the
`host_scd_data` table, where the main "value" in a row is stored in the
`host_bitmap` column, which is a `mediumblob` where each bit encodes a
host ID (bit one represents host ID 1, bit 1444 represents host ID 1444,
etc.). The set of bits set on a row represents that hosts for which that
dataset is "on" during a given time period represented by the
`valid_from` (inclusive) and `valid_to` (exclusive) dates, where a
`valid_to` can have the special "sentinel" value 9999-12-31T00:00:00.000
meaning that the row is still "open" (the value represents everything
from `valid_from` to the present). Additionally an `entity_id` column
can be used for datasets with multiple dimensions, e.g. CVE exposure or
software usage which would have entity IDs representing CVEs or software
items respectively.
### Data collection
Data is collected via a cron job that runs every 10 minutes. Each
dataset has its own `Collect` method which will sample the data for the
given moment. For example the "uptime" dataset gathers the set of hosts
that are online at the moment, and the "cve" dataset will gather the set
of hosts that are vulnerable to each CVE at that moment. The sample can
then be recorded using one of two strategies:
* `accumulate`: bitwise OR the sample with any data already recorded for
the current hour, or add a new pre-closed row for that hour.
* `snapshot`: if there is no open row, create one with the sample and
`valid_to set` to the sentinel. Otherwise:
* If the sample has the same value as the current open row, do nothing
* If the sample has a different value and the current open row's
`valid_from` is within the same hour, update the current row's value
* If the sample has a different value and the current open row's
`valid_from` is not within the same hour, close the current open row and
start a new one with `valid_from` = the start of the current hour
### Data retrieval
1. Gets the set of host IDs to retrieve data for. This starts with the
set of host IDs in the requested fleet (or all the hosts a user has
access to if no `fleet_id` param was passed to the `/charts` endpoint),
and further whittled down by any filter options supplied with the
request (labels, platforms, etc.).
2. Finds all `host_scd_data` rows for the requested dataset and date
range (i.e. all rows whose `valid_from` is < the date range end and
`valid_to` is > the date range start).
3. Calculates the date ranges of the "buckets" to return datapoints for.
For the uptime chart we default to 3-hour buckets, so we want 8 buckets
per day.
4. Iterates over each bucket and finds the row or rows from
host_scd_data that cover that bucket range. For datasets using the
"accumulate" strategy, the values for those rows are ORed together. For
"snapshot"s, we take the one active at the bucket end time to represent
the bucket (e.g. "which hosts had a given CVE at the end of the day")
### Tools
This PR includes two dev tools that don't require deep review:
* **chart-backfill** - used to backfill data to various datasets for
testing
* **charts-collect** - used to collect data from a live server via the
API and put into a local hosts_scd_data table
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- With [front-end branch](https://github.com/fleetdm/fleet/pull/43878)
<img width="712" height="434" alt="image"
src="https://github.com/user-attachments/assets/b2ccce49-b5fd-4076-b47f-0eea6a53260c"
/>
## Database migrations
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added charting bounded context: HTTP API for metrics (uptime, CVE),
dataset registry, hosted dataset collection, background
collection/cleanup with opt-out env.
* New utilities: host bitmap operations and string-list/uint-list
parsers.
* New CLI tools to collect and backfill chart data.
* **Database**
* Migration and schema to store host time-series SCD chart data.
* **Tests**
* Extensive unit and integration tests for service, storage, caching,
cron, and utilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-23 17:43:23 +00:00
|
|
|
// RecordBucketData writes one or more entity bitmaps for the given bucket
|
|
|
|
|
// using the specified sample strategy. See SampleStrategy for semantics.
|
|
|
|
|
RecordBucketData(
|
|
|
|
|
ctx context.Context,
|
|
|
|
|
dataset string,
|
|
|
|
|
bucketStart time.Time,
|
|
|
|
|
bucketSize time.Duration,
|
|
|
|
|
strategy SampleStrategy,
|
|
|
|
|
entityBitmaps map[string][]byte,
|
|
|
|
|
) error
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Host is a minimal host type for authorization checks within the chart bounded context.
|
|
|
|
|
// The JSON tags matter: the OPA rego policy reads object.team_id via the JSON-encoded
|
|
|
|
|
// input, so renaming or dropping the tag silently breaks team-scoped authorization.
|
|
|
|
|
type Host struct {
|
|
|
|
|
ID uint `json:"id"`
|
|
|
|
|
TeamID *uint `json:"team_id"`
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// AuthzType implements platform_authz.AuthzTyper.
|
|
|
|
|
func (h *Host) AuthzType() string { return "host" }
|
|
|
|
|
|
|
|
|
|
// DataPoint represents a single data point in the chart response.
|
|
|
|
|
type DataPoint struct {
|
|
|
|
|
Timestamp time.Time `json:"timestamp"`
|
|
|
|
|
Value int `json:"value"`
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Response is the API response for chart data.
|
|
|
|
|
type Response struct {
|
|
|
|
|
Metric string `json:"metric"`
|
|
|
|
|
Visualization string `json:"visualization"`
|
|
|
|
|
TotalHosts int `json:"total_hosts"`
|
|
|
|
|
Resolution string `json:"resolution"`
|
|
|
|
|
Days int `json:"days"`
|
|
|
|
|
Filters Filters `json:"filters"`
|
|
|
|
|
Data []DataPoint `json:"data"`
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// RequestOpts captures the parsed query parameters for a chart request.
|
|
|
|
|
type RequestOpts struct {
|
|
|
|
|
Days int
|
|
|
|
|
// Resolution is the display granularity in hours. Must be 0 or a positive
|
|
|
|
|
// divisor of 24. 0 means "use the dataset's default resolution."
|
|
|
|
|
Resolution int
|
|
|
|
|
// TZOffsetMinutes is the client's UTC offset as reported by JavaScript's
|
|
|
|
|
// Date.getTimezoneOffset() (positive = west of UTC, e.g. CDT = 300).
|
|
|
|
|
// Used to align hourly bucket boundaries to local time.
|
|
|
|
|
TZOffsetMinutes int
|
|
|
|
|
// TeamID scopes the request to a single team. nil = global (authz + data
|
|
|
|
|
// both fall back to the user's accessible scope). *TeamID == 0 means
|
|
|
|
|
// hosts with no team assignment, matching Fleet's convention elsewhere.
|
|
|
|
|
TeamID *uint
|
|
|
|
|
LabelIDs []uint
|
|
|
|
|
Platforms []string
|
|
|
|
|
IncludeHostIDs []uint
|
|
|
|
|
ExcludeHostIDs []uint
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Filters captures the applied filters for a chart request.
|
|
|
|
|
type Filters struct {
|
|
|
|
|
TeamID *uint `json:"fleet_id,omitempty"`
|
|
|
|
|
LabelIDs []uint `json:"label_ids,omitempty"`
|
|
|
|
|
Platforms []string `json:"platforms,omitempty"`
|
|
|
|
|
IncludeHostIDs []uint `json:"include_host_ids,omitempty"`
|
|
|
|
|
ExcludeHostIDs []uint `json:"exclude_host_ids,omitempty"`
|
|
|
|
|
}
|