<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44609
# Details
This PR optimizes the historical data collection system in two ways:
1. Adds an additional index on the `host_scd_data` table allowing more
efficient lookups of rows by their `valid_to`, to optimize both closing
out open rows and deleting old rows
2. Implements batching in the job that deletes old rows, so that it no
longer blocks writes if the collection job happens to happen at the same
time as the cleanup job
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a, unreleased
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [ ] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
SQL explains -- before:
```
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| 1 | DELETE | host_scd_data | NULL | ALL | NULL | NULL | NULL | NULL | 144320 | 100.00 | Using where |
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
| 1 | UPDATE | host_scd_data | NULL | range | uniq_entity_bucket,idx_dataset_range | uniq_entity_bucket | 604 | const,const | 3030 | 100.00 | Using where |
+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
```
Using a test set of data (~144k "open" rows), UPDATES happened at 9 ops
per second.
after:
```
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+
| 1 | DELETE | host_scd_data | NULL | range | idx_valid_to_dataset | idx_valid_to_dataset | 5 | const | 55749 | 100.00 | Using where |
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+
+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
| 1 | UPDATE | host_scd_data | NULL | range | uniq_entity_bucket,idx_dataset_range,idx_valid_to_dataset | idx_valid_to_dataset | 609 | const,const,const | 4 | 100.00 | Using where; Using temporary |
+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
```
Using the same test set of data, UPDATES happened at 4,910 ops per
second.
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
this should significantly improve results!
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Cleanup now runs in controlled, ordered batches, removing only
closed/historical records while respecting cancellation; error reporting
for cleanup was strengthened.
* Added a new composite index on historical data to improve cleanup and
query performance.
* **Tests**
* Added tests and test helpers validating batched cleanup behavior,
preservation of open records, multi-batch operation, and cancellation
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44330, Resolves#44331
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests. (I'd defer integration tests to a
separate PR since this one is pretty large already.)
- [x] QA'd all new/changed functionality manually. I've tested this on
both the setup flow and the organization settings page. I haven't had
the time to test this on other places where we render the logo (macOS
setup experience / MDM migration dialog).
https://github.com/user-attachments/assets/95d4eae5-3da6-40f4-98a1-8575b97d96b3
## New Fleet configuration settings
- [x] Setting(s) is/are explicitly excluded from GitOps.
Will handle GitOps in a separate PR.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Organizations can upload custom logos for light and dark modes.
* Registration and Org Settings support logo file upload, preview,
per-mode replace/delete, and validation (size & image formats).
* Activity feed records logo changes/deletions; site nav displays
uploaded logos per theme.
* File uploader/preview adds a Fleet logo graphic option and improved
logo validation.
* Config/GitOps outputs now include separate dark/light logo fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43962
Adds two tables: `vpp_app_configurations` and
`in_house_app_configurations`
`vpp_app_configurations` has `team_id` unsigned **not nullable**, rather
than `team_id` nullable + `global_or_team_id`. This is following the
pattern in `software_title_display_names` and `software_title_icons`,
since software installers are team only and cannot be global.
`android_app_configurations` uses team_id + global_or_team_id but that
seems to be unnecessary.
`in_house_app_configurations` keys on `in_house_app_id` only — the
parent `in_house_apps` row already pins the team and platform.
Both use MEDIUMTEXT to store the XML configuration.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added database tables for storing VPP and in-house app configurations,
organized by team/platform with automatic cleanup when parent apps are
deleted.
* **Tests**
* Added migration tests to validate config storage fidelity, uniqueness
and platform-specific constraints, foreign-key enforcement, and
cascade-delete behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43885
Adds a migration and code to capture the value of the fleet managed
admin account if one exists. Changes file added for entire feature
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automatic password rotation for managed local admin accounts on macOS,
triggered after viewing activity.
* Provisioning now captures and persists the managed admin account
identifier (UUID) to support rotation and prevents that account from
being stored as a regular user.
* Hosts will request a best-effort recheck when the managed admin
identifier is not yet available.
* **Chores**
* Database schema updated to store rotation scheduling and pending
credential state.
* **Tests**
* Added tests covering UUID capture, conditional updates, migration, and
ingest behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43943
# Checklist for submitter
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
See
https://github.com/fleetdm/fleet/issues/43943#issuecomment-4329658412
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
## New Fleet configuration settings
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
(see https://github.com/fleetdm/fleet/pull/43877/changes)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled (should be done by
https://github.com/fleetdm/fleet/issues/43947)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a configuration option to preserve host activities during host
re-enrollment, letting admins choose whether activity history is
retained when hosts re-enroll.
* **Chores**
* Updated defaults and database migration state so the new setting is
present in stored and generated configs and in GitOps outputs.
* **Tests**
* Added unit, integration, migration, and GitOps fixtures to validate
behavior, serialization, and upgrade semantics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#44127
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed the unused Windows Updates feature: ingestion, parsing,
persistence APIs, and detail query; added a migration to drop the
related database table.
* **Tests**
* Removed unit and integration tests for Windows update parsing,
ingestion, persistence, and query inclusion.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#42948
- Updated `(mos *MacOSSetup) Validate()` and `(mos *MacOSSetup)
SetDefaultsIfNeeded()` to account for new fields
- Updated default creation and editing for team edit/creation paths
- Updated `generate-gitops` warning message from `macos_setup` to
`setup_experience`
- Updated fields types to optjson and updated test files
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
- Team edit and team creation through GitOps, validated config with `
curl -k -X GET 'https://localhost:8080/api/v1/fleet/fleets/:id'`
- New error message says `setup_experience` instead of `macos_setup`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added macOS MDM settings to control local account behavior: enable
managed local accounts (default false) and specify end-user local
account type (default "admin") for fleet and team configs. GitOps output
now highlights unsupported setup-experience cases.
* **Tests**
* Updated fixtures and integration tests to assert and persist the new
macOS local-account settings across config, team, and GitOps scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #42812
# Details
This PR implements a new bounded context, `chart`, with a single
endpoint `/charts`. The context encompasses a framework for recording
and querying and aggregating historical data for Fleet hosts, and
returning that data via the API for the purpose of charting.
This initial iteration has a full implementation of a dataset called
"uptime" which captures which hosts were online hour-by-hour (online
meaning, having been "seen" at some point during that hour). It has a
partial implementation of a "cve" dataset which will capture which hosts
were vulnerable to which CVEs during a given day.
### Data storage
Data is stored in an SCD (slowly-changing dimension) format in the
`host_scd_data` table, where the main "value" in a row is stored in the
`host_bitmap` column, which is a `mediumblob` where each bit encodes a
host ID (bit one represents host ID 1, bit 1444 represents host ID 1444,
etc.). The set of bits set on a row represents that hosts for which that
dataset is "on" during a given time period represented by the
`valid_from` (inclusive) and `valid_to` (exclusive) dates, where a
`valid_to` can have the special "sentinel" value 9999-12-31T00:00:00.000
meaning that the row is still "open" (the value represents everything
from `valid_from` to the present). Additionally an `entity_id` column
can be used for datasets with multiple dimensions, e.g. CVE exposure or
software usage which would have entity IDs representing CVEs or software
items respectively.
### Data collection
Data is collected via a cron job that runs every 10 minutes. Each
dataset has its own `Collect` method which will sample the data for the
given moment. For example the "uptime" dataset gathers the set of hosts
that are online at the moment, and the "cve" dataset will gather the set
of hosts that are vulnerable to each CVE at that moment. The sample can
then be recorded using one of two strategies:
* `accumulate`: bitwise OR the sample with any data already recorded for
the current hour, or add a new pre-closed row for that hour.
* `snapshot`: if there is no open row, create one with the sample and
`valid_to set` to the sentinel. Otherwise:
* If the sample has the same value as the current open row, do nothing
* If the sample has a different value and the current open row's
`valid_from` is within the same hour, update the current row's value
* If the sample has a different value and the current open row's
`valid_from` is not within the same hour, close the current open row and
start a new one with `valid_from` = the start of the current hour
### Data retrieval
1. Gets the set of host IDs to retrieve data for. This starts with the
set of host IDs in the requested fleet (or all the hosts a user has
access to if no `fleet_id` param was passed to the `/charts` endpoint),
and further whittled down by any filter options supplied with the
request (labels, platforms, etc.).
2. Finds all `host_scd_data` rows for the requested dataset and date
range (i.e. all rows whose `valid_from` is < the date range end and
`valid_to` is > the date range start).
3. Calculates the date ranges of the "buckets" to return datapoints for.
For the uptime chart we default to 3-hour buckets, so we want 8 buckets
per day.
4. Iterates over each bucket and finds the row or rows from
host_scd_data that cover that bucket range. For datasets using the
"accumulate" strategy, the values for those rows are ORed together. For
"snapshot"s, we take the one active at the bucket end time to represent
the bucket (e.g. "which hosts had a given CVE at the end of the day")
### Tools
This PR includes two dev tools that don't require deep review:
* **chart-backfill** - used to backfill data to various datasets for
testing
* **charts-collect** - used to collect data from a live server via the
API and put into a local hosts_scd_data table
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- With [front-end branch](https://github.com/fleetdm/fleet/pull/43878)
<img width="712" height="434" alt="image"
src="https://github.com/user-attachments/assets/b2ccce49-b5fd-4076-b47f-0eea6a53260c"
/>
## Database migrations
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added charting bounded context: HTTP API for metrics (uptime, CVE),
dataset registry, hosted dataset collection, background
collection/cleanup with opt-out env.
* New utilities: host bitmap operations and string-list/uint-list
parsers.
* New CLI tools to collect and backfill chart data.
* **Database**
* Migration and schema to store host time-series SCD chart data.
* **Tests**
* Extensive unit and integration tests for service, storage, caching,
cron, and utilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43978
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Optimized database performance for the app synchronization process by
adding an index to improve query efficiency and reduce CPU load on large
deployments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:**
Ref #34797
Ref #42675
## Problem
When a software installer spec has no `hash_sha256`, Fleet re-downloads
the package, re-extracts metadata, and re-upserts the DB on every GitOps
run, even if the upstream file hasn't changed. For deployments with 50+
URL-only packages across multiple teams, this wastes bandwidth and
processing time on every run.
## Solution
By default, use etags to avoid unnecessary downloads:
1. First run: Fleet downloads the package normally and stores the
server's ETag header
2. Subsequent runs: Fleet sends a conditional GET with `If-None-Match`.
If the server returns 304 Not Modified, Fleet skips the download,
metadata extraction, S3 upload, and DB upsert entirely
Opt-out with `always_download:true`, meaning packages continue to be
downloaded and re-processed on every run, same as today. No UI changes
needed.
```yaml
url: https://nvidia.gpcloudservice.com/global-protect/getmsi.esp?version=64&platform=windows
always_download: true
install_script:
path: install.ps1
```
### Why conditional GET instead of HEAD
Fleet team [analysis of 276 maintained
apps](https://github.com/fleetdm/fleet/pull/42216#issuecomment-4105430061)
showed 7 apps where HEAD requests fail (405, 403, timeout) but GET works
for all. Conditional GET eliminates that failure class: if the server
doesn't support conditional requests, it returns 200 with the full body,
same as today.
### Why opt-in
5 of 276 apps (1.8%) have stale ETags (content changes but ETag stays
the same), caused by CDN caching artifacts (CloudFront, Cloudflare,
nginx inode-based ETags). The `cache` key lets users opt in per package
for URLs where they've verified ETag behavior is correct.
Validation rejects `always_download: true` when hash_sha256` is set
## Changes
- New YAML field: `cache` (bool, package-level)
- New migration: `http_etag` VARCHAR(512) column (explicit
`utf8mb4_unicode_ci` collation) + composite index `(global_or_team_id,
url(255))` on `software_installers`
- New datastore method: `GetInstallerByTeamAndURL`
- `downloadURLFn` accepts optional `If-None-Match` header, returns 304
as `(resp, nil, nil)` with `http.NoBody`
- ETag validated per RFC 7232 (ASCII printable only, no control chars,
max 512 bytes) at both write and read time
- Cache skipped for `.ipa` packages (multi-platform extraInstallers)
- TempFileReader and HTTP response leak prevention on download retry
- Docs updated in `yaml-files.md`
## What doesn't change
- Packages with `hash_sha256`: existing hash-based skip, untouched
- FMA packages: FMA version cache, untouched
- Packages with `always_download: true`: identical to current behavior
- Fleet UI: no changes
## Test plan
Automated testing:
- [x] 16 unit tests for `validETag`
- [x] 8 unit tests for conditional GET behavior (304, 200, 403, 500,
weak ETag, S3 multipart, no ETag)
- [x] MySQL integration test for `GetInstallerByTeamAndURL`
- [x] All 23 existing `TestSoftwareInstallers` datastore tests pass
- [x] All existing service tests pass
Manual testing:
- [x] E2E: 86 packages across 6 CDN patterns, second apply shows 51
conditional hits (304)
- [x] @sgress454 used a local fileserver tool to test w/ a new instance
and dummy packages
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* ETag-based conditional downloads to skip unchanged remote installer
files.
* New always_download flag to force full re-downloads.
* **Tests**
* Added integration and unit tests covering conditional GETs, ETag
validation, retries, edge cases, and payload behavior.
* **Chores**
* Persist HTTP ETag and related metadata; DB migration and index to
speed installer lookups.
* Added installer lookup by team+URL to support conditional download
flow.
* **Bug Fix**
* Rejects using always_download together with an explicit SHA256 in
uploads.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Scott Gress <scott@fleetdm.com>
Co-authored-by: Scott Gress <scott@pigandcow.com>
Co-authored-by: Ian Littman <iansltx@gmail.com>
**Related issue:** Resolves#42883
Added a new premium GET /api/_version_/fleet/rest_api endpoint that
returns the contents of the embedded `api_endpoints.yml` artifact.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40809
**Orbit agent: key rotation replaces decrypt-then-re-encrypt:**
- When the disk is already encrypted, orbit now adds a new Fleet-managed
recovery key protector, removes old ones, and escrows the new key. The
disk is never decrypted.
- If key escrow fails, the rotated key is cached in memory and retried
on subsequent ticks without rotating again.
- Removes `DecryptVolume` and `decrypt()` (no longer called from
production code).
**Server: osquery query returns both protection_status and
conversion_status:**
- The `disk_encryption_windows` query now returns both columns instead
of just checking `protection_status = 1`. This lets the server correctly
identify a disk as encrypted via `conversion_status = 1` even when
`protection_status = 0`.
- New `directIngestDiskEncryptionWindows` function parses both values,
handles parse errors, and normalizes `protection_status = 2` (unknown)
to NULL.
**Server: new `bitlocker_protection_status` column and status logic:**
- Adds `bitlocker_protection_status` column to `host_disks` (DB
migration).
- When a disk is encrypted and key is escrowed but protection is off,
the host shows "Action required" with a detail message explaining the
issue, instead of misleadingly showing "Verified."
- `protection_status = 2` (unknown) and `NULL` (older orbit hosts) are
treated as protection on for backward compatibility.
- The `profiles_verified` and `profiles_verifying` branches in the
combined profiles+BitLocker summary now handle
`bitlocker_action_required`, counting those hosts as "pending".
Contributor docs updates: https://github.com/fleetdm/fleet/pull/43241
Public docs updates: https://github.com/fleetdm/fleet/pull/43243/changes
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
## fleetd/orbit/Fleet Desktop
- [x] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [x] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [x] Verified that fleetd runs on macOS, Linux and Windows
- [x] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Bug Fixes**
* Fixed Windows BitLocker encryption/decryption request loop on systems
with secondary drives and auto-unlock.
* **New Features**
* Added BitLocker recovery key rotation capability, allowing safe key
updates without full disk re-encryption.
* Enhanced BitLocker protection status tracking to correctly display
"Action required" when protection is disabled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Zed + Opus 4.6; prompt: Bump all DB migrations not merged to
`cherry-pick-40177-config-profile-name-status` to happen after the most
recent migration on that branch, while maintaining order otherwise, and
regenerate the test schema.
Resolves#40177 and subissues.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [sorta] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Profile names are now displayed alongside mobile device management
commands for installing or removing profiles. These names are visible in
command details modals and within device activity timelines.
* Added "NotNow" status for deferred profile commands, providing
improved transparency into which profiles are being managed and the
current status of profile installation or removal operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Adds the `host_managed_local_account_passwords` table to persist encrypted managed local admin account passwords and track MDM delivery status for ADE-enrolled macOS hosts (#42942).
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#42841
This change is just new columns in a table. No other functional changes.
# Checklist for submitter
## Testing
- [x] Added/updated automated tests
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added tracking for Windows device enrollment configuration status,
including timestamps indicating when devices entered the
awaiting-configuration state to improve enrollment lifecycle management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43046
# Checklist for submitter
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
**Related issue:** Resolves#42881
- Added user_api_endpoints table to track per user API endpoint
permissions.
- Added service/api_endpoints, used to handle service/api_endpoints.yml
artifact.
- Added check on server start that makes sure that
service/apin_endpoints.yml is a subset of router routes.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#42853
This PR simply adds the `require_all_software_windows` config option. It
doesn't use it. The logic to use it will be hooked up in subsequent PRs.
The fleetctl TestIntegrationsPreview test is expected to fail since it
builds the server against main and doesn't know about our new config
option.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- Not exported. generate-gitops does not export
require_all_software_windows (or require_all_software_macos either). The
generateControls function (generate_gitops.go) outputs a "TODO: update
with your setup_experience configuration" placeholder when any setup
experience config exists, rather than exporting individual field values.
This is a pre-existing limitation that applies equally to both fields -
not something introduced by our PR.
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- Yes. PR #42046 adds require_all_software_windows to both docs/REST
API/rest-api.md and docs/Configuration/yaml-files.md.
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- Yes, it gets cleared to false - both when setup_experience: is present
without the field, and when setup_experience: is omitted entirely. This
is the same behavior as the existing require_all_software_macos field
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
- Covered by #42854 (frontend subtask). The existing macOS checkbox in
InstallSoftwareForm.tsx:271 already checks gitOpsModeEnabled to disable
itself. The Windows checkbox to be added in #42854 will follow the same
pattern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Windows setup experience software requirement setting. When
enabled, Windows devices will cancel the Autopilot setup if any required
software installation fails.
* **Tests**
* Added test coverage for the new Windows software requirement
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** The entire ACME feature branch merge
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
---------
Co-authored-by: Jordan Montgomery <elijah.jordan.montgomery@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#37546
Docs: https://github.com/fleetdm/fleet/pull/42780
Demo: https://www.youtube.com/watch?v=K44wRg9_79M
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automatic retry for Android certificate installations: failed installs
are retried up to 3 times before marked terminal.
* Installation activities recorded: install/failed-install events (with
details) are logged for better visibility and troubleshooting.
* Resend/reset actions now reset retry state so retries behave
predictably after manual resend.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41670
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
## Database migrations
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automatic recovery lock password rotation for Mac devices—passwords
now rotate 1 hour after being viewed or accessed via the API, enhancing
security.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Resolves#41540
* Added new computed column to determinate whether query_result has
data.
* Added new index to query_results to to cover all query patterns.
* Refactored queries used in host report page to improve performance.
* Fixed various bugs with around query filtering for host reports.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41815
### Changes
- Extracted patch policy creation to `pkg/patch_policy`
- Added a `patch_query` column to the `software_installers` table
- By default that column is empty, and patch policies will generate with
the default query if so
- On app manifest ingestion, the appropriate entry in
`software_installers` will save the override "patch" query from the
manifest in patch_query
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
- Relied on integration test for FMA version pinning
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#42008
# Details
Step one in https://github.com/fleetdm/fleet/issues/40171.
This PR adds a new `exceptions` subsection to the current GitOps config,
with boolean keys for software, secrets and labels. For existing
instances a migration is included to set labels and secrets to `true`.
For new instances, only `secrets` will be `true`.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a, will put changelog in when more functionality is implemented.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
(https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] ran migration and verified that app config had `gitops.exceptions`
with `software: false, secrets: true, labels: true`
- [X] created a new instance and verified that that app config had
`gitops.exceptions` with `software: false, secrets: true, labels: false`
- [X] verified that the PATCH /config API works and can update
exceptions independently of other config
## Database migrations
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
n/a
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
n/a
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
n/a
## New Fleet configuration settings
- [X] Setting(s) is/are explicitly excluded from GitOps
these will not be set in GitOps, since they're _about_ how GitOps works.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* GitOps configuration now supports exception settings for granular
resource control. Administrators can configure which specific resource
types (labels, software, and secrets) are included in or excluded from
GitOps mode operations.
* **Improvements**
* Improved GitOps configuration handling to preserve exception settings
during partial updates and system migrations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41664
Switching `kernel_host_counts` to the established swap pattern. Reduce
load on the DB writer by moving the large read to the DB reader.
Do `CleanupSoftwareTitles` in batches. With a single large
select/delete, it took > 16 minutes. In batches, it took ~1.5 minutes in
loadtest with 100K hosts.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Alerted the release DRI if additional load testing is needed
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance Improvements**
* Added indexing and a batched swap/insert flow to speed up
vulnerability-related queries and lower maintenance contention.
* Batched cleanup of orphaned records to reduce long-running delete
operations.
* **Reliability**
* Migration removes a legacy constraint to simplify data maintenance and
avoid migration failures.
* Scheduled vulnerability refresh now runs more atomically to reduce
disruption.
* **Tests**
* Updated assertion logic to improve test clarity for host-count
verification.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Resolves#35484
Concurrent execution of GitOps apply runs and
RecordPolicyQueryExecutions led to database locking issues when the
policy_membership table was large. This occurred because the cleanup
process (DELETE operations) was bundled within the same transaction as
the GitOps policy updates. To resolve this, the deletion logic has been
batched and moved outside the primary GitOps transaction, reducing lock
contention.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41664
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Alerted the release DRI if additional load testing is needed
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Optimized database performance for vulnerability processing to reduce
contention during routine operations.
* Improved query efficiency for software cleanup processes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The [Test DB changes CI
test](https://github.com/fleetdm/fleet/actions/workflows/test-db-changes.yml?query=branch%3Amain)
has been failing on main since yesterday with:
```
❌ fail: server/datastore/mysql/migrations/tables/20260218165545_FixMismatchedSoftwareTitles.go has an older timestamp than server/datastore/mysql/migrations/tables/20260218175704_FMAActiveInstallers.go
this might cause problems if this change is merged
please update the timestamp of server/datastore/mysql/migrations/tables/20260218165545_FixMismatchedSoftwareTitles.go
Error: Process completed with exit code 1.
```
I think because some migrations were merged to main after the fleet 4.82
tag, but they came from a pre-4.82 tag branch. This PR moves the
migrations to the earliest possible time _after_ all of the 4.82
migrations.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40183 and #40182
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35799
Loadtest results for 100K hosts and 300K software titles.
```
=== Performance Test Results: No team_id (all teams) ===
Description Average Worst Results
----------- ------- ----- -------
Page 0, hosts_count DESC 229ms 241ms 20 items
Page 0, hosts_count ASC 203ms 211ms 20 items
Page 1, hosts_count DESC 339ms 423ms 20 items
Page 1000, hosts_count DESC 202ms 219ms 20 items
100 per_page, hosts_count DESC 620ms 708ms 100 items
Default sort (no order params) 229ms 245ms 20 items
Order by name ASC, page 0 4.642s 4.785s 20 items
Order by name ASC, page 1000 6.418s 6.771s 20 items
Vulnerable only 3.431s 3.496s 20 items
Search 'chrome' 9.6s 10.111s 20 items
Known exploit filter 9.792s 10.102s 20 items
Min CVSS score 7.0 12.368s 12.665s 20 items
CVSS range 7.0-9.0 12.221s 12.523s 20 items
Available for install 87ms 93ms NO RESULTS
Self-service only 4.46s 4.757s 20 items
=== Performance Test Results: team_id=0 (no team / unassigned) ===
Description Average Worst Results
----------- ------- ----- -------
Page 0, hosts_count DESC 378ms 404ms 20 items
Page 0, hosts_count ASC 339ms 345ms 20 items
Page 1, hosts_count DESC 478ms 513ms 20 items
Page 1000, hosts_count DESC 398ms 417ms 20 items
100 per_page, hosts_count DESC 864ms 1.025s 100 items
Default sort (no order params) 399ms 411ms 20 items
Order by name ASC, page 0 5.346s 5.41s 20 items
Order by name ASC, page 1000 7.444s 7.615s 20 items
Search 'chrome' 9.051s 9.245s 20 items
Known exploit filter 10.511s 10.884s 20 items
Min CVSS score 7.0 16.589s 16.701s 20 items
CVSS range 7.0-9.0 15.878s 15.999s 20 items
Available for install 1.394s 1.429s 1 items
Self-service only 1.4s 1.456s 1 items
```
Documented in the issue:
The fix includes a small behavior change. The default primary sort of
/software/titles remains host_counts, but the secondary sort is now
software_title_id and not name. This was necessary to optimize the
endpoint. This means that if you have 1 host in your fleet, the software
page will not show the software titles ordered by name anymore. For
large fleets, this does not matter since all titles generally have
different host counts.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Optimized the software titles endpoint for improved query performance
and faster data retrieval.
* Enhanced database efficiency when retrieving software information,
resulting in better overall system responsiveness and reduced query
times.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39672
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information. (Will add in the next task, the bigger piece of
work)
## Testing
- [ ] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35805
Fixed to make sure software host counts tables never have host counts of
0.
Planning to loadtest this fix along with the follow up fix for
https://github.com/fleetdm/fleet/issues/35799
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed accumulation of dead rows in software host count tracking,
improving data accuracy and system performance.
* Enhanced validation to ensure consistent and reliable software
availability records.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- **Gitops specify FMA rollback version (#39582)**
- **Fleet UI: Show versions options for FMA installers (#39583)**
- **rollback: DB and core implementation (#39650)**
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#31919
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
---------
Co-authored-by: Jonathan Katz <44128041+jkatz01@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Carlo DiCelico <carlo@fleetdm.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves: #35528 (followup)
This migration should fix `.pkg` installers that have a mismatched
software title, as well as software (versions) that are using the wrong
title. It's a little bit risky, as it could modify `software`,
`software_installers`, and `software_titles` data.
For software versions it just checks if the title's source is not the
same as the software. For installers it uses installers with
platform=`darwin` where the title's source isn't `apps`, which should
probably be the only case after the previous migration to convert titles
with source `pkg_packages` to `apps`.
Changefile was already added for 35528 in a previous PR.
# Checklist for submitter
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
