<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44723
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Strengthened validation of sorting/order parameters across many list
and cursor-based endpoints — unsupported sort keys now return explicit
errors and prevent unsafe queries.
* Labels listing: label-list pagination query name changed; ordering by
host_count is rejected when host counts are disabled (validated at
request parsing).
* **Tests**
* Added/expanded tests covering allowed order keys, rejection of unknown
keys, and pagination behavior for multiple listing APIs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44609
# Details
This PR optimizes the historical data collection system in two ways:
1. Adds an additional index on the `host_scd_data` table allowing more
efficient lookups of rows by their `valid_to`, to optimize both closing
out open rows and deleting old rows
2. Implements batching in the job that deletes old rows, so that it no
longer blocks writes if the collection job happens to happen at the same
time as the cleanup job
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a, unreleased
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [ ] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
SQL explains -- before:
```
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| 1 | DELETE | host_scd_data | NULL | ALL | NULL | NULL | NULL | NULL | 144320 | 100.00 | Using where |
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
| 1 | UPDATE | host_scd_data | NULL | range | uniq_entity_bucket,idx_dataset_range | uniq_entity_bucket | 604 | const,const | 3030 | 100.00 | Using where |
+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
```
Using a test set of data (~144k "open" rows), UPDATES happened at 9 ops
per second.
after:
```
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+
| 1 | DELETE | host_scd_data | NULL | range | idx_valid_to_dataset | idx_valid_to_dataset | 5 | const | 55749 | 100.00 | Using where |
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+
+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
| 1 | UPDATE | host_scd_data | NULL | range | uniq_entity_bucket,idx_dataset_range,idx_valid_to_dataset | idx_valid_to_dataset | 609 | const,const,const | 4 | 100.00 | Using where; Using temporary |
+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
```
Using the same test set of data, UPDATES happened at 4,910 ops per
second.
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
this should significantly improve results!
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Cleanup now runs in controlled, ordered batches, removing only
closed/historical records while respecting cancellation; error reporting
for cleanup was strengthened.
* Added a new composite index on historical data to improve cleanup and
query performance.
* **Tests**
* Added tests and test helpers validating batched cleanup behavior,
preservation of open records, multi-batch operation, and cancellation
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44330, Resolves#44331
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests. (I'd defer integration tests to a
separate PR since this one is pretty large already.)
- [x] QA'd all new/changed functionality manually. I've tested this on
both the setup flow and the organization settings page. I haven't had
the time to test this on other places where we render the logo (macOS
setup experience / MDM migration dialog).
https://github.com/user-attachments/assets/95d4eae5-3da6-40f4-98a1-8575b97d96b3
## New Fleet configuration settings
- [x] Setting(s) is/are explicitly excluded from GitOps.
Will handle GitOps in a separate PR.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Organizations can upload custom logos for light and dark modes.
* Registration and Org Settings support logo file upload, preview,
per-mode replace/delete, and validation (size & image formats).
* Activity feed records logo changes/deletions; site nav displays
uploaded logos per theme.
* File uploader/preview adds a Fleet logo graphic option and improved
logo validation.
* Config/GitOps outputs now include separate dark/light logo fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#34464
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
---
## What
GitOps `--dry-run` was succeeding when `macos_manual_agent_install` was
set to `true` and a `macos_script` was configured under
`setup_experience`, but the actual GitOps run would fail with:
```
Couldn't add setup experience script. To add script, first disable macos_manual_agent_install.
```
## Why
The `manual_agent_install` conflict validation only existed server-side
in `ee/server/service/setup_experience.go:SetSetupExperienceScript()`.
The script upload call (`uploadMacOSSetupScript()`) was gated by
`!opts.DryRun` in `server/service/client.go`, so during dry-run the
upload was skipped entirely and the validation never fired.
## Fix
Added client-side validation in `server/service/client.go` at the point
where the YAML-parsed `MacOSSetup` struct is processed — before the
script file is validated and loaded. This check runs for **both dry-run
and real runs**, catching the conflict early. Two code paths were fixed:
1. **Team path** (~line 803): Checks `setup.ManualAgentInstall.Value`
when `setup.Script.Value` is set
2. **No-team path** (~line 2603): Checks
`macOSSetup.ManualAgentInstall.Value` when `macOSSetup.Script.Value` is
set
## How I reproduced the issue locally
### Prerequisites
- MySQL and Redis running via Docker: `docker compose up -d mysql_test
redis`
### Steps
1. Wrote an integration test
(`TestDryRunMacOSSetupScriptWithManualAgentInstallConflict`) that:
- Creates a GitOps user and fleetctl config
- Creates a bootstrap package server serving `testdata/signed.pkg`
- Creates a `.sh` script file with `echo "setup script"`
- Creates a **global config** YAML (minimal server settings)
- Creates a **team config** YAML with `macos_manual_agent_install:
true`, `macos_script: <path>`, and `macos_bootstrap_package: <url>`
- Runs `fleetctl gitops --dry-run` and asserts it fails
- Runs `fleetctl gitops` (no dry-run) and asserts it fails
2. Ran the test **before the fix** — confirmed the bug:
```
Dry-run error: <nil> ← BUG: should have failed
Real run error: ...status 422...first disable macos_manual_agent_install
← correctly fails
```
3. Applied the fix and re-ran — **both dry-run and real run now fail**
with the `macos_manual_agent_install` conflict error.
### Test command
```bash
MYSQL_TEST=1 REDIS_TEST=1 go test -v \
-run TestIntegrationsEnterpriseGitops/TestDryRunMacOSSetupScriptWithManualAgentInstallConflict \
./cmd/fleetctl/integrationtest/gitops/... -count=1 -timeout 600s
```
Both sub-tests (team and no-team paths) pass. All related existing tests
continue to pass:
- `TestMacOSSetup`, `TestMacOSSetupScriptWithFleetSecret`,
`TestDeletingNoTeamYAML`, `TestDisallowSoftwareSetupExperience`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* GitOps dry-run now correctly fails when a macOS setup configuration
combines manual agent installation with a provided setup script,
preventing false-positive dry-run success.
* **Tests**
* Added unit and integration regression tests to verify dry-run and
real-run rejection of conflicting macOS setup configurations for both
team-scoped and unassigned host scopes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#42887.
From Claude's audit:
```
[...]
Concerns worth addressing
A. Catalog drift is the real long-term risk. Today the yaml is curated.
If a future engineer adds (say) POST /users/api_only, PATCH /users/api_only/:id, POST /users/roles/spec,
POST /password_reset, or any session-issuing route, an allowlisted api_only user can clone themselves or
broaden a peer's allowlist.
Suggest a CI test that hard-fails if any of those route prefixes show up in api_endpoints.yml,
plus a comment at the top of the yaml listing the categories that must never be added (user/role/invite/password/session/SSO).
[...]
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added validation tests for API endpoint configuration to ensure
security compliance and proper detection of restricted endpoint
combinations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#44190
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a periodic cleanup job that removes aged, acknowledged Windows
MDM command-queue entries to reduce write pressure during ACK
processing.
* **Bug Fixes**
* Pending-command detection now excludes already-ACKed commands from
dispatch; queue rows are retained after ACK and cleaned later.
* **Tests**
* Added and updated tests to validate cleanup behavior and revised
ACK/queue semantics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44533
Adds an option to return a PEM certificate from the request_certificate
endpoint, rather than the PKCS7 envelope an EST server returns. This
allows it to be more easily used in scripts without conversions, at the
(small) cost of among other things dropping the PKCS7 envelope which
could be signed by the server, etc(though the PEM cert itself should
also be)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* The "Request a Certificate" endpoint can optionally return the issued
certificate as a PEM-encoded X.509 CERTIFICATE block instead of a
PEM-encoded PKCS#7 envelope.
* **Tests**
* Added comprehensive tests covering PEM conversion, tolerance for
base64 whitespace/newlines, error handling for malformed PKCS#7, and
multi-certificate envelope cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Found this missed TODO from some old story work.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced OTA enrollment profile validation to properly enforce
end-user authentication requirements when configured, now returning
appropriate error responses for requests missing required authentication
credentials.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40459
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
Recording:
https://drive.google.com/file/d/1_XqLyy-oY-WnIa97R4t9HihiBq3Fui6n/view?usp=drive_link
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Wiping a host now cancels all upcoming and queued activities for that
host in a single, atomic operation to avoid intermediate activations.
* **Bug Fixes**
* Wipe response handling now distinguishes success vs failure and
reliably cancels queued activities; datastore errors during host lookup
or cancellation are surfaced.
* Device lock/erase flows consistently update and propagate datastore
errors.
* **Tests**
* Added integration and datastore tests validating wipe clears upcoming
activities across macOS, Windows, Linux, and mixed-host scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Magnus Jensen <magnus@fleetdm.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Changes**
* Updated the default Apple DEP enrollment profile: devices are now
marked as supervised and the MDM profile is non-removable.
* Simplified the returned default profile by removing several previously
hard-coded enrollment defaults, reducing complexity and aligning
behavior with external templates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41418
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
To manually QA, I put an early return with `msg.Fail` in the
`mdm_scep.go` file under PKIOperation method, and then triggered a SCEP
renewal.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Activity logging for Apple MDM enrollment profile renewal failures to
improve auditing and diagnostics.
* Host display enhancements: include computer name and hardware model to
improve host identification in activities and UI.
* **Tests**
* Integration tests verifying enrollment renewal failure activity
creation, association to the correct host, and activity payload
contents.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43928
This PR adds a Redis-backed cache in front of the two host-by-key
lookups on the agent auth paths.
Docs: https://github.com/fleetdm/fleet/pull/44504
## What changes
**Read path (osquery/orbit auth):**
- `LoadHostByNodeKey` and `LoadHostByOrbitNodeKey` now check Redis
before falling through to MySQL.
- Successful lookups are cached for 60s ± 10% jitter (configurable via
`FLEET_REDIS_HOST_CACHE_TTL`).
- `NotFound` results are cached for 5s as a negative entry, dampening
repeated probes for keys that
do not exist (deleted hosts whose agents are still polling, attacker
scans, retry storms).
- Concurrent lookups for the same key collapse into one DB query via
`singleflight`. The shared
query runs under a context detached from any one caller's deadline so
the leader giving up does
not abort the work for joiners. The shared query is itself bounded by a
30s timeout so a wedged
DB call cannot pin the singleflight slot indefinitely.
**Write path (invalidations):**
- These methods now invalidate the cache after a successful inner call:
`UpdateHost`, `SerialUpdateHost`, `UpdateHostOsqueryIntervals`,
`UpdateHostRefetchRequested`,
`UpdateHostRefetchCriticalQueriesUntil`,
`UpdateHostIdentityCertHostIDBySerial`, `EnrollOsquery`,
`EnrollOrbit`, `NewHost`, `DeleteHost`, `DeleteHosts`,
`CleanupExpiredHosts`,
`CleanupIncomingHosts`, `AddHostsToTeam`.
- `AddHostsToTeam`, `DeleteHosts`, `CleanupExpiredHosts`, and
`CleanupIncomingHosts` use a pipelined
batch invalidator so 10k-host operations stay in the millisecond range
instead of taking minutes
of sequential round-trips.
- Inner-call errors are not invalidations: a failing write leaves cached
state intact.
**Configuration:**
- New flags `FLEET_REDIS_HOST_CACHE_ENABLED` (default `true`) and
`FLEET_REDIS_HOST_CACHE_TTL`
(default `60s`).
- Server refuses to start if the cache is enabled with `TTL <= 0`.
**Observability:**
- Three new OTEL counters under the `fleet` meter:
- `fleet.host_cache.lookups{result=hit|negative_hit|miss}`
- `fleet.host_cache.errors{op=get|set|del}`
-
`fleet.host_cache.invalidations{reason=update|enroll|team|delete|cert}`
- A pre-built SigNoz dashboard ships in
`tools/signoz/host_cache_dashboard.json`.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Optional Redis-backed host lookup cache for osquery and orbit auth,
with automatic invalidation and metrics/monitoring dashboard.
* **Bug Fixes**
* Fixed host-removal batching so cache-related removals use correct
chunks.
* **Tests**
* Added comprehensive host-cache unit tests covering hits, negative
cache, invalidation, concurrency, and JSON round-trips.
* **Chores**
* New config flags to enable the cache and set TTL (default 60s ±10%
jitter).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This is no longer used, but gets flagged by AI.
https://github.com/fleetdm/fleet/pull/44511#discussion_r3169700014
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Removed a now-redundant exported activity list from the codebase,
simplifying internal activity declarations. This streamlines internal
structures without changing user-visible behavior or altering existing
activity types.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44194
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Team-level disk encryption can be toggled when at least one MDM
platform (Windows or Apple) is configured, enabling BitLocker control
for Windows-only deployments.
* **Bug Fixes**
* Updates validation to reject disk-encryption changes only when no MDM
platforms are configured.
* **Tests**
* Added coverage for platform combinations and expected behavior,
including Apple-specific profile creation when applicable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Resolves#36976
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Label operations (create, edit, delete) now generate activities shown
in the activity feed with label and optional fleet context.
* Host label add/remove operations emit corresponding label edited
activities; duplicate label names are deduplicated.
* Label activity types are selectable/filterable in the activity
dashboard.
* **Tests**
* Added unit, integration, and UI tests covering label activity
emission, rendering, filtering, and GitOps label lifecycle scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44170 and Resolves#44422
Pagination is now pushed into each branch of the merged query, so
per-tick work scales with page size instead of total commands. The
Windows side was rewritten to avoid a disjunctive join that forced a
nested-loop plan. `per_page` is capped (default 10), `page` is capped,
and `order_key` is enforced against a closed allowlist on both code
paths. Cursor pagination is fixed and is the recommended way to traverse
beyond the page cap.
This PR improves but does not fix the use case of fetching commands from
all hosts. Deprecate usage without host_identifier:
https://github.com/fleetdm/fleet/pull/44392/changes
API doc updates: https://github.com/fleetdm/fleet/pull/44292
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enforced pagination on MDM commands list: per_page defaults to 10 (max
1,000) and page is capped at 100; traversal beyond page 100 requires
cursor pagination via after.
* **Bug Fixes / Performance**
* Improved MDM command listing performance and de-duplication for large
queries; fixed SQL error when combining host identifier with cursor
pagination.
* **Validation**
* Requests exceeding pagination caps return 400; invalid sort keys
return 422.
* **Tests**
* Added tests for pagination boundaries, cursor behavior, sort-key
validation, and error responses.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Problem
The `hostVPPInstalls` function in `server/datastore/mysql/software.go`
contains a SQL condition:
```sql
(hvsi.platform != 'android' OR ncr.id IS NULL) AND
```
This is a logical tautology — it **never filters any rows**:
- **Android rows**: `ncr.id` is always `NULL` because Android installs
use Google's Android Management API, not nanoMDM. The condition
evaluates to `(FALSE OR TRUE) = TRUE`.
- **Apple rows**: The first operand `hvsi.platform != 'android'` is
`TRUE`, so the whole expression is `TRUE` regardless of `ncr.id`.
The condition was likely added during early Android VPP support to guard
against unexpected NCR joins for Android. However, since
`nano_command_results` is only written by the nanoMDM Apple MDM storage
layer (`server/mdm/nanomdm/storage/mysql/queue.go:168`), the guard can
never trigger.
Elsewhere in the codebase, the canonical pattern for NCR filtering is:
```sql
-- vpp.go:248, software_installers.go:1812
(ncr.id IS NOT NULL OR (:platform = 'android' AND ncr.id IS NULL))
```
This pattern has *different semantics* — it filters per-app aggregate
status counts to only include confirmed installs. The `hostVPPInstalls`
function serves the host software list where showing all statuses
(including pending) is intentional, so no NCR filter is needed.
## Changes
- Removed the dead condition from the `last_vpp_install` UNION branch
- Added a clarifying comment explaining why no NCR filter is applied and
how this differs from other query sites
- Added changelog entry
## Testing
- No behavior change — the removed condition was always TRUE
- Existing tests pass without modification
- `go build ./server/datastore/mysql/...` compiles clean
#android #sql #cleanup
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
**Related issue:** Resolves#44369
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Prevented operations on Windows MDM profiles for hosts without active
enrollments.
* Batch processing now skips hosts lacking current enrollments so only
enrolled hosts receive queued commands.
* Strengthened profile-removal checks to avoid acting on orphaned
profile rows.
* **Tests**
* Added regression tests covering orphaned enrollment/profile scenarios
and mixed-host batch processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43962
Adds two tables: `vpp_app_configurations` and
`in_house_app_configurations`
`vpp_app_configurations` has `team_id` unsigned **not nullable**, rather
than `team_id` nullable + `global_or_team_id`. This is following the
pattern in `software_title_display_names` and `software_title_icons`,
since software installers are team only and cannot be global.
`android_app_configurations` uses team_id + global_or_team_id but that
seems to be unnecessary.
`in_house_app_configurations` keys on `in_house_app_id` only — the
parent `in_house_apps` row already pins the team and platform.
Both use MEDIUMTEXT to store the XML configuration.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added database tables for storing VPP and in-house app configurations,
organized by team/platform with automatic cleanup when parent apps are
deleted.
* **Tests**
* Added migration tests to validate config storage fidelity, uniqueness
and platform-specific constraints, foreign-key enforcement, and
cascade-delete behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#44111
Customers reported certificates deployed via custom SCEP proxy were
silently failing to auto-renew, leaving devices with expired certs. Five
compounding bugs were causing this:
### 1. Cert metadata was wiped on every reconcile re-render
`BulkUpsertMDMManagedCertificates` unconditionally overwrote
`not_valid_before`, `not_valid_after`, and `serial` in `ON DUPLICATE KEY
UPDATE`. Since the SCEP-proxy render-time payload has those fields nil
(cert details aren't known until the device completes the handshake and
osquery reports), every renewal trigger wiped them. Once NULL, the
renewal cron's `HAVING validity_period IS NOT NULL` clause excluded the
row — silently disabling future renewal attempts.
Fixed by switching those columns to `COALESCE(VALUES(col), col)` so a
nil incoming value preserves the existing value. DigiCert's flow (which
does set the fields) and osquery's separate UPDATE in
`updateHostMDMManagedCertDetailsDB` are unaffected.
### 2. 1-hour challenge TTL was too short for offline devices
The challenge is generated at profile-render time but consumed when the
device makes its SCEP request — which can be hours or days later (laptop
asleep, on a plane, etc.). Devices that didn't pick up the
InstallProfile push within the hour hit `challenge not found: sql: no
rows in result set` and the renewal failed.
Bumped `OneTimeChallengeTTL` from 1 hour to 7 days. Once consumed, the
challenge is deleted immediately regardless of TTL.
### 3. Renewal cron re-fired on in-flight deliveries
`WHERE hp.status IS NOT NULL` matched `'pending'` and `'verifying'` too,
so a host whose delivery was still in flight (e.g., offline laptop)
would have its profile re-rendered with a fresh challenge every cron
tick — generating orphan nano commands and challenge rows hourly.
Pre-fix this was masked by bug 1; once the COALESCE preserves cert
metadata, the loop becomes visible.
Tightened the filter to `WHERE hp.status IN ('verified', 'failed')` —
settled states only.
### 4. iOS/iPadOS managed-cert profiles short-circuited to verified
before cert metadata synced
iOS/iPadOS profiles short-circuit `pending` → `verified` directly on MDM
ack (no `verifying` step), since osquery isn't available to drive the
standard verification cycle. That's correct for non-cert profiles, but
for managed-cert profiles it created a window where the renewal cron saw
`status='verified'` paired with stale cert metadata still in the renewal
window — and the new `IN ('verified', 'failed')` filter from bug 3 kept
matching, re-firing renewal each tick until `CertificateList` ingestion
eventually caught up.
Fixed by parking iOS/iPadOS managed-cert profiles at `'verifying'` on
MDM ack and flipping them to `'verified'` from
`updateHostMDMManagedCertDetailsDB` once fresh cert metadata arrives —
i.e., reusing the existing state machine instead of inventing a parallel
"renewal in flight" tracking column. The `EXISTS(SELECT 1 FROM
host_mdm_managed_certificates ...)` check is folded into the existing
platform-detection query, so no extra round-trip. macOS is unaffected:
the new flip is redundant with `VerifyHostMDMProfiles` but idempotent.
**Trade-off worth flagging:** if `CertificateList` ingestion never runs
for an iOS managed-cert profile (broken cron, device offline
indefinitely), the profile sits at `'verifying'` and the renewal cron's
filter excludes it. In practice both run on the same Apple MDM cron loop
— if one is broken, much else is too — but it's a sharper failure mode
than letting renewals re-fire wastefully.
### 5. Permanent-failure profiles loop hourly through the renewal cron
Once `'failed'` was added to the cron's status filter (bug 3), there was
no longer any circuit breaker for profiles that fail at render time for
non-transient reasons — CA deleted from app config, IDP variables
missing from host, premium license downgraded. Each cron tick (1h
interval) the cron flips `'failed'` → NULL, reconcile re-renders and
immediately re-fails via `fleet.MarkProfilesFailed`, status returns to
`'failed'`, repeat. Pre-fix this was masked by bug 1 (metadata wipe
acted as accidental circuit breaker); once metadata is preserved (bug 1
fix), the loop becomes real and produces a profile-render attempt + nano
command per failed cert per hour.
Added a `renewalFailedRetryBackoff` constant (24h) and gated the
`'failed'` branch on `hp.updated_at < DATE_SUB(NOW(), INTERVAL
renewalFailedRetryBackoff SECOND)`. Transient SCEP-server outages still
recover (within at most 24h, well under any cert validity window).
Permanent failures still get retried daily (so a customer fixing the
underlying issue eventually auto-recovers), but they don't churn nano
commands hourly. `'verified'` rows in the renewal window are unaffected
— they bypass the gate.
## Tests
- `testMDMManagedSCEPCertificates`: three new sub-tests covering (a)
cert-metadata preservation across reconcile re-renders, (b)
in-flight-status skip behavior, (c) the permanent-failure backoff.
Exercised against both NDES and Custom SCEP via the existing
table-driven harness.
- New `testIOSManagedCertProfileStaysVerifying`: verifies that on iOS, a
managed-cert profile stays at `'verifying'` after MDM ack and only flips
to `'verified'` once `UpdateHostCertificates` ingests fresh cert
metadata.
- New `challenges_test.go` covering `NewChallenge`/`ConsumeChallenge`
lifecycle and TTL boundaries.
- `TestCustomSCEPIntegration`: updated the hardcoded 2-hour challenge
backdate to use `fleet.OneTimeChallengeTTL` so it stays correct as the
constant evolves.
- New `TestCustomSCEPRenewalPreservesCertMetadata` end-to-end test:
drives the full reconcile path (rather than calling the bare datastore
method) so a future change to the render-time payload structure can't
silently regress the COALESCE preservation.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
Provides better errors on invalid/unexpected sort keys passed to
`/api/v1/fleet/commands`, `/api/v1/fleet/mdm/commands` and
`/api/v1/fleet/mdm/apple/commands` endpoints
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved validation for invalid `order_key` values on MDM command
endpoints (`/api/v1/fleet/commands`, `/api/v1/fleet/mdm/commands`, and
`/api/v1/fleet/mdm/apple/commands`), ensuring only approved sorting
parameters are accepted.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#42898
I was waiting for my [upstream
PR](https://github.com/micromdm/nanomdm/pull/250) to be merged, but I've
waited for 2+ weeks now, so I'll go ahead and do the same change here,
and then if the maintainer requests change I can update this fix
retrospectively
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information. **None, since it's unused in our codebase at this
point**
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Prevented nil push entries from causing panics during concurrent push
processing
* Adjusted worker allocation so concurrency scales down when batch sizes
are smaller
* Clamped configured worker count to a minimum of 1 (documented default
behavior)
* **Tests**
* Added regression test ensuring safe handling of nil entries in
concurrent push inputs and updated test harness to exercise
reduced-worker scenarios
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#42741
## Problem
`goval_dictionary.Analyze` opened a `*sql.DB` via `LoadDb` but never
closed it. `pkg/download/download.go` atomically renames the goval
sqlite on each refresh, unlinking the old inode while the pool still
held FDs on it. lsof showed them as `(deleted)`, accumulating over days
until Fleet server restart.
## Fix
- New `Database.Close()` that delegates to the underlying `*sql.DB`.
- `defer func() { _ = db.Close() }()` in `Analyze` right after `LoadDb`.
## How this was tested
- New unit test `TestDatabaseCloseReleasesFileHandle` opens a
file-backed sqlite, runs a query to force a pool connection, then
asserts Close drains the pool and blocks further queries.
- `go test ./server/vulnerabilities/goval_dictionary/...` passes.
- Standalone Go program reproduced the leak mechanism: `sql.Open` +
query + unlink left the FD on the orphaned inode; adding Close released
it.
## Confidence and QA
~90% confident. I did not reproduce end-to-end through Fleet's vuln cron
locally (the analyzer never entered its query loop; likely
`HostIDsByOSVersion` hadn't populated for the Rocky test host).
Reviewer: flag anything that drops your confidence. @xpkoala for QA
after merge: please exercise in a production-like env with enrolled RHEL
hosts and confirm no `(deleted)` FDs after goval refreshes.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`
(`changes/42741-fix-goval-dictionary-fd-leak`).
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (N/A, no new input paths).
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops (N/A, no new network calls).
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes (N/A,
no endpoint changes).
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, automated tests simulate multiple hosts and
test for host isolation (N/A, package-level unit test).
- [ ] QA'd all new/changed functionality manually (pending, post-merge
by @xpkoala).
## Database migrations
- [x] Checked schema for modified tables for auto-updating timestamp
columns (N/A, no schema changes).
- [x] Confirmed timestamp updates are acceptable (N/A, no schema
changes).
- [x] Ensured correct collation is explicitly set for character columns
(N/A, no schema changes).
## New Fleet configuration settings
- [x] Setting(s) is/are explicitly excluded from GitOps (N/A, no new
settings).
## fleetd/orbit/Fleet Desktop
- [x] Verified compatibility with the latest released version of Fleet
(N/A, server-only change).
- [x] If the change applies to only one platform, confirmed
`runtime.GOOS` is used (N/A).
- [x] Verified fleetd runs on macOS, Linux and Windows (N/A, server-only
change).
- [x] Verified auto-update works (N/A, server-only change).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed a file-descriptor leak in vulnerability processing so deleted
SQLite database files are properly closed without requiring a server
restart, improving stability and resource usage.
* **Tests**
* Added a regression test to ensure database handles are released after
close.
* **Documentation**
* Documented the fix for the file-descriptor leak.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43885
Adds a migration and code to capture the value of the fleet managed
admin account if one exists. Changes file added for entire feature
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automatic password rotation for managed local admin accounts on macOS,
triggered after viewing activity.
* Provisioning now captures and persists the managed admin account
identifier (UUID) to support rotation and prevents that account from
being stored as a regular user.
* Hosts will request a best-effort recheck when the managed admin
identifier is not yet available.
* **Chores**
* Database schema updated to store rotation scheduling and pending
credential state.
* **Tests**
* Added tests covering UUID capture, conditional updates, migration, and
ingest behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41592
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed SSO failures when a custom Apple MDM URL is configured: callback
requests are now redirected to the configured MDM URL when needed, and
SAML validation correctly considers the configured MDM/server URLs so
authentication succeeds for custom MDM setups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #43769
# Details
Adds methods to collect data for the `cve` dataset. As with all sets
this is collected at hourly granularity, but unlike the `uptime` set,
the `cve` set uses the "snapshot" strategy so that we record at most one
change (the most recent) per hour.
For this first iteration, we are _recording_ data for all CVEs (i.e.,
which hosts were exposed to which CVEs at a given time), but we are only
_reporting_ a subset of CVEs for the dashboard chart. See [this
comment](https://github.com/fleetdm/fleet/pull/44124#discussion_r3155554405)
for more info.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- [X] Spot-checked the CVEs chosen by the `trackedCVESoftwareMatchers`
and didn't find any outside of the expected
- [X] With [front-end PR](https://github.com/fleetdm/fleet/pull/44261),
generated chart:
<img width="706" height="421" alt="image"
src="https://github.com/user-attachments/assets/539d9877-6573-4406-a159-1d2a711a045f"
/>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Host vulnerability (CVE) chart added to the dashboard; CVE chart data
collection is now active.
* Critical CVE tracking surfaces high-severity vulnerabilities.
* **Improvements**
* CVE chart refreshes every 3 hours (was daily) for more timely
insights.
* Snapshot collection reconciles and closes prior data during empty runs
to keep charts accurate.
* CVE queries may produce zero datapoints when no tracked CVEs exist,
without affecting other metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed filtering in the /api/v1/fleet/labels/:id/hosts endpoint and
tightened validation to reject invalid sort/order keys with HTTP 422
responses.
* Enforced ordering restrictions tied to feature flags (issues and
device-mapping), rejecting unsupported order_key values.
* **Tests**
* Added extensive integration tests for order_key validation,
deterministic sorting across allowed keys, and cursor pagination.
* **Documentation**
* Added a changelog entry noting the hosts-in-label filtering fix.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#42843
This change shows Windows Enrollment Status Page (ESP) during OOBE
enrollment. It does not track/update the status of that page, so the end
user does not actually see any progress on it. Its purpose is to block
the user from proceeding to desktop until all the profiles have been
sent to the device. Software apps are not being tracked/blocked in this
PR.
This is what the final ESP screen looks for this PR before it takes the
user to set up Windows Hello:
<img width="646" height="549" alt="image"
src="https://github.com/user-attachments/assets/748a2710-9388-4d04-93d1-8f2a518965a1"
/>
# Checklist for submitter
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enrollment Status Page (ESP) support for Windows Autopilot: sends
hold/release commands and advances enrollment states during setup.
* Scoped profile installation checks per host and a default ESP timeout
(3 hours).
* **Bug Fixes**
* Clears prior profile delivery state during reenrollment cleanup to
avoid stale delivery state.
* Safer state transitions for "awaiting configuration" with guarded
compare-and-swap updates.
* **Tests**
* New unit and integration tests validating ESP flows and
awaiting-configuration transitions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Konstantin Sykulev <konst@sykulev.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44052
Improve performance by reducing the time for the synchronous API call to
update profiles or switch teams. And spreading out the application of
profiles by processing 2000 hosts every 30 seconds.
1. **Windows profile reconciliation is no longer synchronous to
bulk-set.**
Apple, Android, and Apple-declaration paths still write their pending
state inside the bulk-set transaction. The Windows path commits the
transactional inputs and lets the existing `mdm_windows_profile_manager`
cron pick the work up on its next tick. The visible effect is that
`host_mdm_windows_profiles` is no longer guaranteed to be populated by
the time bulk-set returns; it converges within one cron interval.
2. **The Windows reconciler now processes hosts in bounded batches, with
a persisted cursor.**
Previous behavior was "scan the universe of pending Windows hosts on
every tick." New behavior is a host-window query bounded by batch size
and a `host_uuid` cursor, advanced after the batch commits successfully
and persisted across ticks. A failed tick leaves the cursor untouched so
the same window is retried.
3. **Two replication races are now explicitly handled.**
- Admin-delete vs reconcile: the existence check the reconciler uses to
avoid touching a just-deleted profile reads from the primary, not a
replica.
- Insert lag in the reconciler's own listings: hosts that appear in the
cursor query but are not yet visible in the scoped listings advance the
cursor instead of jamming the loop.
4. **`updates.WindowsConfigProfile` from `BulkSetPendingMDMHostProfiles`
is now always false in production.**
The only consumer ORs it with the transactional signal from
`BatchSetMDMProfiles`, which is the accurate source. The bulk-set call
no longer attempts to compute or return that activity signal itself.
5. **Tests opt in to the old synchronous behavior via a named hook.**
Default test behavior matches production (deferred). Legacy tests whose
assertions require Windows rows immediately after bulk-set call an
explicit enable-hook and rely on `t.Cleanup` to restore.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Windows MDM profile reconciliation batching improvements enable large
team transfers and bulk profile change operations to complete faster,
with profile updates rolling out in the background without blocking host
check-ins or other MDM activity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This reverts commit 5b8253173e.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38785
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Windows setup experience now supports requiring all software
installations: enrollment can be configured to cancel if any required
software fails to install.
* **Tests**
* Added test coverage for platform-specific setup software requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43659
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Preserves install scripts for script-only software installers when
using hash-based references in GitOps, preventing self-service installs
from silently no‑opping.
* **Tests**
* Added an integration regression test to verify batch installer
resolution by hash preserves uploaded install script contents.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43789
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* View and download the default automatic MDM (Apple Setup Assistant)
enrollment profile via a new endpoint.
* Shows a last-updated timestamp when present; returns the in‑app
default with no timestamp if none is stored.
* **Access**
* Access follows existing team and global permission rules; not
available on Free-tier licenses.
* **Tests**
* Added unit and integration tests covering endpoint behavior and access
controls.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Small PR to improve URL validation for Apple Server URL, frontend now
also requires a protocol, and does require a TLD (aka. no localhost).
Backend requires a scheme/protocol, and no empty hostname, we previously
did not have these validations in place, which breaks if you don't use a
scheme for the URL.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Strengthened validation for MDM Apple Server URL and SSO User URL
settings. MDM Apple Server URL now rejects localhost and non-HTTP/HTTPS
schemes, and reports clearer errors for malformed URLs to reduce
misconfiguration.
* **Tests**
* Added test cases covering malformed MDM Apple Server URL inputs to
ensure validation behaves as expected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43943
# Checklist for submitter
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
See
https://github.com/fleetdm/fleet/issues/43943#issuecomment-4329658412
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
## New Fleet configuration settings
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
(see https://github.com/fleetdm/fleet/pull/43877/changes)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled (should be done by
https://github.com/fleetdm/fleet/issues/43947)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a configuration option to preserve host activities during host
re-enrollment, letting admins choose whether activity history is
retained when hosts re-enroll.
* **Chores**
* Updated defaults and database migration state so the new setting is
present in stored and generated configs and in GitOps outputs.
* **Tests**
* Added unit, integration, migration, and GitOps fixtures to validate
behavior, serialization, and upgrade semantics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
For #36087
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added script execution API supporting asynchronous and synchronous
operations with timeout handling.
* Introduced batch script execution capabilities including batch run
creation, status querying, and execution cancellation.
* Added host management API endpoints for locking, unlocking, and wiping
devices.
* Enhanced script management with create, update, delete, list, and
retrieval operations.
* Improved file download responses with proper content headers and
attachment handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#44189
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance**
* Optimized Windows MDM profile removal to skip redundant database
writes for terminal removals.
* **Bug Fixes**
* Ensure terminal remove responses (both verified and failed) delete the
corresponding profile records without affecting concurrent installs.
* **Tests**
* Added coverage for mixed install/remove responses and re-install after
a verified removal.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44118
# Details
On free tier, ignore exceptions and always apply enroll secrets when
present.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a, unreleased
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
@AndreyKizimenko QA'd manually
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed GitOps to correctly apply enrollment secrets and labels on free
tier licenses, even when exception flags are configured.
* **Tests**
* Added tests validating that GitOps properly applies secrets and labels
for free tier customers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43959#44038
Refactored `checkSoftwareConflictsByIdentifier` to a switch statement
with different logic per platform
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Bug Fixes
- Prevented duplicate software installer entries on Linux.
- Improved conflict detection for software installers across iOS, macOS,
Windows, and Linux platforms to prevent incompatible uploads.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- We should never skip the Accessibility screen on macOS. Some end users
cannot use a computer without these features.
- Added some keys that were released after this list was last updated.
- Removed `Region`, since we have customers in more than the US now.
- Removed `IsSupervised`, as devices are [automatically supervised
now](https://support.apple.com/guide/deployment/about-device-supervision-dep1d89f0bff/web#:~:text=The%20following%20devices%20are%20supervised%20automatically%20when%20they%E2%80%99re%20enrolled%20using%20Automated%20Device%20Enrollment).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Chores**
* Refined the default Apple Device Enrollment Program configuration to
streamline the device setup experience by adjusting which setup
assistant steps are presented during enrollment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44198
Adds checks to the refetch type assertions so we both don't panic and
skip writes when we don't have data(which shouldn't really be happening,
but is a perf increase if it does). Also adds a warning if expected
fields are missing on the checkin so we can still monitor if a customer
is reporting missing fields
I cannot figure out a reliable repro for this so testing was limited to
automated tests added + some basic refetch testing(several times as I
tried to figure out how to egt the thing to send it empty)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually - Couldn't actually
repro the bug but tests do verify the fix
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced Apple Mobile Device Management reliability by improving
server response handling. The system now gracefully processes device
information queries even when optional fields are missing or have
unexpected formats, preventing potential service interruptions and
preserving existing device data when updates are incomplete.
* **Tests**
* Added defensive unit tests covering various edge cases in Apple device
information synchronization, including scenarios with missing or
malformed data fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
