Fixes#32096
The gist of the fix is that when syncing devices from DEP we save the
migration deadline to our host_dep_assignments table. The next
enrollment, which we assume should be the migration, looks at
host_dep_assignments, sees that mdm_migration_deadline is non-Null and
mdm_migration_completed is NULL, and uses that as the signal that a
migration is in progress and skips enqueuing setup experience items. It
then marks the migration as complete which sets mdm_migration_completed
= mdm_migration_deadline. Once this is set setup experience will run as
normal unless mdm_migration_completed gets set to NULL and/or
mdm_migration_deadline gets set to a value in the future(which e.g.
would happen if the customer assigned to another MDM server then
assigned to migrate to fleet again)
DB test failure is expected here because it won't like the migration
timestamp but that is a necessary failure because this fix is going to
be backported into 4.73
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
## Database migrations
- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* New Features
* Tracks and stores Apple DEP MDM migration deadlines per device/host.
* Detects “migration in progress” during DEP sync and check-in.
* Automatically marks migration complete and skips Setup Assistant items
while migration is in progress to prevent conflicts.
* Bug Fixes
* Improved DEP compatibility by updating the protocol version and
User-Agent used for Apple’s APIs, reducing the chance of blocked or
rejected requests.
* Migrations
* Adds fields to support migration deadlines and completion status (no
action required).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Magnus Jensen <magnus@fleetdm.com>
Fixes https://github.com/fleetdm/fleet/issues/32819
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
Speculative fix for #30915
For why this is needed, see
https://github.com/fleetdm/fleet/issues/30915#issuecomment-3259641371
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved reliability of Apple device profile installation and removal
by performing coordinated, read-only transactional reads. Reduces race
conditions and intermittent discrepancies during profile syncs, leading
to more consistent outcomes across fleets.
* **Tests**
* Added tests to verify the combined install/remove results remain
consistent with the individual lists, ensuring accurate and stable
behavior under various state changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes: #32067
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
For #32571.
Original PR from the community:
https://github.com/fleetdm/fleet/pull/32573.
Changes on this PR:
- Only setting the checksum algorithm when using GCS as backend (to not
break other S3 backends).
- Changes for carves, bootstrap packages, and software icons which also
use S3.
## Testing
- [X] QA'd all new/changed functionality manually
```sh
FLEET_S3_SOFTWARE_INSTALLERS_BUCKET=some-software-installers-bucket \
FLEET_S3_SOFTWARE_INSTALLERS_ACCESS_KEY_ID=... \
FLEET_S3_SOFTWARE_INSTALLERS_SECRET_ACCESS_KEY=... \
FLEET_S3_SOFTWARE_INSTALLERS_ENDPOINT_URL=https://storage.googleapis.com \
FLEET_S3_SOFTWARE_INSTALLERS_REGION=us \
FLEET_S3_SOFTWARE_INSTALLERS_FORCE_S3_PATH_STYLE=true \
FLEET_S3_CARVES_BUCKET=some-carves-bucket \
FLEET_S3_CARVES_ACCESS_KEY_ID=... \
FLEET_S3_CARVES_SECRET_ACCESS_KEY=... \
FLEET_S3_CARVES_ENDPOINT_URL=https://storage.googleapis.com \
FLEET_S3_CARVES_REGION=us \
FLEET_S3_CARVES_FORCE_S3_PATH_STYLE=true \
./build/fleet serve --dev --dev_license --logging_debug 2>&1 | tee ~/fleet.txt
```
Fixes#29909
- Do not update DB if rules haven't changed
- Cache Yara rules when retrieved by hosts. This should reduce DB
accesses with large number of hosts retrieving large numbers of rules
I manually QA'd using OpenTelemetry (APM would also work) and monitoring
the DB accesses when updating or retrieving yara rules.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance Improvements**
* Faster config saves when many YARA rules are present (incremental
updates, reduced work).
* Lower latency and load when many hosts fetch YARA rules (caching and
smarter retrieval).
* More efficient handling of unchanged, added, modified, and removed
YARA rules.
* **Documentation**
* Changelog entry noting YARA rules performance and fetch improvements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
For #29478, sans GitOps.
---------
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Konstantin Sykulev <konst@sykulev.com>
#30877
We need to send `platform_like` during orbit enrollment for proper setup
experience for Linux
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
## fleetd/orbit/Fleet Desktop
- [X] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [x] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
Fixes#32648
It looks like we don't specify a precision for the jobs table so mysql
does us the un-favor of rounding the timestamp in such a way that if
it's 59 and some change it might get rounded to the next minute after
the roundtrip and truncation to minute. Fixed the test's expectations by
rounding to second before DB insertion
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
For #29426
For #30941
We didn't have an activities subtask for Hydrant so some changes related
to them were unfortunately missed and discovered during engineering
test. This addresses that and also fixes one other API issue noticed
where the GET endpoint was not returning the proper timestamps. No
changes file added because the Hydrant story changes file already covers
this feature
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
Fixes#32295
The issue was identified/fixed using this performance test:
https://gist.github.com/getvictor/b289b7b14981fb7bf77e57c80af117d1
With the fix:
- 100 records: 2.6ms (similar)
- 1,000 records: ~7ms (32x faster)
- 5,000 records: ~10ms (530x faster)
- 10,000 records: 13ms (1,430x faster)
- 20,000 records: ~25ms (2,960x faster)
- 40,000 records: 50ms (6,000x faster)
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Script details now surface the most relevant “latest” status per
script, prioritizing upcoming executions when present.
- Performance Improvements
- Significantly faster loading of host script results and script
details, especially at large scale.
- Improved responsiveness when filtering/sorting script results.
- Documentation
- Added changelog entry describing the optimization to script results
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
# Details
Applying the patches from
https://github.com/fleetdm/fleet/pull/32516#discussion_r2316985807 and
https://github.com/fleetdm/fleet/pull/32563
onto `main`.
This fixes:
* Batches where all remaining hosts are deleted will be correctly marked
as "finished"
* Batches scheduled for the future, and then canceled, will have al
hosts marked as "canceled" rather than pending
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- [X] Started a new batch script run with one host, deleted that host,
and triggered the batch_activity_completion_checker schedule, and
verified that the batch was moved to "finished".
- [X] Scheduled script
- Created a new batch script run scheduled for a future date
- Canceled that batch run
- Clicked on the batch run in the "finished" tab of the batch scripts
list
- Verified that the number of canceled hosts = the number of targeted
hosts for that batch, and all other numbers were 0.
- Verified that clicking the canceled hosts number navigated to the
correct list of canceled hosts
- [X] "Run now" script
- Created a new batch script run with "run now"
- Waited for at least one host to run.
- Canceled that batch
- Clicked on the batch run in the "finished" tab of the batch scripts
list
- Verified that the number of canceled hosts = the number that were
still pending when I canceled the script, and that the # of pending
hosts was 0
- Verified that clicking the canceled hosts number navigated to the
correct list of canceled hosts
- [X] Multiple batches with the same hosts don't bleed into each other
- Created another batch script with the same hosts, scheduled for the
future
- Verified that the "pending" host list is correct
There are still some TODOs particularly within Gitops test code which
will be worked on in a followup PR
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
## Database migrations
- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
---------
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Magnus Jensen <magnus@fleetdm.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
For #32040.
---
Backend changes to unblock the development of the orbit and frontend
changes.
New GET and PUT APIs for setting/getting software for Linux Setup
Experience:
```
curl -k -X GET -H "Authorization: Bearer $TEST_TOKEN" https://localhost:8080/api/latest/fleet/setup_experience/linux/software?team_id=8&per_page=3000
curl -k -X PUT -H "Authorization: Bearer $TEST_TOKEN" https://localhost:8080/api/latest/fleet/setup_experience/linux/software -d '{"team_id":8,"software_title_ids":[3000, 3001, 3007]}'
```
New setup_experience/init API called by orbit to trigger the Linux setup
experience on the device:
```
curl -v -k -X POST -H "Content-Type: application/json" "https://localhost:8080/api/fleet/orbit/setup_experience/init" -d '{"orbit_node_key": "ynYEtFsvv9xZ7rX619UE8of1I28H+GCj"}'
```
Get status API to call on "My device":
```
curl -v -k -X POST "https://localhost:8080/api/latest/fleet/device/7d940b6e-130a-493b-b58a-2b6e9f9f8bfc/setup_experience/status"
```
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [X] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Added Linux support for Setup Experience alongside macOS.
- Introduced platform-specific admin APIs to configure and retrieve
Setup Experience software (macOS/Linux).
- Added device API to report Setup Experience status and an Orbit API to
initialize Setup Experience on non-macOS devices.
- Setup Experience now gates policy queries on Linux until setup is
complete.
- New activity log entry when Setup Experience software is edited
(includes platform and team).
- Documentation
- Updated audit logs reference to include the new “edited setup
experience software” event.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#32160
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Dedicated automations for “No team,” including failing policy webhooks
and Jira/Zendesk ticketing, separate from global settings.
- Policies without a team now use the default team automation
configuration instead of global.
- Chores
- Database migration splits global vs “No team” policy IDs and copies
applicable automation settings to the default team.
- To enable this behavior, set FLEET_PARTNERSHIPS_ENABLE_PRIMO=1 before
running migrations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
for #1817
# Details
This PR gives Fleet servers the ability to connect to RDS MySQL and
Elasticache Redis via AWS [Identity and Access Management
(IAM)](https://aws.amazon.com/iam/). It is based almost entirely on the
work of @titanous, branched from his [original pull
request](https://github.com/fleetdm/fleet/pull/31075). The main
differences between his branch and this are:
1. Removal of auto-detection of AWS region (and cache name for
Elasticache) in favor of specifying these values in configuration. The
auto-detection is admittedly handy but parsing AWS host URLs is not
considered a best practice.
2. Relying on the existence of these new configs to determine whether or
not to connect via IAM. This sidesteps a thorny issue of whether to try
an IAM-based Elasticache connection when a password is not supplied,
since this is technically a valid setup.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually - besides using
@titanous's excellent test tool, I verified the following end-to-end:
- [X] regular (non RDS) MySQL connection
- [X] RDS MySQL connection using username/password
- [X] RDS MySQL connection using IAM (no role)
- [X] RDS MySQL connection using IAM (assuming role)
- [X] regular (non Elasticache) Redis connection
- [X] Elasticache Redis connection using username/password
- [X] Elasticache Redis connection using NO password (without IAM)
- [X] Elasticache Redis connection using IAM (no role)
- [X] Elasticache Redis connection using IAM (assuming role)
---------
Co-authored-by: Jonathan Rudenberg <jonathan@titanous.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Fixes: #30403
Keys for deletedTitles map were generated differently, causing the same
software title to be marked removed even when a new version of the same
title was inserted.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
Fixes#31580
Fixes issues
- When updating a script to exactly match the content of another script,
we fail
- When updating one script which happens to match content of another
script, both get updated and not just the one being edited
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Resolved error when updating a script to exactly match another
script’s contents.
* Improved handling of script content updates: identical contents are
deduplicated and unused versions are cleaned up.
* Scheduled/pending runs are canceled on content updates with clearer
cancellation messaging.
* **Documentation**
* Added changelog entry describing the fix.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#32060
This PR adds:
- new default_team_config_json table
- caching of config from that table, including deep copy methods -- all
of this is not absolutely needed for this change since we are only using
`webhook_settings.failing_policies_webhook` here but added for
completeness/future
- teams/0 API updates
- GitOps updates
- generate gitops updates
Future PRs will add:
- ticket automation
- primo mode migration
- frontend changes
- documentation
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked table schema to confirm autoupdate
## New Fleet configuration settings
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Configure failing-policy webhooks for “No team” via GitOps
(no-team.yml) and API, including enable/disable, destination URL, policy
IDs, and batch size; settings clear when omitted.
- GitOps and CLI now read/apply the real “No team” settings with dry-run
support.
- Policy automation evaluates hosts without a team and triggers “No
team” webhooks when applicable.
- GET/PATCH team 0 returns/accepts a minimal, webhook-focused config.
- Chores
- Added persistence and caching for the default “No team” configuration.
- Introduced a database table to store the default configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Fixes#31432
- Added campaign target cleanup: Deletes targets from campaigns
completed >24h ago. Uses 10% or 50k min per run, processes in 10k
batches. Added DB index, integrated into hourly cron, includes tests.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Automatic cleanup of live query campaign targets 24 hours after
campaign completion to reduce clutter and storage usage.
- Chores
- Added a database index to speed up live query target operations for
improved performance at scale.
- Enhanced scheduled maintenance to log cleanup counts and execution
time for better observability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
for #31536
# Details
This PR adds a new API as specced in [the API
PR](9bf150580b/docs/REST%20API/rest-api.md (list-hosts-targeted-in-batch-script))
for scheduled scripts.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
ran a batch script on 100 hosts and ran the API in Postman for each
status, then canceled the batch and ran the API to check the canceled
status.
---------
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Fixes: #31581
Undo changes to defFilter, they weren't necessary
Add unit test for team_id=0
# Checklist for submitter
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
for #32231
# Details
This PR adjusts the queries for listing batch scripts slightly to count
_every_ row in `batch_activities` matching the filters, regardless of
whether any `batch_activity_host_results` rows exist for it. This
handles the edge case of a batch script where all the hosts have been
deleted.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [ ] Added/updated automated tests
I didn't add tests for this because these tests have already changed
quite a bit in https://github.com/fleetdm/fleet/pull/32174. I can add
tests in there when this merges.
- [X] QA'd all new/changed functionality manually
* Select a host in Manage Hosts, click Run Script, select a script and
do Run Now
* Delete that host
* Go to the batch scripts list (Controls -> Scripts -> Batch Progress)
* Verify that the batch script is still listed.
We don't have clear expectations for what numbers should be displayed
for the progress of a batch like this, but this PR at least ensures the
batch doesn't disappear.
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
Closes#31581
Note:
- When no team id is provided it lists all installers, but they don't
have software_package fielded. I don't know if this is the intended
behavior or not.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
fixes: #30888
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked table schema to confirm autoupdate
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* New Features
* Added support for an IdP full name variable in Apple configuration
profiles, enabling substitution of a user’s full name.
* Profiles automatically re-deploy when a user’s IdP full name changes,
is removed, or the user is assigned to a host.
* Bug Fixes
* Added clearer failure handling: profiles that require an IdP full name
now report a delivery failure with a specific message if the value is
missing.
* Tests
* Expanded test coverage for full name handling and failure scenarios.
* Chores
* Seeded the new variable in the database.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
For #31167
Adds API Changes for showing user-scoped profile scoeps and managed
local user accounts
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
Closes https://github.com/fleetdm/fleet/issues/32257
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
For #28133.
When generating the payload for the BitLocker CSP used for setting the
TPM PIN policy, make sure the UUID used in the Atomic enclousure matches
the UUID used for the whole command.
Fixes#31173
Reproduced and fixed in loadtest environment. Uncovered another source
of deadlocks, filed as a separate:
https://github.com/fleetdm/fleet/issues/32201
- Also, still seeing some deadlocks (a lot fewer) in DB, and they are
hidden from the API results by retries. They may still be happening
because locks happen row by row and not all at once. A potential fix
would be to lock the whole policy_membership table.
Additional frontend fix, which is needed to prevent potential timeouts:
https://github.com/fleetdm/fleet/pull/32212
Backend + frontend fix should be a sufficient fix for this issue
(ignoring the issue with the long software transaction).
Also, this PR contains some refactoring to split out the 1-host use
case.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* Bug Fixes
* Resolved rare deadlocks during concurrent policy updates and bulk
automations.
* Correctly clears stale MDM data and actions on host re-enrollment and
platform changes.
* Performance Improvements
* Optimized policy issue recalculation with per-host updates to reduce
contention.
* Improved concurrency handling for bulk policy updates to avoid lock
contention.
* Reliability
* More robust host enrollment: updates seen time, display name, and
label membership consistently.
* Ensures accurate policy-issue counts after membership changes and
re-enrollment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#31977
Tested with:
224 scripts, 102 Apple configuration profiles, 105 Apple declarations,
37 Windows configuration profiles. Scattered in 7 teams.
No variables on profiles/scripts:
- scan scripts: 22.079416ms
- scan apple profiles: 1.865083ms
- scan apple declarations: 683.75µs
- scan windows profiles: 531.167µs
A variable "X" on with profiles/scripts that have variable "Y":
- scan scripts: 26.305459ms
- scan apple profiles: 1.841708ms
- scan apple declarations: 756.917µs
- scan windows profiles: 571.875µs
---
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
- [X] Alerted the release DRI if additional load testing is needed
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Prevents deleting a secret that’s still in use. The system now checks
scripts, Apple/Windows MDM profiles, and Apple declarations across teams
before allowing deletion.
- The delete dialog now shows a clear message indicating which item (and
team) is using the secret, replacing the previous generic error.
- Deletion succeeds once all references are removed, ensuring safer
secret management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
# Overview
This PR implements the S3 upload manager under-the-hood of our
`datastore/s3` client's `Put()` method.
# Description
As surfaced by #31667, the current S3 implementation utilizes the `PUT`
operation which means services, such as MinIO, take issue with attempted
uploads that are [too
large](https://github.com/minio/minio/blob/master/cmd/streaming-signature-v4.go#L260).
The `PUT` operation can also present challenges in memory-constrained
environments as the entire upload target is read into memory before it's
shipped.
# Notes
- See the `TODO` comment section, there's more cool stuff we can and
should do with this in the future!
# Standard Pull Request Details
## Testing
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
---------
Signed-off-by: Illbjorn <am@hades.so>
Fixes#29894 and probably #31980.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
fixes: #29222
This is a feature branch that was completed last week, but did not get
merged in time.
All pr's going in was approved, and reviewed.
I will after this is merged, do a cherry pick onto the RC 4.73 branch,
and initiate the FR merge process.
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
fixes#31932
The problem here was that `hostInstalledSoftware` returns a slice of
pointers (`[]*hostSoftware`), so when iterating through and assigning
`LastOpenedAt` the original records were getting modified. This code
duplicates the records being put into `bySoftwareTitleID` so that the
records being stored in `bySoftwareID` are the original records.
As a side benefit to this I modified the logic to store the most recent
`LastOpenedAt` for the software title. I think we may be doing something
similar to this on the front end to show the "last used" column when we
have multiple version of a software installed. But this can potentially
be fetched from the API now.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
#31532
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
For #31055.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
for #31282
# Details
This PR adds the ability to filter hosts by the "incompatible with batch
script" status. These hosts were previously included in the "Error"
state for a batch script when viewing the script summary.
The current script summary modal doesn't include a row for incompatible
(this modal will be replaced in the next iteration of the batch script
scheduling feature). To see the filter at work, you can either use the
API directly, or:
1. View the summary modal for a batch script by clicking on its activity
item in the global feed
2. Click on the number in any row (e.g. "Error" or "Pending")
3. Change the dropdown beneath the team selector to "Incompatible"
<img width="472" height="339" alt="image"
src="https://github.com/user-attachments/assets/04c6bc05-fe88-4be3-91ca-8b7162e1c6f3"
/>
Also renamed `cancelled` to `canceled` in a couple places to make the
spelling consistent.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- **linux vulns API changes (#31490)**
- **31214 linux vulns optimization (#31722)**
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
Fixes#31781
Only test-related changes.
I found this useful for agentic AI workflows. For example, you have an
AI agent debugging/rerunning a test. Meanwhile, you can spin up another
`mysql_test` instance in another workarea and work there in parallel.
