fleet/server/datastore/mysql
Jordan Montgomery 572536d466
Skip setup experience during AxM based migrations (#32822)
Fixes #32096

The gist of the fix is that when syncing devices from DEP we save the
migration deadline to our host_dep_assignments table. The next
enrollment, which we assume should be the migration, looks at
host_dep_assignments, sees that mdm_migration_deadline is non-Null and
mdm_migration_completed is NULL, and uses that as the signal that a
migration is in progress and skips enqueuing setup experience items. It
then marks the migration as complete which sets mdm_migration_completed
= mdm_migration_deadline. Once this is set setup experience will run as
normal unless mdm_migration_completed gets set to NULL and/or
mdm_migration_deadline gets set to a value in the future(which e.g.
would happen if the customer assigned to another MDM server then
assigned to migrate to fleet again)

DB test failure is expected here because it won't like the migration
timestamp but that is a necessary failure because this fix is going to
be backported into 4.73

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed

## Database migrations

- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* New Features
  * Tracks and stores Apple DEP MDM migration deadlines per device/host.
  * Detects “migration in progress” during DEP sync and check-in.
* Automatically marks migration complete and skips Setup Assistant items
while migration is in progress to prevent conflicts.

* Bug Fixes
* Improved DEP compatibility by updating the protocol version and
User-Agent used for Apple’s APIs, reducing the chance of blocked or
rejected requests.

* Migrations
* Adds fields to support migration deadlines and completion status (no
action required).

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Magnus Jensen <magnus@fleetdm.com>
2025-09-11 09:40:40 -04:00
..
common_mysql Add Read-only Transaction to fetch profiles to install and remove all at once (#32737) 2025-09-10 09:29:04 -04:00
migrations Skip setup experience during AxM based migrations (#32822) 2025-09-11 09:40:40 -04:00
testdata Followup for #32284, packages_only works for team_id=0 (#32352) 2025-08-27 16:40:48 -04:00
activities.go API endpoints for Linux setup experience (#32493) 2025-09-04 12:58:47 -03:00
activities_test.go Updated SQL modes in tests to match production. (#31445) 2025-08-03 08:18:13 +02:00
aggregated_stats.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
aggregated_stats_test.go Performance stats for live queries (#15440) 2023-12-13 14:46:59 -06:00
android.go Add IdP email to host vitals (#32807) 2025-09-10 13:54:42 -04:00
android_device_test.go Merge Android datastore into main Fleet datastore (#32233) 2025-08-25 11:41:28 -04:00
android_enterprise_test.go Merge Android datastore into main Fleet datastore (#32233) 2025-08-25 11:41:28 -04:00
android_enterprises.go Merge Android datastore into main Fleet datastore (#32233) 2025-08-25 11:41:28 -04:00
android_hosts.go Merge Android datastore into main Fleet datastore (#32233) 2025-08-25 11:41:28 -04:00
android_mysql.go Merge Android datastore into main Fleet datastore (#32233) 2025-08-25 11:41:28 -04:00
android_test.go Add IdP email to host vitals (#32807) 2025-09-10 13:54:42 -04:00
app_configs.go Improved performance when modifying config with a large number of yara rules (#32696) 2025-09-08 10:24:22 -05:00
app_configs_test.go Improved performance when modifying config with a large number of yara rules (#32696) 2025-09-08 10:24:22 -05:00
apple_mdm.go Skip setup experience during AxM based migrations (#32822) 2025-09-11 09:40:40 -04:00
apple_mdm_ddm_test.go Updated SQL modes in tests to match production. (#31445) 2025-08-03 08:18:13 +02:00
apple_mdm_test.go Skip setup experience during AxM based migrations (#32822) 2025-09-11 09:40:40 -04:00
ca_config_assets.go Add custom SCEP configs (#27045) 2025-03-14 12:16:51 -05:00
ca_config_assets_test.go Add custom SCEP configs (#27045) 2025-03-14 12:16:51 -05:00
calendar_events.go Enable staticcheck Go linter. (#23487) 2024-11-05 11:16:24 -06:00
calendar_events_test.go Handle null HostID on calendar webhook endpoint (#30130) 2025-06-23 13:10:10 -04:00
campaigns.go Add CleanupCompletedCampaignTargets to cleanup old campaign targets. (#32385) 2025-08-28 11:04:05 -05:00
campaigns_test.go Add CleanupCompletedCampaignTargets to cleanup old campaign targets. (#32385) 2025-08-28 11:04:05 -05:00
carves.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
carves_test.go Improve performance of the Go test suite (#2060) 2021-09-20 14:09:38 -04:00
certificate_authorities.go Hydrant CA followup (#32606) 2025-09-05 06:37:20 -04:00
certificate_authorities_test.go Hydrant CA Feature Branch (#31807) 2025-09-04 12:39:41 -04:00
challenges.go Update documentation for custom SCEP proxy (#29971) 2025-06-16 12:00:27 -05:00
conditional_access_microsoft.go Microsoft Compliance Partner backend changes (#29540) 2025-06-11 14:22:46 -03:00
conditional_access_microsoft_test.go Add app_sso_platform table to orbit and use table in Entra ID query ingestion (#30140) 2025-06-20 17:01:38 -03:00
config.go Refactoring common mysql (#26367) 2025-02-18 15:28:54 -06:00
cron_stats.go Monitor and alert on errors in cron jobs (#24347) 2024-12-19 15:55:29 -06:00
cron_stats_test.go Monitor and alert on errors in cron jobs (#24347) 2024-12-19 15:55:29 -06:00
delete.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
delete_test.go Updated factory method for creating queries in tests 2023-07-07 09:05:51 -04:00
disk_encryption.go Refactoring suggested in #31634 (#31839) 2025-08-13 10:24:32 -04:00
disk_encryption_test.go Added new global activity when disk encryption key is escrowed (#31634) 2025-08-08 12:14:48 -04:00
email_changes.go Add mechanism to force read from primary DB, use it for puppet matching (#12396) 2023-06-19 13:55:15 -04:00
email_changes_test.go Improve performance of the Go test suite (#2060) 2021-09-20 14:09:38 -04:00
errors.go Refactoring common mysql (#26367) 2025-02-18 15:28:54 -06:00
fulltext.go Update to Go 1.24.1 (#27506) 2025-03-31 11:14:09 -05:00
fulltext_test.go Using '@' in target search causes 422 error response (#5148) 2022-04-19 09:28:49 -04:00
host_certificates.go Fixed issue ingesting certs with long country codes. (#31443) 2025-07-31 23:06:36 +02:00
host_certificates_test.go Fixed issue ingesting certs with long country codes. (#31443) 2025-07-31 23:06:36 +02:00
host_identity_scep.go Host identity SCEP rate limit. (#31038) 2025-07-18 10:04:14 -03:00
host_identity_scep_test.go Fleet server verifies HTTP signature (#30825) 2025-07-16 20:08:27 +02:00
hosts.go Skip setup experience during AxM based migrations (#32822) 2025-09-11 09:40:40 -04:00
hosts_test.go Skip setup experience during AxM based migrations (#32822) 2025-09-11 09:40:40 -04:00
invites.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
invites_test.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
jobs.go Cancel batch execution API (#31757) 2025-08-11 15:17:57 -04:00
jobs_test.go Allow worker to filter queue by job type (#31556) 2025-08-06 17:22:48 -05:00
labels.go Labels optimization recommended in code review. (#31856) 2025-08-13 17:42:33 +02:00
labels_test.go Add missing platform_like during orbit enrollment (#32671) 2025-09-05 16:05:19 -03:00
linux_mdm.go Linux OS settings + disk encryption host filter fixes (#24200) 2024-11-26 19:26:22 -06:00
linux_mdm_test.go Added new global activity when disk encryption key is escrowed (#31634) 2025-08-08 12:14:48 -04:00
locks.go Optimize software_titles query to use indexes (#25722) 2025-01-23 15:48:21 -03:00
locks_test.go Improvements for select next Apple MDM command query. (#24128) 2024-12-05 12:02:48 -06:00
maintained_apps.go Support for fleet maintained apps in gitops (#28751) 2025-05-07 18:16:08 -05:00
maintained_apps_test.go Persist download URL when adding FMAs via non-GitOps API, fix software versions on GitOps YAML generation (#30331) 2025-06-26 14:29:23 -05:00
mdm.go Add IdP email to host vitals (#32807) 2025-09-10 13:54:42 -04:00
mdm_idp_accounts_test.go Add IdP email to host vitals (#32807) 2025-09-10 13:54:42 -04:00
mdm_test.go Added support of $FLEET_VAR_HOST_UUID in Windows MDM configuration profiles (#31695) 2025-08-10 12:24:38 +02:00
microsoft_mdm.go Fixed issue due to UUID mistmatch on BitLocker CSP cmd. (#32210) 2025-08-22 14:32:23 -04:00
microsoft_mdm_test.go Added support of $FLEET_VAR_HOST_UUID in Windows MDM configuration profiles (#31695) 2025-08-10 12:24:38 +02:00
migrations_test.go Refactoring common mysql (#26367) 2025-02-18 15:28:54 -06:00
mysql.go Add Read-only Transaction to fetch profiles to install and remove all at once (#32737) 2025-09-10 09:29:04 -04:00
mysql_test.go Refactoring common mysql (#26367) 2025-02-18 15:28:54 -06:00
nanomdm_storage.go Bugfix: Clear the unified queue when turning off and re-enrolling in MDM (#26816) 2025-03-05 08:59:07 -05:00
nanomdm_storage_test.go 19016 ingest certs on start (#19360) 2024-05-30 18:18:42 -03:00
operating_system_vulnerabilities.go filter out kernels with 0 hosts (#32292) 2025-08-25 20:55:57 -04:00
operating_system_vulnerabilities_test.go filter out kernels with 0 hosts (#32292) 2025-08-25 20:55:57 -04:00
operating_systems.go Android scaffold (#26274) 2025-02-13 14:32:19 -06:00
operating_systems_test.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
packs.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
packs_test.go Enable staticcheck Go linter. (#23487) 2024-11-05 11:16:24 -06:00
password_reset.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
password_reset_test.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
policies.go Prevent deadlocks by adding FOR UPDATE locks (#32173) 2025-08-22 12:36:03 -05:00
policies_test.go Add missing platform_like during orbit enrollment (#32671) 2025-09-05 16:05:19 -03:00
queries.go Refactor ApplyQueries to improve performance (#32394) 2025-09-03 12:54:02 -04:00
queries_test.go Fixing PR review comments (#31550) 2025-08-03 16:41:38 +02:00
query_results.go Added server_settings.query_report_cap (#19692) 2024-06-14 12:24:01 -03:00
query_results_test.go 30311: Fix race condition in test (#30903) 2025-07-17 10:20:49 -04:00
scep.go Add SCEP endpoint for host identity. (#30589) 2025-07-11 11:44:07 -03:00
scep_test.go Updated SQL modes in tests to match production. (#31445) 2025-08-03 08:18:13 +02:00
scheduled_queries.go Update to Go 1.24.1 (#27506) 2025-03-31 11:14:09 -05:00
scheduled_queries_test.go Performance stats for live queries (#15440) 2023-12-13 14:46:59 -06:00
schema.sql Skip setup experience during AxM based migrations (#32822) 2025-09-11 09:40:40 -04:00
scim.go Add full name IdP Fleet variable to Apple configuration profiles (#32246) 2025-08-26 17:55:58 +02:00
scim_test.go Add full name IdP Fleet variable to Apple configuration profiles (#32246) 2025-08-26 17:55:58 +02:00
scripts.go Optimized GetHostScriptExecutionResults MySQL query for for large numbers of script results. (#32595) 2025-09-04 15:48:18 -05:00
scripts_test.go Fix testBatchScriptSchedule() flakiness in server/datastore/mysql/scripts_test.go (#32650) 2025-09-05 11:53:00 -04:00
secret_variables.go Prevent IT admins from deleting a secret variable in use (#32161) 2025-08-22 11:22:37 -03:00
secret_variables_test.go Prevent IT admins from deleting a secret variable in use (#32161) 2025-08-22 11:22:37 -03:00
sessions.go Include expiration info, drop "personalized", in MFA email (#24630) 2024-12-10 18:58:40 -06:00
sessions_test.go Include expiration info, drop "personalized", in MFA email (#24630) 2024-12-10 18:58:40 -06:00
setup_experience.go API endpoints for Linux setup experience (#32493) 2025-09-04 12:58:47 -03:00
setup_experience_test.go API endpoints for Linux setup experience (#32493) 2025-09-04 12:58:47 -03:00
software.go software library page fetches vpp icons properly (#32828) 2025-09-10 18:24:58 -05:00
software_installers.go Add custom software icons (#32652) 2025-09-05 17:31:03 -05:00
software_installers_test.go API endpoints for Linux setup experience (#32493) 2025-09-04 12:58:47 -03:00
software_test.go software library page fetches vpp icons properly (#32828) 2025-09-10 18:24:58 -05:00
software_title_icons.go Add custom software icons (#32652) 2025-09-05 17:31:03 -05:00
software_title_icons_test.go Add custom software icons (#32652) 2025-09-05 17:31:03 -05:00
software_titles.go Add custom software icons (#32652) 2025-09-05 17:31:03 -05:00
software_titles_test.go Followup for #32284, packages_only works for team_id=0 (#32352) 2025-08-27 16:40:48 -04:00
statistics.go Add statistic to measure ABM pending hosts (#28226) 2025-04-15 11:30:07 -04:00
statistics_test.go Add statistic to measure ABM pending hosts (#28226) 2025-04-15 11:30:07 -04:00
targets.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
targets_test.go Add missing platform_like during orbit enrollment (#32671) 2025-09-05 16:05:19 -03:00
teams.go Remove scripts/software from TeamConfig copy. (#32708) 2025-09-08 10:23:43 -05:00
teams_test.go Allow configuring webhook policy automations for "No team" (#32129) 2025-08-28 16:38:27 -05:00
testing_utils.go Updated SQL modes in tests to match production. (#31445) 2025-08-03 08:18:13 +02:00
unicode_test.go Bugfix: only count hosts that user has permission to see in count field of label endpoints (#18859) 2024-05-21 12:02:08 -04:00
users.go Prevent user invite race condition (#29559) 2025-05-29 15:26:02 -04:00
users_test.go Prevent user invite race condition (#29559) 2025-05-29 15:26:02 -04:00
vpp.go Add custom software icons (#32652) 2025-09-05 17:31:03 -05:00
vpp_test.go Fleet UI: VPP Token All teams option bug fix (#31587) 2025-08-07 09:00:51 -04:00
vulnerabilities.go Fix insufficient deduplication on vulnerabilities count query (#31021) 2025-07-17 17:40:21 -05:00
vulnerabilities_test.go Fix insufficient deduplication on vulnerabilities count query (#31021) 2025-07-17 17:40:21 -05:00
windows_updates.go Add mechanism to force read from primary DB, use it for puppet matching (#12396) 2023-06-19 13:55:15 -04:00
windows_updates_test.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
wstep.go Add SCEP endpoint for host identity. (#30589) 2025-07-11 11:44:07 -03:00
wstep_test.go Move nanomdm dependency in monorepo (#16015) 2024-01-11 23:28:48 -03:00