Elgato_dark/fleet
1
0
Fork 0
mirror of https://github.com/fleetdm/fleet synced 2026-05-22 00:18:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 139
fleet/server/service
History
Scott Gress 26e4395926
Allow GitOps to clear global settings more easily using overwrite option (#29215) 
for #28118 

# Checklist for submitter

- [X] Manual QA for all new/changed functionality

## Details

This PR adds an `overwrite` option to the "modify app config" API which,
if set, causes the code to replace certain keys in the existing config
with keys from the incoming config, without attempting any merge. This
is then used by GitOps to allow it to easily clear settings that were
otherwise being merged together or ignored entirely due to the PATCH
semantics expected for the `fleetctl apply` use case.

The new setting is utilized in this first pass for the following
settings:

* `sso_settings`
* `smtp_settings`
* `features`
* `mdm.end_user_authentication`

It could be expanded to several more keys that we currently handle
piecemeal in the GitOps code by attempting to send empty values to the
server (with varying success).

Targeting `mdm.end_user_authentication` vs. all of `mdm` is based on
[this bug](https://github.com/fleetdm/fleet/issues/26175) being opened.
The concern with doing all of `mdm` would be that anyone who had e.g.
VPP set up in their app and hadn't set it up in GitOps would have it
wiped out. If we're comfortable with that risk I can update that here
and update the warning accordingly.

### More detail 

**The way this code works _without_ Overwrite mode on**

1. We unmarshall the incoming JSON from GitOps into a fresh AppConfig
struct `newAppConfig`. Anything keys not present in the incoming JSON
will result in default values being set in `newAppConfig`
2. We unmarshall the incoming JSON from GitOps into the current
`appConfig`. This uses an internal merge algorithm where keys not
present in the JSON will generally leave the matching keys in
`appConfig` untouched. We've been dealing with this by having GitOps
find missing keys and explicitly set them to non-nil empty states. When
arrays are encountered, they are _merged_, not replaced, which is
problematic for the `features.additional_queries` use case and probably
others.
3. We piecemeal replace certain data in `appConfig` with data from
`newAppConfig`, and save it to the db.

**The way this works _with_ Overwrite mode on**

Between steps 1 and 2 above, we _copy_ certain keys from `newAppConfig`
to `appConfig`. If the incoming JSON didn't have a key, the effect will
be that `appConfig` now has default values for that key. For nested
arrays like `features.additionalQueries`, the value in `appConfig` will
be precisely what the user put in GitOps.

## Testing

I tested adding/removing these settings with GitOps manually via
`fleetctl gitops`. On the main branch I could reproduce the issue where
omitting out these keys in my YAML did not lead to the settings being
reset on my instance. With the Features settings, the issue was more
granular, with inconsistent behavior when trying to remove individual
nested settings. On this branch, the settings are cleared as expected at
all levels of granularity.

I also added some new automated tests to verify the expected behavior
for these keys. All existing tests pass.

If accepted this PR would supercede
https://github.com/fleetdm/fleet/pull/29180 which approaches the issue
from the GitOps side for sso, smtp and mdm. Adapting that approach for
`features` would require custom logic to declare nested properties as
"cleared".
2025-05-19 11:18:28 -05:00
..
async Update to Go 1.24.1 (#27506) 2025-03-31 11:14:09 -05:00
calendar Update Google Calendar event bodies and relevant previews in the Fleet UI (#28715) 2025-05-01 13:12:53 -07:00
contract Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
externalsvc remove unused logic specific to DEP+Okta+ROP (#11261) 2023-04-21 14:57:52 -03:00
integrationtest Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
middleware Update error message for VerifyMDMWindowsConfigured (#28646) 2025-04-29 14:11:31 -05:00
mock Update nanomdm dependency with latest bug fixes and improvements. (#23906) 2024-11-20 11:47:11 -06:00
osquery_utils Organize contributor docs and establish ADR process and template (#29101) 2025-05-17 15:03:52 -05:00
redis_key_value Make software batch endpoint asynchronous (#22258) 2024-09-20 11:55:47 -03:00
redis_lock Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
redis_policy_set Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
schedule Clear cron schedule errors before each run (#26775) 2025-03-03 16:41:48 -06:00
testdata Only allow distribution packages for bootstrap package (#28787) 2025-05-06 11:02:13 -05:00
activities.go Cancel upcoming activities: create past canceled activities (#27956) 2025-04-09 16:08:51 -04:00
activities_test.go Cancel upcoming activities: create past canceled activities (#27956) 2025-04-09 16:08:51 -04:00
appconfig.go Allow GitOps to clear global settings more easily using overwrite option (#29215) 2025-05-19 11:18:28 -05:00
appconfig_test.go IdP: resend profiles when IdP values change (#28436) 2025-04-29 14:35:37 -04:00
apple_mdm.go Fixing issues with Apple DDM profile status (#29059) 2025-05-15 13:05:25 -05:00
apple_mdm_ddm_test.go Fixing issues with Apple DDM profile status (#29059) 2025-05-15 13:05:25 -05:00
apple_mdm_test.go Add timestamp to host_mdm_apple_profiles for variable update times (#29074) 2025-05-13 17:22:27 -04:00
base_client.go Added more logging for troubleshooting of software package installation (#27291) 2025-03-20 11:09:57 -03:00
base_client_errors.go fleetctl: Handle "password reset required" errors (#27132) 2025-03-17 09:44:59 -07:00
base_client_test.go prevent baseClient from trying to decode 204 responses (#16060) 2024-01-11 17:55:35 -03:00
build_tags_test.go Return 405 when receiving POST request on the route path (#16750) 2024-02-14 12:40:43 -03:00
calendar.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
campaigns.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
campaigns_test.go Improved label(s) validation when running queries (#23834) 2024-11-21 16:13:30 -06:00
carves.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
carves_test.go Update carve request block id mismatch error code from 500 –> 400 (#17974) 2024-04-04 09:38:10 -07:00
client.go Allow GitOps to clear global settings more easily using overwrite option (#29215) 2025-05-19 11:18:28 -05:00
client_appconfig.go fleetctl gitops --dry-run now errors on duplicate (or conflicting) global/team enroll secrets. (#19344) 2024-05-31 07:01:13 -05:00
client_carves.go
client_debug.go Merge conflicts 2024-04-16 10:20:59 -05:00
client_hosts.go 17257 validate bulk ops request filters (#17514) 2024-03-11 10:02:51 -06:00
client_labels.go Improved label(s) validation when running queries (#23834) 2024-11-21 16:13:30 -06:00
client_live_query.go fleetctl, API, copy updates around host identifiers (#20220) 2024-07-09 10:25:01 -07:00
client_live_query_test.go fleetctl, API, copy updates around host identifiers (#20220) 2024-07-09 10:25:01 -07:00
client_mdm.go Clear bootstrap package and enrollment profile with GitOps (#26095) 2025-02-07 14:35:51 -06:00
client_mdm_test.go fix fleetctl to set different CmdUID for Windows (#14895) 2023-11-02 15:06:37 -03:00
client_packs.go Add fleetctl upgrade-packs command to migrate 2017 packs to queries (#13078) 2023-08-08 08:21:57 -04:00
client_policies.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
client_profiles.go Add "generate-gitops" command (#28555) 2025-05-06 15:25:44 -05:00
client_queries.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
client_scripts.go Add "generate-gitops" command (#28555) 2025-05-06 15:25:44 -05:00
client_secret_variables.go Adding secret support to scripts via gitops. (#24649) 2024-12-11 15:05:48 -06:00
client_sessions.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
client_setup.go
client_software.go Add "generate-gitops" command (#28555) 2025-05-06 15:25:44 -05:00
client_targets.go Add display_name to more results (#8204) 2022-10-14 11:14:18 -03:00
client_teams.go Add VPP install automation in GitOps (#25400) 2025-01-14 12:52:39 -06:00
client_test.go Error when deleting non managed GitOps labels #28440 (#29067) 2025-05-13 20:16:16 -04:00
client_trigger.go Add fleetctl trigger command (#8877) 2022-12-06 09:56:54 -06:00
client_users.go Return token when creating API-only users (#19525) 2024-06-13 19:10:27 -03:00
cron_schedules.go Implement schedule triggers (#8747) 2022-11-28 13:28:06 -06:00
cron_schedules_test.go Add instanceID to schedule logging (#10413) 2023-03-13 15:37:03 -03:00
debug_handler.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
debug_handler_test.go
device_client.go change URL for my device menu item (#28480) 2025-04-23 17:30:08 -04:00
device_client_test.go 7766 backmerge main (#14324) 2023-10-05 11:39:48 -06:00
devices.go Implement self-service install status endpoints (#28424) 2025-04-24 12:20:21 -05:00
devices_test.go Improve LUKS escrow trigger error messages (#24030) 2024-11-21 13:33:37 -06:00
endpoint_campaigns.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
endpoint_middleware.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
endpoint_middleware_test.go Android scaffold (#26274) 2025-02-13 14:32:19 -06:00
endpoint_setup.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
endpoint_utils.go Refactoring endpoint_utils (#26342) 2025-02-18 11:09:43 -06:00
endpoint_utils_test.go Added GET enterprise API endpoint. (#26555) 2025-02-26 10:47:05 -06:00
frontend.go Android: fix BYOD enrollment page to get up-to-date MDM enabled status (#27011) 2025-03-11 11:06:25 -04:00
frontend_test.go Android: fix BYOD enrollment page to get up-to-date MDM enabled status (#27011) 2025-03-11 11:06:25 -04:00
full_test.go Return 405 when receiving POST request on the route path (#16750) 2024-02-14 12:40:43 -03:00
global_policies.go Add GitOps for policy labels (#27781) 2025-04-04 09:46:51 -05:00
global_policies_test.go Add GitOps for policy labels (#27781) 2025-04-04 09:46:51 -05:00
global_schedule.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
global_schedule_test.go 21855 – Paginate and filter Queries on the server, update platform filtering from compatible to targeted platforms (#24446) 2024-12-11 10:50:28 -08:00
handler.go Add backend for uninstalls in My device UI (#29035) 2025-05-13 12:14:45 -05:00
handler_test.go Android scaffold (#26274) 2025-02-13 14:32:19 -06:00
hosts.go Fix MDM last checkin and enrollment names to match API spec (#29073) 2025-05-13 08:17:11 -04:00
hosts_test.go Fix MDM last checkin and enrollment names to match API spec (#29073) 2025-05-13 08:17:11 -04:00
http_auth_test.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
integration_core_test.go Immediately ask for a host refetch when a host re-enrolls and reuses an existing host row (#29081) 2025-05-14 09:38:53 -04:00
integration_desktop_test.go Always allow passwords for users (#26334) 2025-02-25 16:27:58 -05:00
integration_ds_only_test.go Moving Go integration tests to integration test job (#21126) 2024-08-07 14:00:25 +02:00
integration_enterprise_test.go Improve .pkg metadata extraction for names and bundle IDs, let custom package metadata extraction tool check an entire directory at a time (#29249) 2025-05-19 10:32:36 -05:00
integration_install_test.go Upcoming Activities feature branch (#25450) 2025-02-11 14:53:11 -05:00
integration_live_queries_test.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
integration_logger_test.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
integration_mdm_ddm_test.go Speculative fixes for flaky TestIntegrationsMDM tests. (#25409) 2025-01-14 16:45:10 -06:00
integration_mdm_dep_test.go Improve .pkg metadata extraction for names and bundle IDs, let custom package metadata extraction tool check an entire directory at a time (#29249) 2025-05-19 10:32:36 -05:00
integration_mdm_lifecycle_test.go Skip bootstrap package and other setup items when renewing Apple MDM enrollment profiles (#27560) 2025-03-28 16:33:22 -05:00
integration_mdm_profiles_test.go BRP: implement API endpoint for host status summary of single profile (#29039) 2025-05-13 08:49:08 -04:00
integration_mdm_test.go Improve .pkg metadata extraction for names and bundle IDs, let custom package metadata extraction tool check an entire directory at a time (#29249) 2025-05-19 10:32:36 -05:00
integration_smtp_test.go 🧹 friday cleanup party: substitute deprecated import of go-kit (#19774) 2024-06-17 10:27:31 -03:00
integration_sso_test.go 🧹 friday cleanup party: substitute deprecated import of go-kit (#19774) 2024-06-17 10:27:31 -03:00
invites.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
invites_test.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
jitter.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
jitter_test.go Update to Go 1.24.1 (#27506) 2025-03-31 11:14:09 -05:00
labels.go Add author ID to labels (#27055) 2025-03-20 16:05:16 -05:00
labels_test.go Add author ID to labels (#27055) 2025-03-20 16:05:16 -05:00
linux_mdm.go Include Linux disk encryption status in configuration profiles aggregate status response when applicable, fix disk encryption/MDM configuration order-of-operations issues, add integration tests for LUKS (#24114) 2024-11-25 08:34:43 -06:00
linux_mdm_test.go Add Linux encryption states to APIs (#23806) 2024-11-19 13:11:59 -07:00
live_queries.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
mail_test.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
maintained_apps.go Switch Fleet-maintained apps to use manifest-based structure (#27201) 2025-03-20 21:21:56 -05:00
mdm.go BRP: implement API endpoint for host status summary of single profile (#29039) 2025-05-13 08:49:08 -04:00
mdm_scep.go Updating scep package with latest fixes (#22372) 2024-09-27 07:04:11 -05:00
mdm_test.go BRP: implement API endpoint for host status summary of single profile (#29039) 2025-05-13 08:49:08 -04:00
mem_failing_policies_set_test.go
metrics.go
metrics_appconfig.go Fix SMTP e-mail send when SMTP server has credentials (#10758) 2023-03-28 15:23:15 -03:00
metrics_change_email.go
metrics_invites.go
metrics_labels.go add ability to create manual labels (#18303) 2024-04-16 17:22:08 +01:00
metrics_sessions.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
metrics_users.go
microsoft_mdm.go Log invalid SOAP message and return 400 (#28340) 2025-04-18 11:13:30 -05:00
microsoft_mdm_test.go fix: process Add commands in Windows MDM profiles (#17570) 2024-03-12 15:21:17 -04:00
orbit.go Add host id to fleet enrolled activity (#28068) 2025-04-10 13:50:14 -07:00
orbit_client.go Added more logging for troubleshooting of software package installation (#27291) 2025-03-20 11:09:57 -03:00
orbit_client_test.go Perform early restart before starting sub-systems and minor refactor of orbit sub-systems (#20610) 2024-07-19 12:44:43 -03:00
orbit_test.go Upcoming Activities feature branch (#25450) 2025-02-11 14:53:11 -05:00
osquery.go Hold off on policy queries until after setup experience (#29159) 2025-05-16 14:56:27 -04:00
osquery_test.go Hold off on policy queries until after setup experience (#29159) 2025-05-16 14:56:27 -04:00
packs.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
packs_test.go Activity feed webhooks backend (#19261) 2024-05-24 11:25:27 -05:00
queries.go Add labels to queries using gitops (#27259) 2025-03-20 15:32:52 -05:00
queries_test.go Add labels to queries using gitops (#27259) 2025-03-20 15:32:52 -05:00
scheduled_queries.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
scheduled_queries_test.go Combine Schedules and Queries: API changes (#12778) 2023-07-24 20:17:20 -04:00
scim.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
scripts.go Bulk script execution backend (#28299) 2025-04-30 12:54:46 -04:00
scripts_test.go Allow use of bash as a script interpreter (#25449) 2025-02-04 12:42:40 -05:00
secret_variables.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
secret_variables_test.go Adding secret support to scripts via gitops. (#24649) 2024-12-11 15:05:48 -06:00
service.go GitOps support for DigiCert and Custom SCEP (#27229) 2025-03-20 11:36:00 -05:00
service_appconfig.go API, Environment variable for bespoke functionality for customer-preston (#28893) 2025-05-12 13:04:18 -07:00
service_appconfig_test.go fix: reset token team assignments to defaults (#22326) 2024-10-01 10:26:16 -04:00
service_campaign_test.go Improvements to go tests in CI (#21545) 2024-08-26 08:55:53 -05:00
service_campaigns.go Full-stack: Granular host count data for live queries/policies (#27258) 2025-03-19 09:53:55 -07:00
service_errors.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
service_users.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
sessions.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
sessions_test.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
setup_experience.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
setup_experience_test.go Embedded secrets validation (#24624) 2024-12-17 17:14:12 -05:00
software.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
software_installers.go Add backend for uninstalls in My device UI (#29035) 2025-05-13 12:14:45 -05:00
software_installers_test.go Add labels and editing for VPP apps (#25979) 2025-02-03 11:16:21 -06:00
software_test.go Fixing tests. (#17073) 2024-02-22 16:03:13 -06:00
software_titles.go Add experimental software title name update endpoint for titles with a bundle ID (#26938) 2025-03-07 11:36:17 -06:00
software_titles_test.go Add experimental software title name update endpoint for titles with a bundle ID (#26938) 2025-03-07 11:36:17 -06:00
status.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
targets.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
targets_test.go Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
team_policies.go Include labels when creating team policy (#28142) 2025-04-11 11:04:44 -04:00
team_policies_test.go Add VPP policy automation support to backend (#25154) 2025-01-13 15:53:24 -06:00
team_schedule.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
team_schedule_test.go 21855 – Paginate and filter Queries on the server, update platform filtering from compatible to targeted platforms (#24446) 2024-12-11 10:50:28 -08:00
teams.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
teams_test.go Update to Go 1.24.1 (#27506) 2025-03-31 11:14:09 -05:00
testing_client.go Support for fleet maintained apps in gitops (#28751) 2025-05-07 18:16:08 -05:00
testing_utils.go Update error message for VerifyMDMWindowsConfigured (#28646) 2025-04-29 14:11:31 -05:00
translator.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
transport.go Refactoring endpoint_utils (#26342) 2025-02-18 11:09:43 -06:00
transport_setup.go
transport_test.go disable_issues and host issues support for other API endpoints (#19802) 2024-06-18 14:13:38 -05:00
trigger.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
user_roles.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
users.go Always allow passwords for users (#26334) 2025-02-25 16:27:58 -05:00
users_test.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
validation_setup.go Full-stack: Make "Server url" validation conditions consistent across Fleet, update Web Address form validation and submission logic per Fleet best practices (frontend/docs/patterns.md) (#27455) 2025-03-27 13:56:38 -07:00
vpp.go software categories: backend (#28479) 2025-05-02 11:41:26 -04:00
vpp_test.go NDES SCEP proxy backend (#22542) 2024-10-09 13:47:27 -05:00
vulnerabilities.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
vulnerabilities_test.go Updated /api/v1/fleet/vulnerabilities/{cve} endpoint (#21463) 2024-08-21 12:52:28 -05:00