Elgato_dark/fleet
1
0
Fork 0
mirror of https://github.com/fleetdm/fleet synced 2026-05-14 12:38:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 143
fleet/server/service
History
Scott Gress d716265641
Add "generate-gitops" command (#28555) 
For #27476

# Checklist for submitter

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

# Details

This PR adds a new command `generate-gitops` to the `fleetctl` tool. The
purpose of this command is to output GitOps-ready files that can then be
used with `fleetctl-gitops`.

The general usage of the command is:

```
fleectl generate-gitops --dir /path/to/dir/to/add/files/to
```

By default, the outputted files will not contain sensitive data, but
will instead add comments where the data needs to be replaced by a user.
In cases where sensitive data is redacted, the tool outputs warnings to
the user indicating which keys need to be updated.

The tool uses existing APIs to gather data for use in generating
configuration files. In some cases new API client methods needed to be
added to support the tool:

* ListConfigurationProfiles
* GetProfileContents
* GetScriptContents
* GetSoftwareTitleByID

Additionally, the response for the /api/latest/fleet/software/batch
endpoint was updated slightly to return `HashSHA256` for the software
installers. This allows policies that automatically install software to
refer to that software by hash.

Other options that we may or may not choose to document at this time:

* `--insecure`: outputs sensitive data in plaintext instead of leaving
comments
* `--print`: prints the output to stdout instead of writing files
* `--key`: outputs the value at a keypath to stdout, e.g. `--key
agent_options.config`
* `--team`: only generates config for the specified team name
* `--force`: overwrites files in the given directory (defaults to false,
which errors if the dir is not empty)

# Technical notes

The command is implemented using a `GenerateGitopsCommand` type which
holds some state (like a list of software and scripts encountered) as
well as a Fleet client instance (which may be a mock instance for tests)
and the CLI context (containing things like flags and output writers).
The actual "action" of the CLI command calls the `Run()` method of the
`GenerateGitopsCommand` var, which delegates most of the work to other
methods like `generateOrgSettings()`, `generateControls()`, etc.

Wherever possible, the subroutines use reflection to translate Go struct
fields into JSON property names. This guarantees that the correct keys
are written to config files, and protects against the unlikely event of
keys changing.

When sensitive data is encountered, the subroutines call `AddComment()`
to get a new token to add to the config files. These tokens are replaced
with comments like `# TODO - Add your enrollment secrets here` in the
final output.

# Known issues / TODOs:

* The `macos_setup` configuration is not output by this tool yet. More
planning is required for this. In the meantime, if the tool detects that
`macos_setup` is configured on the server, it outputs a key with an
invalid value and prints a warning to the user that they'll need to
configure it themselves.
* `yara_rules` are not output yet. The tool adds a warning that if you
have Yara rules (which you can only upload via GitOps right now) that
you'll have to migrate them manually. Supporting this will require a new
API that we'll have to discuss the authz for, so punting on it for now.
* Fleet maintained apps are not supported by GitOps yet (coming in
https://github.com/fleetdm/fleet/issues/24469). In the meantime, this
tool will output a `fleet_maintained_apps` key and trigger a warning,
and GitOps will fail if that key is present.

---------

Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
2025-05-06 15:25:44 -05:00
..
async Update to Go 1.24.1 (#27506) 2025-03-31 11:14:09 -05:00
calendar Update Google Calendar event bodies and relevant previews in the Fleet UI (#28715) 2025-05-01 13:12:53 -07:00
contract Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
externalsvc remove unused logic specific to DEP+Okta+ROP (#11261) 2023-04-21 14:57:52 -03:00
integrationtest Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
middleware Update error message for VerifyMDMWindowsConfigured (#28646) 2025-04-29 14:11:31 -05:00
mock Update nanomdm dependency with latest bug fixes and improvements. (#23906) 2024-11-20 11:47:11 -06:00
osquery_utils Custom SCEP renewal (#28616) 2025-04-30 15:31:45 -04:00
redis_key_value Make software batch endpoint asynchronous (#22258) 2024-09-20 11:55:47 -03:00
redis_lock Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
redis_policy_set Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
schedule Clear cron schedule errors before each run (#26775) 2025-03-03 16:41:48 -06:00
testdata Only allow distribution packages for bootstrap package (#28787) 2025-05-06 11:02:13 -05:00
activities.go Cancel upcoming activities: create past canceled activities (#27956) 2025-04-09 16:08:51 -04:00
activities_test.go Cancel upcoming activities: create past canceled activities (#27956) 2025-04-09 16:08:51 -04:00
appconfig.go IdP: resend profiles when IdP values change (#28436) 2025-04-29 14:35:37 -04:00
appconfig_test.go IdP: resend profiles when IdP values change (#28436) 2025-04-29 14:35:37 -04:00
apple_mdm.go IdP vars: add integration test, address last PR comments (#28794) 2025-05-05 11:46:10 -04:00
apple_mdm_test.go IdP vars: add integration test, address last PR comments (#28794) 2025-05-05 11:46:10 -04:00
base_client.go Added more logging for troubleshooting of software package installation (#27291) 2025-03-20 11:09:57 -03:00
base_client_errors.go fleetctl: Handle "password reset required" errors (#27132) 2025-03-17 09:44:59 -07:00
base_client_test.go prevent baseClient from trying to decode 204 responses (#16060) 2024-01-11 17:55:35 -03:00
build_tags_test.go Return 405 when receiving POST request on the route path (#16750) 2024-02-14 12:40:43 -03:00
calendar.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
campaigns.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
campaigns_test.go Improved label(s) validation when running queries (#23834) 2024-11-21 16:13:30 -06:00
carves.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
carves_test.go Update carve request block id mismatch error code from 500 –> 400 (#17974) 2024-04-04 09:38:10 -07:00
client.go Add "generate-gitops" command (#28555) 2025-05-06 15:25:44 -05:00
client_appconfig.go fleetctl gitops --dry-run now errors on duplicate (or conflicting) global/team enroll secrets. (#19344) 2024-05-31 07:01:13 -05:00
client_carves.go
client_debug.go Merge conflicts 2024-04-16 10:20:59 -05:00
client_hosts.go 17257 validate bulk ops request filters (#17514) 2024-03-11 10:02:51 -06:00
client_labels.go Improved label(s) validation when running queries (#23834) 2024-11-21 16:13:30 -06:00
client_live_query.go fleetctl, API, copy updates around host identifiers (#20220) 2024-07-09 10:25:01 -07:00
client_live_query_test.go fleetctl, API, copy updates around host identifiers (#20220) 2024-07-09 10:25:01 -07:00
client_mdm.go Clear bootstrap package and enrollment profile with GitOps (#26095) 2025-02-07 14:35:51 -06:00
client_mdm_test.go fix fleetctl to set different CmdUID for Windows (#14895) 2023-11-02 15:06:37 -03:00
client_packs.go Add fleetctl upgrade-packs command to migrate 2017 packs to queries (#13078) 2023-08-08 08:21:57 -04:00
client_policies.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
client_profiles.go Add "generate-gitops" command (#28555) 2025-05-06 15:25:44 -05:00
client_queries.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
client_scripts.go Add "generate-gitops" command (#28555) 2025-05-06 15:25:44 -05:00
client_secret_variables.go Adding secret support to scripts via gitops. (#24649) 2024-12-11 15:05:48 -06:00
client_sessions.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
client_setup.go
client_software.go Add "generate-gitops" command (#28555) 2025-05-06 15:25:44 -05:00
client_targets.go Add display_name to more results (#8204) 2022-10-14 11:14:18 -03:00
client_teams.go Add VPP install automation in GitOps (#25400) 2025-01-14 12:52:39 -06:00
client_test.go Manage labels in GitOps (#27038) 2025-03-19 16:35:11 -05:00
client_trigger.go Add fleetctl trigger command (#8877) 2022-12-06 09:56:54 -06:00
client_users.go Return token when creating API-only users (#19525) 2024-06-13 19:10:27 -03:00
cron_schedules.go Implement schedule triggers (#8747) 2022-11-28 13:28:06 -06:00
cron_schedules_test.go Add instanceID to schedule logging (#10413) 2023-03-13 15:37:03 -03:00
debug_handler.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
debug_handler_test.go
device_client.go change URL for my device menu item (#28480) 2025-04-23 17:30:08 -04:00
device_client_test.go 7766 backmerge main (#14324) 2023-10-05 11:39:48 -06:00
devices.go Implement self-service install status endpoints (#28424) 2025-04-24 12:20:21 -05:00
devices_test.go Improve LUKS escrow trigger error messages (#24030) 2024-11-21 13:33:37 -06:00
endpoint_campaigns.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
endpoint_middleware.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
endpoint_middleware_test.go Android scaffold (#26274) 2025-02-13 14:32:19 -06:00
endpoint_setup.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
endpoint_utils.go Refactoring endpoint_utils (#26342) 2025-02-18 11:09:43 -06:00
endpoint_utils_test.go Added GET enterprise API endpoint. (#26555) 2025-02-26 10:47:05 -06:00
frontend.go Android: fix BYOD enrollment page to get up-to-date MDM enabled status (#27011) 2025-03-11 11:06:25 -04:00
frontend_test.go Android: fix BYOD enrollment page to get up-to-date MDM enabled status (#27011) 2025-03-11 11:06:25 -04:00
full_test.go Return 405 when receiving POST request on the route path (#16750) 2024-02-14 12:40:43 -03:00
global_policies.go Add GitOps for policy labels (#27781) 2025-04-04 09:46:51 -05:00
global_policies_test.go Add GitOps for policy labels (#27781) 2025-04-04 09:46:51 -05:00
global_schedule.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
global_schedule_test.go 21855 – Paginate and filter Queries on the server, update platform filtering from compatible to targeted platforms (#24446) 2024-12-11 10:50:28 -08:00
handler.go Bulk script execution backend (#28299) 2025-04-30 12:54:46 -04:00
handler_test.go Android scaffold (#26274) 2025-02-13 14:32:19 -06:00
hosts.go software categories: backend (#28479) 2025-05-02 11:41:26 -04:00
hosts_test.go software categories: backend (#28479) 2025-05-02 11:41:26 -04:00
http_auth_test.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
integration_core_test.go Custom SCEP renewal (#28616) 2025-04-30 15:31:45 -04:00
integration_desktop_test.go Always allow passwords for users (#26334) 2025-02-25 16:27:58 -05:00
integration_ds_only_test.go Moving Go integration tests to integration test job (#21126) 2024-08-07 14:00:25 +02:00
integration_enterprise_test.go Rewrite label scoping logic for listing host software, fix other ListHostSoftware bugs (#28592) 2025-05-06 12:32:35 -05:00
integration_install_test.go Upcoming Activities feature branch (#25450) 2025-02-11 14:53:11 -05:00
integration_live_queries_test.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
integration_logger_test.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
integration_mdm_ddm_test.go Speculative fixes for flaky TestIntegrationsMDM tests. (#25409) 2025-01-14 16:45:10 -06:00
integration_mdm_dep_test.go Added macos_setup.manual_agent_install to global/team config (#28419) 2025-04-24 15:18:01 -05:00
integration_mdm_lifecycle_test.go Skip bootstrap package and other setup items when renewing Apple MDM enrollment profiles (#27560) 2025-03-28 16:33:22 -05:00
integration_mdm_profiles_test.go IdP vars: store profile-variable associations when new profile is created/updated (#28645) 2025-04-30 16:03:23 -04:00
integration_mdm_test.go Only allow distribution packages for bootstrap package (#28787) 2025-05-06 11:02:13 -05:00
integration_smtp_test.go 🧹 friday cleanup party: substitute deprecated import of go-kit (#19774) 2024-06-17 10:27:31 -03:00
integration_sso_test.go 🧹 friday cleanup party: substitute deprecated import of go-kit (#19774) 2024-06-17 10:27:31 -03:00
invites.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
invites_test.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
jitter.go Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
jitter_test.go Update to Go 1.24.1 (#27506) 2025-03-31 11:14:09 -05:00
labels.go Add author ID to labels (#27055) 2025-03-20 16:05:16 -05:00
labels_test.go Add author ID to labels (#27055) 2025-03-20 16:05:16 -05:00
linux_mdm.go Include Linux disk encryption status in configuration profiles aggregate status response when applicable, fix disk encryption/MDM configuration order-of-operations issues, add integration tests for LUKS (#24114) 2024-11-25 08:34:43 -06:00
linux_mdm_test.go Add Linux encryption states to APIs (#23806) 2024-11-19 13:11:59 -07:00
live_queries.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
mail_test.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
maintained_apps.go Switch Fleet-maintained apps to use manifest-based structure (#27201) 2025-03-20 21:21:56 -05:00
mdm.go IdP vars: add integration test, address last PR comments (#28794) 2025-05-05 11:46:10 -04:00
mdm_scep.go Updating scep package with latest fixes (#22372) 2024-09-27 07:04:11 -05:00
mdm_test.go IdP vars: add integration test, address last PR comments (#28794) 2025-05-05 11:46:10 -04:00
mem_failing_policies_set_test.go
metrics.go
metrics_appconfig.go Fix SMTP e-mail send when SMTP server has credentials (#10758) 2023-03-28 15:23:15 -03:00
metrics_change_email.go
metrics_invites.go
metrics_labels.go add ability to create manual labels (#18303) 2024-04-16 17:22:08 +01:00
metrics_sessions.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
metrics_users.go
microsoft_mdm.go Log invalid SOAP message and return 400 (#28340) 2025-04-18 11:13:30 -05:00
microsoft_mdm_test.go fix: process Add commands in Windows MDM profiles (#17570) 2024-03-12 15:21:17 -04:00
orbit.go Add host id to fleet enrolled activity (#28068) 2025-04-10 13:50:14 -07:00
orbit_client.go Added more logging for troubleshooting of software package installation (#27291) 2025-03-20 11:09:57 -03:00
orbit_client_test.go Perform early restart before starting sub-systems and minor refactor of orbit sub-systems (#20610) 2024-07-19 12:44:43 -03:00
orbit_test.go Upcoming Activities feature branch (#25450) 2025-02-11 14:53:11 -05:00
osquery.go Add query IDs to query automation logs (#28570) 2025-05-01 15:21:30 -04:00
osquery_test.go Add query IDs to query automation logs (#28570) 2025-05-01 15:21:30 -04:00
packs.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
packs_test.go Activity feed webhooks backend (#19261) 2024-05-24 11:25:27 -05:00
queries.go Add labels to queries using gitops (#27259) 2025-03-20 15:32:52 -05:00
queries_test.go Add labels to queries using gitops (#27259) 2025-03-20 15:32:52 -05:00
scheduled_queries.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
scheduled_queries_test.go Combine Schedules and Queries: API changes (#12778) 2023-07-24 20:17:20 -04:00
scim.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
scripts.go Bulk script execution backend (#28299) 2025-04-30 12:54:46 -04:00
scripts_test.go Allow use of bash as a script interpreter (#25449) 2025-02-04 12:42:40 -05:00
secret_variables.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
secret_variables_test.go Adding secret support to scripts via gitops. (#24649) 2024-12-11 15:05:48 -06:00
service.go GitOps support for DigiCert and Custom SCEP (#27229) 2025-03-20 11:36:00 -05:00
service_appconfig.go Webhook logging destination (#28692) 2025-05-01 16:13:04 -04:00
service_appconfig_test.go fix: reset token team assignments to defaults (#22326) 2024-10-01 10:26:16 -04:00
service_campaign_test.go Improvements to go tests in CI (#21545) 2024-08-26 08:55:53 -05:00
service_campaigns.go Full-stack: Granular host count data for live queries/policies (#27258) 2025-03-19 09:53:55 -07:00
service_errors.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
service_users.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
sessions.go Added scim/details endpoint (#28007) 2025-04-10 14:08:45 -05:00
sessions_test.go Refactoring service layer. Part 1 (#25945) 2025-02-03 11:23:26 -06:00
setup_experience.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
setup_experience_test.go Embedded secrets validation (#24624) 2024-12-17 17:14:12 -05:00
software.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
software_installers.go software categories: backend (#28479) 2025-05-02 11:41:26 -04:00
software_installers_test.go Add labels and editing for VPP apps (#25979) 2025-02-03 11:16:21 -06:00
software_test.go Fixing tests. (#17073) 2024-02-22 16:03:13 -06:00
software_titles.go Add experimental software title name update endpoint for titles with a bundle ID (#26938) 2025-03-07 11:36:17 -06:00
software_titles_test.go Add experimental software title name update endpoint for titles with a bundle ID (#26938) 2025-03-07 11:36:17 -06:00
status.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
targets.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
targets_test.go Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
team_policies.go Include labels when creating team policy (#28142) 2025-04-11 11:04:44 -04:00
team_policies_test.go Add VPP policy automation support to backend (#25154) 2025-01-13 15:53:24 -06:00
team_schedule.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
team_schedule_test.go 21855 – Paginate and filter Queries on the server, update platform filtering from compatible to targeted platforms (#24446) 2024-12-11 10:50:28 -08:00
teams.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
teams_test.go Update to Go 1.24.1 (#27506) 2025-03-31 11:14:09 -05:00
testing_client.go software categories: backend (#28479) 2025-05-02 11:41:26 -04:00
testing_utils.go Update error message for VerifyMDMWindowsConfigured (#28646) 2025-04-29 14:11:31 -05:00
translator.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
transport.go Refactoring endpoint_utils (#26342) 2025-02-18 11:09:43 -06:00
transport_setup.go
transport_test.go disable_issues and host issues support for other API endpoints (#19802) 2024-06-18 14:13:38 -05:00
trigger.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
user_roles.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
users.go Always allow passwords for users (#26334) 2025-02-25 16:27:58 -05:00
users_test.go Allow opting in users to email verification on login (#24273) 2024-12-05 08:37:10 -06:00
validation_setup.go Full-stack: Make "Server url" validation conditions consistent across Fleet, update Web Address form validation and submission logic per Fleet best practices (frontend/docs/patterns.md) (#27455) 2025-03-27 13:56:38 -07:00
vpp.go software categories: backend (#28479) 2025-05-02 11:41:26 -04:00
vpp_test.go NDES SCEP proxy backend (#22542) 2024-10-09 13:47:27 -05:00
vulnerabilities.go service.errorer to fleet.Errorer (#26362) 2025-02-14 16:19:34 -06:00
vulnerabilities_test.go Updated /api/v1/fleet/vulnerabilities/{cve} endpoint (#21463) 2024-08-21 12:52:28 -05:00