fleet/changes
Scott Gress d716265641
Add "generate-gitops" command (#28555)
For #27476

# Checklist for submitter

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

# Details

This PR adds a new command `generate-gitops` to the `fleetctl` tool. The
purpose of this command is to output GitOps-ready files that can then be
used with `fleetctl-gitops`.

The general usage of the command is:

```
fleectl generate-gitops --dir /path/to/dir/to/add/files/to
```

By default, the outputted files will not contain sensitive data, but
will instead add comments where the data needs to be replaced by a user.
In cases where sensitive data is redacted, the tool outputs warnings to
the user indicating which keys need to be updated.

The tool uses existing APIs to gather data for use in generating
configuration files. In some cases new API client methods needed to be
added to support the tool:

* ListConfigurationProfiles
* GetProfileContents
* GetScriptContents
* GetSoftwareTitleByID

Additionally, the response for the /api/latest/fleet/software/batch
endpoint was updated slightly to return `HashSHA256` for the software
installers. This allows policies that automatically install software to
refer to that software by hash.

Other options that we may or may not choose to document at this time:

* `--insecure`: outputs sensitive data in plaintext instead of leaving
comments
* `--print`: prints the output to stdout instead of writing files
* `--key`: outputs the value at a keypath to stdout, e.g. `--key
agent_options.config`
* `--team`: only generates config for the specified team name
* `--force`: overwrites files in the given directory (defaults to false,
which errors if the dir is not empty)

# Technical notes

The command is implemented using a `GenerateGitopsCommand` type which
holds some state (like a list of software and scripts encountered) as
well as a Fleet client instance (which may be a mock instance for tests)
and the CLI context (containing things like flags and output writers).
The actual "action" of the CLI command calls the `Run()` method of the
`GenerateGitopsCommand` var, which delegates most of the work to other
methods like `generateOrgSettings()`, `generateControls()`, etc.

Wherever possible, the subroutines use reflection to translate Go struct
fields into JSON property names. This guarantees that the correct keys
are written to config files, and protects against the unlikely event of
keys changing.

When sensitive data is encountered, the subroutines call `AddComment()`
to get a new token to add to the config files. These tokens are replaced
with comments like `# TODO - Add your enrollment secrets here` in the
final output.

# Known issues / TODOs:

* The `macos_setup` configuration is not output by this tool yet. More
planning is required for this. In the meantime, if the tool detects that
`macos_setup` is configured on the server, it outputs a key with an
invalid value and prints a warning to the user that they'll need to
configure it themselves.
* `yara_rules` are not output yet. The tool adds a warning that if you
have Yara rules (which you can only upload via GitOps right now) that
you'll have to migrate them manually. Supporting this will require a new
API that we'll have to discuss the authz for, so punting on it for now.
* Fleet maintained apps are not supported by GitOps yet (coming in
https://github.com/fleetdm/fleet/issues/24469). In the meantime, this
tool will output a `fleet_maintained_apps` key and trigger a warning,
and GitOps will fail if that key is present.

---------

Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
2025-05-06 15:25:44 -05:00
..
.keep Issue 1009 calculate diff software (#1305) 2021-07-08 13:57:43 -03:00
10098-org-logo Handle wide logos in MDM setup experience and migration dialog on Apple devices (#28539) 2025-04-29 09:39:28 -05:00
19235-microsoft-compliance-conditional-access UI: Conditional access - Microsoft Entra (#27982) 2025-04-15 13:55:07 -07:00
23497-gitops-sha enable specifying installers by SHA256 in gitops (#28349) 2025-04-18 16:41:41 -04:00
24087-app-names fix: change how macOS software names are calculated to avoid erroneous duplicates (#28037) 2025-04-11 19:19:07 -04:00
24209-windows-mdm-error-msg Update API message for Windows MDM not configured error (#28247) 2025-04-15 13:52:17 -05:00
24394-wlan-xml-profile-verification Update WLAN XML profile verification so they aren't resent (#28296) 2025-04-18 08:45:18 -04:00
24475-manual_agent_install Added macos_setup.manual_agent_install to global/team config (#28419) 2025-04-24 15:18:01 -05:00
24878-renew-ndes NDES cert renewal (#28712) 2025-05-01 12:16:45 -05:00
25700-luks-escrow-longtime Removed indicator for background LUKS validation (#28218) 2025-04-16 12:25:41 -04:00
26114-new-team-with-vpp-apps Fixed fleetctl gitops issue where creating a new team containing VPP apps caused an error. (#28624) 2025-04-29 16:28:25 -05:00
26366-update-sql-parser Update SQL parser to handle more modern syntax (#28211) 2025-04-16 10:10:52 -05:00
26553-renew-digicert Renewal of DigiCert certificates on macOS (#28449) 2025-04-24 08:35:15 -04:00
26692-upload-install-tarballs Fleet UI: Upload and install tarball archives (#27839) 2025-05-02 10:17:09 -04:00
27003-list-host-software Refactoring ListHostSoftware (#27490) 2025-04-10 17:29:15 -05:00
27007-host-certificate-serial-display Display host certificate decimal serials in addition to hex for smaller values to match keychain (#28732) 2025-05-01 14:29:11 -04:00
27052-round-live-policy-percentages UI: Fix live policy response percentage rounding (#28719) 2025-05-01 10:03:23 -07:00
27436-query-automation-query-ids Add query IDs to query automation logs (#28570) 2025-05-01 15:21:30 -04:00
27445-webhook-automation-backend Webhook logging destination (#28692) 2025-05-01 16:13:04 -04:00
27458-update-calendar-events Update Google Calendar event bodies and relevant previews in the Fleet UI (#28715) 2025-05-01 13:12:53 -07:00
27463-add-script-syntax-highlighting Add syntax highlighting support for shell and powershell scripts (#28417) 2025-04-29 10:24:05 -05:00
27475-fix-query-page-selection-clear Fx query page clearing selection after load (#28228) 2025-04-16 10:10:21 -05:00
27476-add-generate-gitops-cmd Add "generate-gitops" command (#28555) 2025-05-06 15:25:44 -05:00
27477-do-not-interpolate-gitops-text-sections Exclude certain sections from var interpolation when running gitops cmd (#28625) 2025-04-29 15:09:25 -04:00
27501-clear-windows-mdm-info-on-linux-enroll Clear host_mdm table row when existing Windows hosts enroll as a different OS (#28463) 2025-04-23 08:53:24 -04:00
27700-validate-bootstrap-is-distribution-pkg Only allow distribution packages for bootstrap package (#28787) 2025-05-06 11:02:13 -05:00
27822-dep-host-cleanup Fix CleanupExpiredHosts to prevent deletion of DEP-assigned hosts (#28313) 2025-04-18 12:49:03 -05:00
27909-hover-checkbox-radio Fleet UI: Added hover cursors to checkbox and radio components (#28113) 2025-04-16 13:29:08 -04:00
27941-GOM-shouldn't-apply-to-VPP-token UI: Allow VPP token upload in GitOps mode (#28746) 2025-05-02 09:31:36 -07:00
28032-email-to-header Add To: header when constructing emails (#28507) 2025-04-24 09:00:35 -04:00
28033-support-end-user-idp-fleet-vars IdP vars: Populate IdP fleet variables in macOS configuration profiles (#28291) 2025-04-22 09:09:00 -04:00
28049-conditional-access-fe-updates UI: Update conditional access on a per-policy basis (#28658) 2025-05-01 11:43:38 -07:00
28054-surface-policy-count-triggering-install Fleet UI: Surface policy count triggering automatic installations (#28726) 2025-05-02 11:01:26 -04:00
28137-my-device-page-layout-changes Fleet UI: Device user/Host details page layout changing including split out host header and summary card (#28598) 2025-04-28 13:00:13 -04:00
28138-categories-be software categories: backend (#28479) 2025-05-02 11:41:26 -04:00
28154-fix-gitops-dry-run-labels Don't validate software/profile labels in dry run mode (#28201) 2025-04-17 08:39:24 -05:00
28158-batch-script-backend Bulk script execution backend (#28299) 2025-04-30 12:54:46 -04:00
28159-batch-scripts UI: Add ability to run scripts on batches of hosts (#28563) 2025-04-28 16:32:41 -07:00
28166-webhook-logging UI: Support webhook logging configuration (#28737) 2025-05-02 09:04:05 -07:00
28240-log-invalid-soap-msg Log invalid SOAP message and return 400 (#28340) 2025-04-18 11:13:30 -05:00
28266-clean-up-settings UI: Clean up various states on the Settings page (#28752) 2025-05-02 09:14:30 -07:00
28343-reverting-some-mcx-validation Removing Apple MDM profile validation checks for some com.apple.MCX keys (#28498) 2025-04-23 14:06:17 -05:00
28379-vpp-app-install-status Fleet UI: VPP apps with self service shows correct install status (#28739) 2025-05-02 10:41:06 -04:00
28411-self-service-install-status Implement self-service install status endpoints (#28424) 2025-04-24 12:20:21 -05:00
28443-add-hash-to-software-title-response Add hash_sha256 field to "List Software Titles" API response (#28447) 2025-04-24 12:08:59 -05:00
28543-exe-edits Fix broken installer PATCH (always failing with "missing install script" error) on EXEs (#28670) 2025-04-29 17:53:52 -05:00
28616-renew-custom-scep Custom SCEP renewal (#28616) 2025-04-30 15:31:45 -04:00
28619-remove-paths-err Downgrade sotware paths error (#28736) 2025-05-05 14:47:59 -06:00
28743-gom-premium-only UI: Premium feature message when viewing GitOps Mode toggle on Fleet Free (#28744) 2025-05-02 09:34:37 -07:00
28747-allow-advanced-save-in-GOM UI: Enable Integrations > Advanced save button in GitOps Mode (#28748) 2025-05-02 09:32:34 -07:00
C10298-additional-stats Add statistic to measure ABM pending hosts (#28226) 2025-04-15 11:30:07 -04:00
issue-26070-add-ui-for-manual-agent-install Add UI for enabling manual agent install of a bootstrap package (#28550) 2025-04-29 15:29:21 +01:00
issue-26148-remove-config-settings-when-no-team-removed remove no team gitops setting when no-team.yml is not supplied (#28082) 2025-04-22 16:40:17 +01:00
issue-26505-keyboard-accssibility-to-activites Add keyboard accessibility controls to the activity items on host details and dashboard pages (#28433) 2025-04-25 10:21:56 +01:00
issue-27198-long-host-name add truncation and tooltip to host details host with long name (#28547) 2025-04-28 17:09:42 +01:00
issue-27775-fix-activity-date-tooltip Fix host upcoming activites showing wrong created at date in tooltip (#28242) 2025-04-16 10:17:48 +01:00
migration-speedup Add temporary index during migration to update host software installed paths more quickly (#28627) 2025-04-28 20:38:40 -05:00
update-go-1.24.2 Fix detected CVEs and docker scout exit code to fail the Github Action (#28836) 2025-05-06 13:35:27 -03:00