<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** The entire ACME feature branch merge
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
---------
Co-authored-by: Jordan Montgomery <elijah.jordan.montgomery@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#36751
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] Verified that `fleetctl generate-gitops` correctly outputs
policies with `install_software.fleet_maintained_app_slug` populated
when the policies have FMA automation
- [X] Verified that running `fleetctl gitops` using files with
`install_software.fleet_maintained_app_slug` creates/updates FMA policy
automation correctly
- [X] Verified no changes to the above for custom packages or VPP apps
- [X] Verified that when software is excepted from GitOps, FMA policy
automations still work (correctly validates FMAs exist before applying)
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
checking on this
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [X] Verified that any relevant UI is disabled when GitOps mode is
enabled
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#42180
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enhanced GitOps exception handling for labels, secrets, and software
with clearer enforcement and omission semantics.
* Server-side prefetch of team software so omitted team software can
preserve existing installers during validation.
* Presence flags track whether top-level keys (labels, secrets,
software) were provided versus omitted.
* **Behavior Changes**
* Omitted vs empty sections are now distinguished: omission can mean
“no-op” or “delete-all” depending on exception settings.
* GitOps YAML can define and manage labels directly; validations now
reject YAML that includes keys marked as excepted.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
* **Labels**
- [ ] Validated that with label exceptions off, omitting `labels:` key
from default.yml clears all global labels
- [ ] Validated that with label exceptions off, omitting `labels:` key
from a fleet .yml clears all labels for that fleet
- [ ] Validated that with label exceptions off, setting empty `labels:`
key from default.yml clears all global labels
- [ ] Validated that with label exceptions off, setting empty `labels:`
key from a fleet .yml clears all labels for that fleet
- [ ] Validated that with label exceptions on, omitting `labels:` key
from default .yml leaves existing global labels as-is
- [ ] Validated that with label exceptions on, omitting `labels:` key
from a fleet .yml leaves existing labels as-is
- [ ] Validated that with label exceptions on, setting `labels:` key on
default .yml generates an error
- [ ] Validated that with label exceptions on, setting `labels:` key on
a fleet .yml generates an error
- [ ] Validated that with label exceptions on, a policy using
`labels_include_any` referencing an existing label succeeds without
`labels:` key
- [ ] Validated that with label exceptions on, a query using
`labels_include_any` referencing an existing label succeeds without
`labels:` key
- [ ] Validated that with label exceptions on, an MDM profile using
`labels_include_any` referencing an existing label succeeds without
`labels:` key
- [ ] Validated that with label exceptions on, a software package using
`labels_include_any` referencing an existing label succeeds without
`labels:` key (requires software exceptions off)
- [ ] Validated that with label exceptions on, an app store app using
`labels_include_any` referencing an existing label succeeds without
`labels:` key (requires software exceptions off)
- [ ] Validated that with label exceptions on, a fleet maintained app
using `labels_include_any` referencing an existing label succeeds
without `labels:` key (requires software exceptions off)
* **Secrets**
- [ ] Validated that with secrets exceptions off, omitting `secrets:`
key from default.yml clears all global secrets
- [ ] Validated that with secrets exceptions off, omitting `secrets:`
key from a fleet .yml clears all secrets for that fleet
- [ ] Validated that with secrets exceptions on, omitting `secrets:` key
from default .yml leaves existing global secrets as-is
- [ ] Validated that with secrets exceptions on, omitting `secrets:` key
from a fleet .yml leaves existing secrets as-is
- [ ] Validated that with secrets exceptions on, setting `secrets:` key
on default .yml generates an error
- [ ] Validated that with secrets exceptions on, setting `secrets:` key
on a fleet .yml generates an error
* **Software**
- [ ] Validated that with software exceptions off, omitting `software:`
key from no-team.yml/unassigned.yml clears all software for "no team"
- [ ] Validated that with software exceptions off, omitting `software:`
key from a fleet .yml clears all software for that fleet
- [ ] Validated that with software exceptions off, setting empty
`software:` key on a fleet .yml clears all software for that fleet
- [ ] Validated that with software exceptions off, setting empty
`software:` key on no-team.yml/unassigned.yml clears all software for
"no team
- [ ] Validated that with software exceptions on, omitting `software:`
key from a fleet .yml leaves existing software as-is
- [ ] Validated that with software exceptions on, setting `software:`
key on a fleet .yml generates an error
- [ ] Validated that with software exceptions on, omitting `software:`
key from no-team.yml/unassigned.yml leaves existing software as-is for
"no team"
- [ ] Validated that with software exceptions on, setting `software:`
key on no-team.yml/unassigned.yml generates an error
- [ ] Validated that with software exceptions on, a policy using
`install_software.hash_sha256` referencing an existing package succeeds
without `software:` key
- [ ] Validated that with software exceptions on, a policy using
`install_software.app_store_id` referencing an existing VPP app succeeds
without `software:` key
- [ ] Validated that with software exceptions on, a patch policy using
`fleet_maintained_app_slug` referencing an existing FMA succeeds without
`software:` key
- [ ] Validated that with software exceptions on,
`setup_experience.software` referencing existing software succeeds
without `software:` key (server-side validation fallback)
- [ ] Validated that with software exceptions on, omitting `software:`
from no-team.yml/unassigned.yml preserves existing no-team software
- [ ] Validated that with software exceptions on, a policy in
no-team.yml/unassigned.yml using `install_software.hash_sha256`
referencing existing no-team software succeeds without `software:` key
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
I don't think so. There is a bit of overhead when this feature is used
since we have to fetch software from the server, but it would be done in
a specific test, so even if there is an impact it should affect existing
load testing, only new, specific tests.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41345
# Details
This PR:
* Adds a new `fleetctl new` command which creates a starter GitOps repo
file structure
* Adds support for file globs for the `configuration_profiles:` key in
GitOps, to support its use in the `fleetctl new` templates. This
involved moving the `BaseItem` type and `SupportsFileInclude` interface
into the `fleet` package so that the `MDMProfileSpec` type could
implement the interface and do glob expansion.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] added unit and intg tests for globbing profiles
- [ ] added tests for `fleetctl new`
- [X] QA'd all new/changed functionality manually
- [X] `fleetctl new` with no args prompted for org name and created a
new `it-and-security` folder under current folder w/ correct files
- [X] `fleetctl new --dir /tmp/testnew` created correct files under
`/tmp/testnew`
- [X] `fleetctl new --dir /tmp/testexisting --force` with an existing
`/tmp/testexisting` folder created correct files under
`/tmp/testexisting`
- [X] `fleetctl new --org-name=foo` created correct files under
`it-and-security` without prompting for org name
- [X] `paths:` in `configuration_profiles` picks up multiple matching
profiles
- [X] `paths:` + `path:` in `configuration_profiles` will error if the
same profile is picked up twice
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `fleetctl new` command to initialize GitOps repository structure
via CLI.
* Added glob pattern support for `configuration_profiles` field,
enabling flexible profile selection.
* **Chores**
* Updated CLI dependencies to support enhanced user interactions.
* Removed legacy website generator configuration files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41672
# Details
Updates GitOps label functionality so that omitting the `hosts:` key
under a manual label will _not_ clear hosts from that label, but will
instead preserve the existing membership. This allows users to manage
manual hosts with an external system (via the labels API), while still
managing the labels themselves in GitOps.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] verified that you can still add a manual label with `hosts:`
- [X] verified that leaving `hosts:` off a manual label doesn't change
the host assignment
- [X] verified that putting `hosts:` with no value on a manual label
clears the hosts
- [X] verified that you can still add a dynamic label
- [X] verified that generate-gitops still exports manual hosts
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40721
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
I (Martin) did test `labels_include_all` for FMA, custom installer, IPA
and VPP apps, and it seemed to all work great for gitops apply and
gitops generate, **except for VPP apps** which seem to have 2 important
pre-existing bugs, see
https://github.com/fleetdm/fleet/issues/40723#issuecomment-4041780707
## New Fleet configuration settings
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
---------
Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
Resolves#36312
- Validate and trim fleet names in NewTeam, ModifyTeam, and
ApplyTeamSpecs
- Trim fleet names in gitops YAML parsing (parseName)
- Disable submit button in CreateTeamModal and RenameTeamModal when name
is whitespace-only
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41280
# Details
Phase 2 of the "detect unknown keys in GitOps" work. The `org_settings`
and `settings` top-level keys mainly shadow the `fleet.AppConfig` and
`fleet.TeamConfig` types, but they have a couple of extra GitOps-only
fields, so we add new GitOps-specific types for them (similar to what we
already have for `GitOpsControls` and `GitOpsSoftware`. The
`org_settings:` case is further complicated by the fact that its extra
fields are themselves `any` types which we need to parse, so we add
those to the `anyFieldTypes` registry in the validator to tell it what
types to check them against.
Also had to add some new logic to handle the GoogleCalendarAPI case
which doesn't expose its keys as `json` tags at all, since we use a
special method to obfuscate the values.
I've tested this by routing the output from `fleetctl generate_gitops`
back through `fleetctl gitops`, which is how I caught the
`end_user_license_agreement` issue.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a - already added in previous PR
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
Did the `fleetctl generate-gitops` -> `fleetctl gitops` loop as
mentioned above.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added support for managing secrets and certificate authorities through
GitOps configuration
* Improved detection of configuration errors with clear error messages
when using unknown or misspelled settings keys, including suggestions
for common typos
* Enhanced error reporting for nested configuration files with precise
location information
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40627
# Details
This PR updates the way we enable failed policy reporting (via webhook
or ticket integration) for individual policies in GitOps. The existing
method is to declare a `policy_ids` key underneath
`failing_policies_webhook:` in either the global or a fleet .yml file,
and specify a list of policy IDs to enable the automation for. This PR
maintains this feature for backwards compatibility, and adds a new
feature where you can set `webhook_and_tickets_enabled: true` key in the
policy declaration itself. If _both_ these methods are used, the GitOps
run will fail.
**Implementation note:**
Because we're keeping the old way of doing this until Fleet 5, I took
the easy route and just translated the new way into the old way; that
is, we gather up the list of policies with `webhook_and_tickets_enabled:
true`, get their IDs and send that list to the server under the same
config we did previously. This works fine and there's nothing _wrong_
with it but ideally this flag would work the same as other per-policy
flags like `calendar_events_enabled` that are stored on the policy
record. That requires a migration and more new code that we'd have to
maintain alongside the existing code (or translate the old strategy to
the new one). I'm taking the lower-touch path here.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [x] Verified that `generate-gitops` outputs the new
`webhooks_and_tickets_enabled` flag instead of outputting `policy_ids`
under `failing_policies_webhook`
- [X] Verified that using the new flag in a fleet .yml file results in
the specified policies being enabled in the "other" automations for
policies (whether the webhook automation is enabled or not)
- [X] Verified the same for a global .default.yml file
- [X] Verified that using the old `failing_policies_webhook.policy_ids`
a fleet .yml file results in the specified policies being enabled in the
"other" automations for policies (whether the webhook automation is
enabled or not)
- [X] Verified the same for a global .default.yml file
- [X] Verified that trying to use both `webhooks_and_tickets_enabled`
and `failing_policies_webhook.policy_ids` at the same time results in an
error.
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
see
https://github.com/fleetdm/fleet/issues/40627#issuecomment-4024988552
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added configuration flag to enable webhooks and tickets for policies
in GitOps settings.
* System automatically resolves and assigns policy IDs when using the
new flag.
* **Tests**
* Added comprehensive test coverage for webhook and ticket enablement in
GitOps workflows, including conflict detection and policy ID assignment
validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41307
# Details
* Fixes a potential issue where running `fleetctl gitops` with only the
global file, with no controls provided, could wipe out global controls
that are provided in the "no team" file.
* Fixes error message when controls are missing.
# Checklist for submitter
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [x] `fleetctl gitops -f /path/to/default.yml` without controls, gives
`error: 'controls' must be set on global config, no-team.yml or
unassigned.yml`
- [x] `fleetctl gitops -f /path/to/default.yml` with empty controls
works
- [x] `fleetctl gitops -f /path/to/default.yml -f /path/to/no-team.yml`
without controls, gives `error: 'controls' must be set on global config
or no-team.yml`
- [x] `fleetctl gitops -f /path/to/default.yml -f
/path/to/unassigned.yml` without controls, gives `error: 'controls' must
be set on global config or unassigned.yml`
- [x] `fleetctl gitops -f /path/to/default.yml -f /path/to/no-team.yml`
with empty controls in no-team.yml works
- [x] `fleetctl gitops -f /path/to/default.yml -f
/path/to/unassigned.yml` with empty controls in unassigned.yml works
- [x] `fleetctl gitops -f /path/to/no-team.yml` gives error `global
config must be provided alongside no-team.yml`
- [x] `fleetctl gitops -f /path/to/no-team.yml` gives error `global
config must be provided alongside unassigned.yml`
- [x] `fleetctl gitops -f /path/to/some-real-team.yml` with no controls
works
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41006
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
Added tests for using path, paths and inline declaration for reports,
policies and labels.
- [X] QA'd all new/changed functionality manually
- [x] tested that `path:` works for policies
- [x] tested that `paths:` works for policies
- [x] tested that incline declaration works for policies
- [x] tested that `path:` works for reports
- [x] tested that `paths:` works for reports
- [x] tested that incline declaration works for reports
- [x] tested that `path:` works for labels
- [x] tested that `paths:` works for labels
- [x] tested that incline declaration works for labels
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for glob patterns in path specifications within reports,
labels, and policies configuration sections.
* Enhanced validation and error handling for external file references.
* Improved logging and error messages during configuration parsing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41012
# Details
This PR makes it allowable to leave out almost all top-level keys from
GitOps files. The only required keys are _either_ `name:` (for a fleet
settings file) or `org_settings:` (for a global settings file). Omitting
a key is identical to supplying it with no value.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
Updated the "missing all global keys test", and added some new tests to
verify that omitting the key was the same as supplying it with an empty
value.
- [X] QA'd all new/changed functionality manually
1. Ran `fleetctl generate-gitops` to get a clean set of GitOps yml files
2. Removed all removable keys from default.yml and ran `fleetctl gitops`
3. Ran `fleetctl generate-gitops` again into a different dir
4. Ran `fleetctl gitops` with the original files to get back to original
state
5. Cleared out all now-removable keys and replaced them with empty value
(e.g. `reports:` with nothing under it)
6. Ran `fleetctl generate-gitops` again into a third dir
7. Compared the files from the second and third generate-gitops runs to
verify that omitting the key had the same result as supplying it with an
empty value
8. Did the above steps with a fleet (i.e. non-global) .yml file.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* GitOps files now support omitting top-level configuration keys instead
of requiring them to be explicitly set to empty values.
* org_settings is now required when team name is not specified.
* **Tests**
* Added integration tests validating behavior when omitting top-level
keys in global and team-level GitOps configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40496
# Details
This is the first phase of an effort to detect unknown keys in GitOps
.yml files. In the regular `fleetctl gitops` case, it will fail when
unknown keys are detected. This behavior can be changed with a new
`--allow-unknown-keys` flag which will log the issues and continue.
In this first phase we are detecting unknown keys in _most_ GitOps
sections, other than the top-level `org_settings:` and `settings:`
sections which have more complicated typing. I will tackle those
separately as they require a bit more thought. Also ultimately I'd like
us to be doing this validation in a more top-down fashion in one place,
rather than spreading it across the code by doing it in each individual
section, but this is a good first step.
As a bonus, I invited my pal Mr. Levenshtein to the party so that we can
make suggestions when unknown keys are detected, like:
```
* unknown key "queyr" in "./lib/some-report.yml"; did you mean "query"?
```
> Note: the goal is to return as many validation errors as possible to
the user, so they don't have to keep running `fleetctl gitops` to get
the next error. I did _not_ update any other errors to stop returning
early, in an effort to keep this as low-touch as possible.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] Tested this against existing it-and-security folder and one with
updated keys from https://github.com/fleetdm/fleet/pull/40959; no
unknown keys detected
- [X] Added unknown keys at various levels, GitOps errored with helpful
messages
- [X] Same as above but with `--allow-unknown-keys`; GitOps outputted
helpful messages but continued.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* GitOps runs now fail when unknown or misspelled keys are present in
configuration files.
* New CLI flag --allow-unknown-keys lets unknown keys be treated as
warnings instead of errors.
* Unknown-key messages include suggested valid key names to help correct
mistakes.
* **Tests**
* Expanded test coverage to validate unknown-key detection and the
allow-as-warning option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40488
# Details
Implements the renames requested in #40488:
- [X] Add a second name for `macos_setup`: `setup_experience`
- [X] Add a second name for `macos_settings`: `apple_settings`
- [X] Add a second name for `custom_settings`: `configuration_profiles`
- [X] Add a second name for `macos_setup_assistant`:
`apple_setup_assistant`
Prior names are deprecated and log warnings. This uses the same
`renameto` tags as previous aliases, and adds code in relevant sections
in gitops.go to run the existing "rename new to old keys" function so
that we can unmarshall into the existing structs (that still have their
`json` tags set to the old key names until Fleet 5).
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] Ran current it-and-security GitOps files successfully locally
(removing mdm stuff that wouldn't work for me locally, but wasn't
relevant to the updated keys
- [X] Run same files successfully after changing the deprecated key
names to their new aliases
- [X] Verified that new keys show up in API responses:
<img width="506" height="243" alt="image"
src="https://github.com/user-attachments/assets/db1eb522-a702-4d17-b313-81ca203632b6"
/>
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
n/a
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduces new configuration key aliases: apple_settings (macOS),
configuration_profiles (profiles for macOS/Windows/Android),
setup_experience (macOS setup), and apple_setup_assistant (macOS setup
assistant).
* Old configuration keys remain supported for backward compatibility;
tooling and generated controls will accept either the new or legacy
names.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40911
# Details
Updates some GitOps error messages to make them 1) use "report" instead
of query where applicable and 2) be more helpful by including filename
and path and not being confusing.
These IMO don't need to be cherry-picked to 4.82 since users won't be
getting deprecation warnings yet so the new error might actually be
_more_ confusing in this case, but I encountered them while working on
the "validate unknown keys" ticket and they looked really bad, so fixing
before I forget.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] Change "query name is required for each query" to "name is
required for each report in {filename} at {path}"
- [X] Change "query SQL query is required for each query" to "query is
required for each report in {filename} at {path}"
- [X] Change "query name must be in ASCII: {name}" to "name must be in
ASCII: {name} in {filename} at {path}"
- [X] Change "duplicate query names: {names}" to "duplicate report
names: {names}
Tested all in both main file and in a file included via `path:`
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40900
# Details
This PR makes the `secrets:` key under the top-level `org_settings` (for
default.yml) or `settings:` (for fleet .yml files) optional. Omitting
the key causes any enroll secrets present on the server to be retained.
There is more to the parent story that will require more design, but I
am getting this one out early because:
1. Our updated it-and-security files will not have `secrets:` and
2. This is not a breaking change, since currently omitting this key
results in a fatal error, _not_ the removal of all secrets (that
requires specifying an empty `secrets:` key)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [x] Using `secrets:` with correct syntax in `defaults.yml` updated
global secrets
- [x] Using `secrets:` with no value in `defaults.yml` removed all
global secrets
- [x] Omitting `secrets:` in `defaults.yml` retained all global secrets
- [x] Using `secrets:` with correct syntax in a fleet .yml file updated
that fleet's secrets
- [x] Using `secrets:` with no value in in a fleet .yml file removed
that fleet's secrets
- [x] Omitting `secrets:` in in a fleet .yml file retained that fleet's
secrets
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40302
# Details
This PR adds support for a `paths:` key for scripts declared under
`controls:` in a GitOps fleet file. If supplied, `paths:` must contain a
"glob" expression (as [supported by the doublestar
package](https://github.com/bmatcuk/doublestar?tab=readme-ov-file#patterns)).
The existing `path:` key still works but may not contain glob
expressions. When a `paths:` key is encountered, we expand it and add
all matching valid (as in, `.sh` or `.ps1`) files to the set of script
files to process.
Subsequent PRs will add this functionality to other entities that use
`path:` (such as reports and policies).
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
Tried with various combinations of `*` and `**` in gitops runs, and
mixing of `path:` and `paths:`
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #40433
# Details
This PR updates `fleetctl gitops` and `fleetctl generate_gitops` to use
`unassigned.yml` in place of `no-team.yml`. The two files are utilized
identically, except that `unassigned.yml` expects the `name:` to be
`Unassigned` rather than `No team`.
Internally, we still map some things to the string "no team" before
sending to the back end so that we don't have to update back-end code
and make more spaghetti to clean up when we 🔪 No Team as a concept in
Fleet 5.
We do pass the filename into the main `DoGitOps` method, but both I and
Claude did our best to determine that it's not used in any way that
would break with this change.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
From test plan:
- [X] With a pre-existing GitOps folder w/ `no-team.yml`, run `fleetctl
gitops -f /path/to/no-team.yml --dry-run
--enable-log-topics=deprecated-field-names` and verify that everything
works as expected and you get the deprecation warning.
- [X] Do the above without `--dry-run` and verify via the UI and/or
`fleetctl generate-gitops` that the Fleet config is as expected.
- [X] Change `no-team.yml` to `unassigned.yml`, try a gitops run with
`fleetctl gitops -f /path/to/unassigned.yml --dry-run
--enable-log-topics=deprecated-field-names` and verify that you get an
error because the `name:` is still `No team`
- [X] Change the `name:` to `Unassigned`, repeat the run above and
verify that the output is the same as with `no-team.yml`, and that no
deprecation warning is listed.
- [X] Do the same as the above without `--dry-run` and verify that the
Fleet config is as expected.
- [X] Run `fleetctl generate-gitops` and verify that `unassigned.yml` is
output rather than `no-team.yml`, and any related files are under the
`lib/unassigned` folder rather than `lib/no-team`, and any paths inside
`unassigned.yml` (e.g. for scripts) are pointed at `lib/unassigned`.
## New Fleet configuration settings
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
@noahtalerman will work on this
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](da43bf8371/server/service/endpoint_utils.go (L108))
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39265
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34376
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
**Related issue:** Resolves#35460, #35462
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added certificate templates for managing Android device certificates
at global and team levels
* Introduced API endpoints to create, list, retrieve, and delete
certificate templates
* Enabled GitOps workflow support for certificate template
specifications
* Implemented automatic variable substitution in certificate subjects
for host identifiers
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Scott Gress <scottmgress@gmail.com>
Co-authored-by: Scott Gress <scott@fleetdm.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35307
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
Fixes#34066. Included a changes file because this catches fields other
than software icon.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
Fixes#31897.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [ ] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- GitOps now supports software icons: generate and include icon
files/paths in specs for packages and App Store apps.
- CLI adds flags to control concurrent icon uploads/updates.
- Icons are uploaded, updated, or deleted automatically during GitOps
runs.
- UI YAML modal now includes icon_url and offers icon download.
- Improvements
- Robust path resolution for icon assets across specs.
- Non-YAML outputs handle both string and byte file contents.
- Bug Fixes
- Removes stale icons after App Store app re-association.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Scott Gress <scottmgress@gmail.com>
Co-authored-by: Scott Gress <scott@fleetdm.com>
Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
For #30849.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [n/a] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
For #30095.
#32482 is additional cleanup. Merging this to unblock orchestration
Linux setup experience work. Code has already been reviewed prior to
merging into the feature branch.
---------
Co-authored-by: Konstantin Sykulev <konst@sykulev.com>
Co-authored-by: Anthony Maxwell <133805840+Illbjorn@users.noreply.github.com>
- Added Jira and Zendesk integrations for "No team". (These are not
supported by GitOps for teams)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [x] Setting(s) is/are explicitly excluded from GitOps
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Default (No Team) responses now include limited integrations (Jira,
Zendesk).
- You can configure or clear Jira/Zendesk integrations for the Default
(No Team) settings.
- Bug Fixes
- More consistent handling of the Default (No Team) when fetching team
details.
- Improved validation to prevent conflicting automation settings between
webhooks and integrations.
- Documentation
- Clarified that Jira/Zendesk integrations aren’t supported via GitOps
or at the team level (including No Team).
- Noted that certain options (e.g., Google Calendar, Conditional Access)
aren’t supported for the Default (No Team).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Fixes#32060
This PR adds:
- new default_team_config_json table
- caching of config from that table, including deep copy methods -- all
of this is not absolutely needed for this change since we are only using
`webhook_settings.failing_policies_webhook` here but added for
completeness/future
- teams/0 API updates
- GitOps updates
- generate gitops updates
Future PRs will add:
- ticket automation
- primo mode migration
- frontend changes
- documentation
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked table schema to confirm autoupdate
## New Fleet configuration settings
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Configure failing-policy webhooks for “No team” via GitOps
(no-team.yml) and API, including enable/disable, destination URL, policy
IDs, and batch size; settings clear when omitted.
- GitOps and CLI now read/apply the real “No team” settings with dry-run
support.
- Policy automation evaluates hosts without a team and triggers “No
team” webhooks when applicable.
- GET/PATCH team 0 returns/accepts a minimal, webhook-focused config.
- Chores
- Added persistence and caching for the default “No team” configuration.
- Introduced a database table to store the default configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Also removed the automatic install flag on YAML FMAs as it's
undocumented/unspec'd
Fixes#25636.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
For #31063
# Details
This PR adds the `RequireBitLockerPIN` config to app-wide and team
configs. This maps to a new `windows_require_bitlocker_pin` JSON field
for gitops and `fleetctl apply`.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
* Will add changelog when feature is complete
- For new Fleet configuration settings
- [X] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. If managing
via Gitops:
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Added the setting to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
* Will add to docs when feature is complete
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
* No UI yet
- [X] Manual QA for all new/changed functionality
* Tested No Team and team config via Postman API calls
* Tested Gitops for no-team and team YML files using `fleetctl`
* Tested `fleetctl generate-gitops`
Fixes#30435.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
# Details
This PR adds support for a new label membership type, `host_vitals`.
Membership for these labels is based on a database query created from
user-supplied criteria. In this first iteration, the allowed criteria
are very simple: a label can specify either an IdP group or IdP
department, and hosts with linked users with a matching group or
department.
Groundwork is laid here for more complex host vitals queries, including
`and` and `or` logic, different data types and different kinds of vitals
(rather than just the "foreign" vitals of which IdP is an example).
Note that this PR does _not_ include the cron job that will trigger
membership updating, and it doesn't include ; for sake of simplicity in
review that will be done in a follow-on PR.
## Basic flow
### Creating a host vitals label
1. A new label is created via the API / GitOps with membership type
`host_vitals` and a `criteria` property that's a JSON blob. Currently
the JSON can only contain `vital` and `value` keys (and must contain
those keys)
2. The server validates that the specified `vital` exists in our [set of
known host
vitals](https://github.com/fleetdm/fleet/pull/30278/files#diff-b6d4c48f2624b82c2567b2b88db1de51c6b152eeb261d40acfd5b63a890839b7R418-R436).
3. The server validates that the [criteria can be parsed into a
query](https://github.com/fleetdm/fleet/pull/30278/files?diff=unified&w=1#diff-4ac4cfba8bed490e8ef125a0556f5417156f805017bfe93c6e2c61aa94ba8a8cR81-R86).
This also happens during GitOps dry run.
4. The label is saved (criteria is saved as JSON in the db)
### Updating membership for a host vitals label
1. The label's criteria is used to generate a query to run on the
_Fleet_ db.
1. For each vital criteria, check the vital type. Currently only foreign
vitals are supported.
2. For foreign vitals, add its group to a set we keep track of.
3. Add a `WHERE` clause section for the vital and value, e.g.
`end_user_idp_groups = ?`
4. Once we have all the `WHERE` clauses, create the query as `SELECT %s
FROM %s` + any joins contributed by foreign vitals groups + `WHERE ` +
all the `WHERE` clauses we just calculated. The `%s` provide some
flexibility if we want to use these queries in other contexts.
2. Delete all existing label members
3. Do an `INSERT...SELECT` using the query we calculated from the label
criteria. The query will be `SELECT <label id> as label_id, hosts.id
FROM hosts JOIN ...`
## Future work
### Domestic vitals
These can be anything that we already store in the `hosts` table.
Domestic vitals won't add any `JOIN`s to the calculated label query, and
will simply be e.g. `hosts.hostname = ?`
### Custom vitals
We currently support an `additional_queries` config that will cause
other queries to run on hosts. The data returned from these queries is
stored in a `hosts_additional` table as a JSON blob. We can use MySQL
JSON functions to match values in this data, e.g.
`JSON_EXTRACT(host_additional, `$.some_custom_vital`) = ?`
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
> I'll add the changelog item when I add the cron job PR
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] If database migrations are included, checked table schema to
confirm autoupdate
- For new Fleet configuration settings
- [X] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. If managing
via Gitops:
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
For #28215
Allows users to use fleet secret variables for macos setup script for
gitops.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [ ] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
https://github.com/fleetdm/fleet/issues/24469
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #27476
# Checklist for submitter
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
# Details
This PR adds a new command `generate-gitops` to the `fleetctl` tool. The
purpose of this command is to output GitOps-ready files that can then be
used with `fleetctl-gitops`.
The general usage of the command is:
```
fleectl generate-gitops --dir /path/to/dir/to/add/files/to
```
By default, the outputted files will not contain sensitive data, but
will instead add comments where the data needs to be replaced by a user.
In cases where sensitive data is redacted, the tool outputs warnings to
the user indicating which keys need to be updated.
The tool uses existing APIs to gather data for use in generating
configuration files. In some cases new API client methods needed to be
added to support the tool:
* ListConfigurationProfiles
* GetProfileContents
* GetScriptContents
* GetSoftwareTitleByID
Additionally, the response for the /api/latest/fleet/software/batch
endpoint was updated slightly to return `HashSHA256` for the software
installers. This allows policies that automatically install software to
refer to that software by hash.
Other options that we may or may not choose to document at this time:
* `--insecure`: outputs sensitive data in plaintext instead of leaving
comments
* `--print`: prints the output to stdout instead of writing files
* `--key`: outputs the value at a keypath to stdout, e.g. `--key
agent_options.config`
* `--team`: only generates config for the specified team name
* `--force`: overwrites files in the given directory (defaults to false,
which errors if the dir is not empty)
# Technical notes
The command is implemented using a `GenerateGitopsCommand` type which
holds some state (like a list of software and scripts encountered) as
well as a Fleet client instance (which may be a mock instance for tests)
and the CLI context (containing things like flags and output writers).
The actual "action" of the CLI command calls the `Run()` method of the
`GenerateGitopsCommand` var, which delegates most of the work to other
methods like `generateOrgSettings()`, `generateControls()`, etc.
Wherever possible, the subroutines use reflection to translate Go struct
fields into JSON property names. This guarantees that the correct keys
are written to config files, and protects against the unlikely event of
keys changing.
When sensitive data is encountered, the subroutines call `AddComment()`
to get a new token to add to the config files. These tokens are replaced
with comments like `# TODO - Add your enrollment secrets here` in the
final output.
# Known issues / TODOs:
* The `macos_setup` configuration is not output by this tool yet. More
planning is required for this. In the meantime, if the tool detects that
`macos_setup` is configured on the server, it outputs a key with an
invalid value and prints a warning to the user that they'll need to
configure it themselves.
* `yara_rules` are not output yet. The tool adds a warning that if you
have Yara rules (which you can only upload via GitOps right now) that
you'll have to migrate them manually. Supporting this will require a new
API that we'll have to discuss the authz for, so punting on it for now.
* Fleet maintained apps are not supported by GitOps yet (coming in
https://github.com/fleetdm/fleet/issues/24469). In the meantime, this
tool will output a `fleet_maintained_apps` key and trigger a warning,
and GitOps will fail if that key is present.
---------
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
