# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- [x] Added/updated automated tests
- [x] Where appropriate, automated tests simulate multiple hosts and
test for host isolation (updates to one hosts's records do not affect
another.)
- [x] Manual QA for all new/changed functionality
#30461
This PR contains the changes for the happy path.
On a separate PR we will be adding tests and further fixes for edge
cases.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Make sure fleetd is compatible with the latest released version of
Fleet (see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md)).
- [ ] Orbit runs on macOS, Linux and Windows. Check if the orbit
feature/bugfix should only apply to one platform (`runtime.GOOS`).
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for using a TPM-backed key and SCEP-issued certificate
to sign HTTP requests, enhancing security through hardware-based key
management.
* Introduced new CLI and environment flags to enable TPM-backed client
certificates for Linux packages and Orbit.
* Added a local HTTPS proxy that automatically signs requests using the
TPM-backed key.
* **Bug Fixes**
* Improved cleanup and restart behavior when authentication fails with a
host identity certificate.
* **Tests**
* Added comprehensive tests for SCEP client functionality and TPM
integration.
* **Chores**
* Updated scripts and documentation to support TPM-backed client
certificate packaging and configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#30475
# Checklist for submitter
- [x] Manual QA for all new/changed functionality
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced support for HTTP message signing in agent-server
communications, enhancing request authentication.
* Added a configurable option to control the probability of agents using
HTTP message signatures via a new command-line flag.
* **Bug Fixes**
* Improved error logging for issues encountered during HTTP signature
key retrieval, providing better visibility into failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
> Closes#31020
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Added/updated automated tests
- [x] Where appropriate, automated tests simulate multiple hosts and
test for host isolation (updates to one hosts's records do not affect
another.)
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
Fixes#27758.
<img width="807" height="303" alt="image"
src="https://github.com/user-attachments/assets/58e5b9bc-42d6-4195-868e-bf6206ec9cd5"
/>
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #30947
# Checklist for submitter
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for requiring HTTP message signatures for fleetd
requests, configurable via a new setting.
* Enhanced middleware to enforce HTTP message signature requirements
when enabled.
* **Tests**
* Introduced integration tests to verify host identity endpoints enforce
HTTP message signature requirements.
* Updated test utilities and suite setup to support configurable
signature enforcement.
* **Chores**
* Refactored configuration and test server options to support the new
signature enforcement feature.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#30473
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Added/updated automated tests
- [ ] Manual QA for all new/changed functionality
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for TPM-backed host identity certificates enabling
hardware-backed HTTP signature authentication for hosts.
* Introduced HTTP signature verification middleware for API requests,
applied conditionally for premium licenses.
* Hosts presenting identity certificates must authenticate with matching
HTTP message signatures during enrollment and authentication.
* Added SCEP-based certificate issuance for secure host identity
management.
* Updated enrollment endpoints to use standardized request/response
contract types.
* **Bug Fixes**
* Enhanced authentication logic to verify consistency between host
identity certificates and host records, preventing duplicate or
mismatched identities.
* **Chores**
* Updated dependencies and test infrastructure to support HTTP signature
verification and host identity certificate workflows.
* Added comprehensive integration and datastore tests for host identity
certificate issuance, storage, and authentication.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Profile status respect remove operation for pending and failed status,
where the declarations did not, meaning the host would show up with a
wrong status if only a declaration was pending or failed removal.
This was also affecting the `os_setting` api filter option for list
hosts (maybe elsewhere), which is also fixed by this change.
A part of #29824
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
Fixes#29849.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
Fixes#30473
Refactore Datastore.EnrollHost and Datastore.EnrollOrbit methods to use
functional options. Doing this refactor before adding new options to
those methods. This should make the code more maintainable and easier to
understand.
No functional changes here. Just refactoring.
# Checklist for submitter
- [x] Added/updated automated tests
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Streamlined host and Orbit enrollment methods to use a flexible
options-based pattern instead of fixed parameter lists.
* Updated related tests and service logic to use the new options
approach, improving clarity and extensibility for enrollment operations.
* **New Features**
* Introduced configuration options for host and Orbit enrollment,
allowing more explicit and customizable parameter setting during
enrollment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes 30636
I am adding a handful of additional unit tests but this is ready for
review now. Integrates changes from Victor's PoC for Account Driven User
Enrollment including a nice end to end integration test including the
SAML portion
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> Closes https://github.com/fleetdm/fleet/issues/30678
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
Fixes#30458
Contributor docs PR: https://github.com/fleetdm/fleet/pull/30651
# Checklist for submitter
- We will add changes file later.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- Did not do manual QA since the SCEP client I have doesn't support ECC.
Will rely on next subtasks for manual QA.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced Host Identity SCEP (Simple Certificate Enrollment Protocol)
support, enabling secure host identity certificate enrollment and
management.
* Added new API endpoints for Host Identity SCEP, including certificate
issuance and retrieval.
* Implemented MySQL-backed storage and management for host identity SCEP
certificates and serials.
* Added new database tables for storing host identity SCEP certificates
and serial numbers.
* Provided utilities for encoding certificates and keys, and handling
ECDSA public keys.
* **Bug Fixes**
* None.
* **Tests**
* Added comprehensive integration and unit tests for Host Identity SCEP
functionality, including certificate issuance, validation, and error
scenarios.
* **Chores**
* Updated test utilities to support unique test names and new SCEP
storage options.
* Extended mock datastore and interfaces for new host identity
certificate methods.
* **Documentation**
* Added comments and documentation for new SCEP-related interfaces,
methods, and database schema changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Mocking the manifest server to ensure that an app with a no_check hash
is always returned (in the event that chrome no longer has this property
about it)
Mocking the file content in the integration test. There is no reason to
really download the real 1password or chrome binaries for this
particular case.
For: https://github.com/fleetdm/fleet/issues/30325
- [x] Added/updated automated tests
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Enhanced test coverage for software installer payloads with simulated
manifest endpoints and SHA256 hash validations.
* Added mock servers to verify installer downloads and hash computations
for maintained apps, improving batch installer processing tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fix for #23899.
I found this while preparing the demo (I had a user with department and
no groups).
Assigning to @getvictor because Scott is OOO.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved host filtering for Identity Provider (IdP) group labels to
include hosts without corresponding group entries.
* **Tests**
* Enhanced label tests to cover IdP department labels, including
scenarios where users have no groups.
* Added new subtests to verify correct host inclusion and label counts
for department-based labels.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
For https://github.com/fleetdm/confidential/issues/9931.
[Here](ec3e8edbdc/docs/Contributing/Testing-and-local-development.md (L339))'s
how to test SAML locally with SimpleSAML.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved SSO and SAML integration with enhanced session management
using secure cookies.
* Added support for IdP-initiated login flows.
* Introduced new tests covering SSO login flows, metadata handling, and
error scenarios.
* **Bug Fixes**
* Enhanced validation and error handling for invalid or tampered SAML
responses.
* Fixed session cookie handling during SSO and Apple MDM SSO flows.
* **Refactor**
* Replaced custom SAML implementation with the crewjam/saml library for
improved reliability.
* Simplified SAML metadata parsing and session store management.
* Streamlined SSO authorization request and response processing.
* Removed deprecated fields and redundant code related to SSO.
* **Documentation**
* Updated testing and local development docs with clearer instructions
for SSO and IdP-initiated login.
* **Chores**
* Upgraded dependencies including crewjam/saml and related packages.
* Cleaned up tests and configuration by removing deprecated fields and
unused imports.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
relates to [#28691](https://github.com/fleetdm/fleet/issues/28691)
This fixes an issue where `end_user_license_agreeement` is defined but
is an empty string or nil.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved handling of the end user license agreement (EULA) field in
app configuration to ensure it is always set correctly and removed from
the configuration map after processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fixes: #30325
Related to incorrect behavior introduced at
https://github.com/fleetdm/fleet/pull/28945
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* When uploading software batches, if the installer SHA is set to
"no_check," the system will now automatically generate and use the
SHA256 checksum of the installer file.
* **Bug Fixes**
* Fixed an issue ensuring the latest Google Chrome version is pulled
during Fleet-maintained app updates.
* Corrected the display of the SHA256 hash in the UI and API to show
valid values.
* Improved handling of installer uploads to ensure a valid SHA256
checksum is always applied, even when "no_check" is specified.
* **Tests**
* Added a test to verify correct SHA256 hash calculation for installer
files.
* Extended integration tests to validate batch software installer
operations for maintained apps with SHA256 hash checks.
* Added tests covering behavior when SHA256 checksum is marked as
"no_check" for maintained apps.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
> Fixes#29980
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved support for iOS VPP app installations, including enhanced
verification and activity logging.
* Hosts now automatically request a software data update after
successful app installs.
* Installation status now includes "Installing" state for better
tracking.
* **Tests**
* Expanded integration tests to cover iOS VPP app installations
alongside macOS.
* Added checks to verify refetch requests and correct MDM command
behavior after app installs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
This was added to support the "All Software" when listing software on
the host.
Fixes#30188
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an option to explicitly exclude software available for install
from host software listings.
* **Bug Fixes**
* Improved accuracy of software inventory results when filtering by
availability for install.
* **Tests**
* Added a test to verify exclusion of available-for-install software
when the relevant option is set.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
Fixes#29916.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Host vitals data now refreshes automatically after successful software
installation or uninstallation, ensuring up-to-date status information.
* **Tests**
* Enhanced tests to verify that host vitals are only refreshed after
successful software changes, improving reliability and accuracy of the
system’s behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Maps are reference types in Go, so the called function can modify the
map even when it is passed by value (because the pointer is passed by
value, pointing to the same underlying map). This simplifies the code.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Added/updated automated tests
- [ ] Manual QA for all new/changed functionality
For #30385.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
> Fixes ##28738
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #29867 . Includes latest copy requested by product.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
for #30502
# Details
This PR fixes an issue where `fleetctl generate-gitops` would not always
add a `macos_setup` setting to a .yml file even if the team had a setup
experience configured. This was due to relying on the `MacOSSetup`
config returned by app/team config APIs to have this data populated,
which turned out to be an incorrect assumption. Instead, we now utilize
various APIs to check for the presence of setup software, scripts,
bootstrap packages and profiles.
Note that for now, `generate-gitops` will only output a `TODO` line if
setup experience is detected;
https://github.com/fleetdm/fleet/issues/30210 is open to flesh this out.
In the meantime `fleetctl gitops` will fail if this TODO is inserted, so
that the user must go and fix it manually.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
# Testing
I set up MDM on a local instance and tried the following both on No Team
and a regular team:
* Turned "End user authentication on", verified that `fleetctl
generate-gitops` output a `macos_setup` setting for the team. Turned it
back off and verified that `macos_setup` was no longer exported by
`fleetctl generate-gitops`.
* Did the same for bootstrap package.
* Did the same for install software, and additionally verified that
having software available but _not_ selected did not cause `macos_setup`
to be exported. Same for teams with no software available at all.
* Did the same for setup assistant.
I also tested that changes to No Team didn't affect the output when
exporting a regular team.
---------
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
relates to [#28691](https://github.com/fleetdm/fleet/issues/28691)
This adds the ability to upload the EULA users see during the setup
experience via gitops. It follows patterns used for uploading the
bootstrap package via gitops.
I've also added a sha256 column to the `eulas` table in order to easily
compare the existing eula with a new one to see if we need to perform an
upload.
Finally I added the support to generate this new gitops setting with the
`generate-gitops` command
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For new Fleet configuration settings
- [x] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. If managing
via Gitops:
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Added the setting to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
- For database migrations:
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> Fixes#29851
> Fixes#29902
> Mainly followups from https://github.com/fleetdm/fleet/pull/30295,
plus improved integration testing
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
#29482
[Migrate to the AWS SDK for Go
v2](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/migrate-gosdk.html)
documents how to migrate codebases.
QA on features that use AWS SDK Go:
- Bootstrap package:
- upload: ✅
- download: ✅
- cleanup: ✅
- Software (upload, download, installation, etc.) ✅
- Cloudfront: Luckly, this feature was already using aws-sdk-go-v2.
- Carves ✅
- Logging:
- Firehose ✅
- Kinesis ✅
- Lambda ✅ (tested result logs to a lambda function on our AWS Dogfood
account)
- Email:
- Amazon SES TODO ⚠️ (this is what Dogfood uses and a few customers)
- We cannot easily test locally, we can use dogfood or load testing
(AWS) environments.
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Manual QA for all new/changed functionality
For #29609.
From @marko-lisica:
> In #23900 we decided to error if $FLEET_VAR_HOST_END_USER_IDP_GROUPS
or $FLEET_VAR_HOST_END_USER_IDP_USERNAME is added to a profile and the
host doesn't have a username or groups assigned to it. We decided that
because we think that if user deploys profile with variables above then
user would expected that host have some value for those. I think we
should be consistent with departments.
Context:
https://fleetdm.slack.com/archives/C084F4MKYSJ/p1751281745736999?thread_ts=1751025745.430259&cid=C084F4MKYSJ.
# Details
This PR adds support for a new label membership type, `host_vitals`.
Membership for these labels is based on a database query created from
user-supplied criteria. In this first iteration, the allowed criteria
are very simple: a label can specify either an IdP group or IdP
department, and hosts with linked users with a matching group or
department.
Groundwork is laid here for more complex host vitals queries, including
`and` and `or` logic, different data types and different kinds of vitals
(rather than just the "foreign" vitals of which IdP is an example).
Note that this PR does _not_ include the cron job that will trigger
membership updating, and it doesn't include ; for sake of simplicity in
review that will be done in a follow-on PR.
## Basic flow
### Creating a host vitals label
1. A new label is created via the API / GitOps with membership type
`host_vitals` and a `criteria` property that's a JSON blob. Currently
the JSON can only contain `vital` and `value` keys (and must contain
those keys)
2. The server validates that the specified `vital` exists in our [set of
known host
vitals](https://github.com/fleetdm/fleet/pull/30278/files#diff-b6d4c48f2624b82c2567b2b88db1de51c6b152eeb261d40acfd5b63a890839b7R418-R436).
3. The server validates that the [criteria can be parsed into a
query](https://github.com/fleetdm/fleet/pull/30278/files?diff=unified&w=1#diff-4ac4cfba8bed490e8ef125a0556f5417156f805017bfe93c6e2c61aa94ba8a8cR81-R86).
This also happens during GitOps dry run.
4. The label is saved (criteria is saved as JSON in the db)
### Updating membership for a host vitals label
1. The label's criteria is used to generate a query to run on the
_Fleet_ db.
1. For each vital criteria, check the vital type. Currently only foreign
vitals are supported.
2. For foreign vitals, add its group to a set we keep track of.
3. Add a `WHERE` clause section for the vital and value, e.g.
`end_user_idp_groups = ?`
4. Once we have all the `WHERE` clauses, create the query as `SELECT %s
FROM %s` + any joins contributed by foreign vitals groups + `WHERE ` +
all the `WHERE` clauses we just calculated. The `%s` provide some
flexibility if we want to use these queries in other contexts.
2. Delete all existing label members
3. Do an `INSERT...SELECT` using the query we calculated from the label
criteria. The query will be `SELECT <label id> as label_id, hosts.id
FROM hosts JOIN ...`
## Future work
### Domestic vitals
These can be anything that we already store in the `hosts` table.
Domestic vitals won't add any `JOIN`s to the calculated label query, and
will simply be e.g. `hosts.hostname = ?`
### Custom vitals
We currently support an `additional_queries` config that will cause
other queries to run on hosts. The data returned from these queries is
stored in a `hosts_additional` table as a JSON blob. We can use MySQL
JSON functions to match values in this data, e.g.
`JSON_EXTRACT(host_additional, `$.some_custom_vital`) = ?`
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
> I'll add the changelog item when I add the cron job PR
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] If database migrations are included, checked table schema to
confirm autoupdate
- For new Fleet configuration settings
- [X] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. If managing
via Gitops:
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
#29609
Verified the changes with [Entra ID's
validator](https://scimvalidator.microsoft.com/) and adding the
department attribute to the tester:
<img width="1312" alt="Screenshot 2025-06-27 at 8 54 32 AM"
src="https://github.com/user-attachments/assets/45a5deb8-7c65-49df-b3e8-eb05bea11f6b"
/>
<img width="1312" alt="Screenshot 2025-06-27 at 8 54 21 AM"
src="https://github.com/user-attachments/assets/91b554b5-b0b9-4bb6-a0cf-4e3b40e6ce21"
/>
- Tested with Okta
- TODO: Test with Entra ID and Google Workspace.
- I decided to not fail profile deployment if a user has no department
because it's not a required attribute, instead the
`FLEET_VAR_HOST_END_USER_IDP_DEPARTMENT` will be replaced with the empty
string.
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] If database migrations are included, checked table schema to
confirm autoupdate
(https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- For database migrations:
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
