For #26519
This PR allows Fleet server to use Android with either fleetdm.com proxy
or locally. It also removes the Android feature flag from the backend.
The frontend changes and proxy API documentation will be in separate
PRs.
Updated contributor docs:
https://github.com/fleetdm/fleet/pull/29880/files
Integration tests are missing and tracked as a separate issue:
https://github.com/fleetdm/fleet/issues/27080
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #29958
Found during 4.69.0 load testing.
We need to call disablePoliciesDurignSetupExperience after calling
GetHostPolicyReportedAt and running the time-based logic which often
results in us not contacting the DB for policies at all(since the time
based logic just hits redis)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
Re-ordering migrations for cherry-picking #27042 into v4.69.0.
Migrations in `main`:
```
[...]
20250519161614_AddExecutableSha256HostSoftwareInstalledPaths.go
20250520153848_AddCancelledStatusToSetupExperienceStatusResults.go
20250528115932_AddInviteIDToUsersTable.go
20250529102706_SetMediumTextColumns.go
20250603105558_AlterLegacyEnrollRefsUUIDSize.go
20250609102714_AddHostCertificateSourcesTable.go
20250609120000_MicrosoftCompliancePartner.go
````
Migrations in rc-minor-fleet-v4.69.0:
```
[...]
20250519161614_AddExecutableSha256HostSoftwareInstalledPaths.go
```
So we have to move this one between
`20250519161614_AddExecutableSha256HostSoftwareInstalledPaths.go` and
`20250520153848_AddCancelledStatusToSetupExperienceStatusResults.go`.
For #21979
Extended POST /api/v1/fleet/hosts/:id/wipe end-point to allow users to
specify an optional payload for specifying what type of remote wipe to
perform on Win hosts.
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
For #27042.
Ready for review, just missing integration tests that I will be writing
today.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] If database migrations are included, checked table schema to
confirm autoupdate
- For new Fleet configuration settings
- [X] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. If managing
via Gitops:
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [X] Added the setting to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: jacobshandling <61553566+jacobshandling@users.noreply.github.com>
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
For #29614.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- For new Fleet configuration settings
- [x] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. - Excluded
from GitOps (env var)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #28846.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
For #26375.
When listing software titles for 'All teams', do not join against
software installers nor vpps to avoid duplicates.
Since filters related to software installers/VPP apps are no longer used
when viewing titles for 'All teams', the filter dropdown is disabled if
'All teams' is selected.
Tests (automated and manual) and contributor API docs will arrive in
another PR
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
for #29443
# Checklist for submitter
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
## Details
This PR adds two new `HostListOptions` for the `ListHosts` method:
* `BatchScriptExecutionIDFilter`: if set, only returns hosts that were
targeted by the specified batch script run
* `BatchScriptExecutionStatus`: if set, further filters hosts by the
status of their script execution in the specified batch
## Testing
I added new automated tests that check:
* Starting a batch script run on 5 hosts and calling `ListHosts` with
only `BatchScriptExecutionIDFilter` set returns all 5 targeted hosts
* Calling `ListHosts` with `BatchScriptExecutionStatus` set to
`pending`, `run`, `errored` or `cancelled` returns the expected set of
hosts
* Simulating various script run outcomes (success, failure, cancel) and
calling `ListHosts` with each status again returns the expected set of
hosts
I also tested manually by executing a batch script in the UI and calling
the API via Postman with each status.
Fixes (partially) #29720.
Split from website changes since this has different reviewers. Assigned
to MDM for review since the non-comment change here touches setup
experience.
- [x] Changes file added to `orbit/changes`
Fixes part of unreleased for #28846.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [ ] Manual QA for all new/changed functionality
Also adds some more rate limiter tests to make sure separate rate limit
buckets interact as expected.
Fixes#29614.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- For new Fleet configuration settings
- [x] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. (excluded;
env var or YAML)
- [x] Added/updated automated tests
- [ ] Manual QA for all new/changed functionality
---------
Co-authored-by: George Karr <georgekarrv@users.noreply.github.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
fixes#29434
This fixes an issue where a dry run on gitops would not show a failure
if a user was trying to upload a non distributed bootstrap package.
- [x] Manual QA for all new/changed functionality
Fixes#28589
Previously "include any" was not being properly honored when we queried
profiles for verification and so we never would actually verify profiles
where the user had an include any rule with multiple labels and the host
included a subset of the labels. Updated the query that returns profiles
to verify for Windows and Apple to return those profiles with "include
any" labels and a nonzero number of the targeted labels applied
Also rearranged and refactored the associated tests slightly and added a
single test that does the various permutations of "include all" "include
any" and "exclude any"
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
Env var: `FLEET_MDM_SSO_RATE_LIMIT_PER_MINUTE`. **Not** managed via
GitOps.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- For new Fleet configuration settings
- [x] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps.
- [ ] Added/updated automated tests
- [ ] Manual QA for all new/changed functionality
> closes#26403
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
Fixes#28261.
~~Of note, this logic will prefer a non-primary CVSSv3.1 score over a
primary CVSSv3.0 score if 3.1 doesn't have primary but 3.0 does. I
haven't seen any evidence of this in our dataset (looked at 2024
output).~~
Updated with logic that will prefer a primary CVSSv3.0 score over a
secondary CVSSv3.1 score for a given vulnerability. In the test dataset
(2023 vuln snapshot, ~20k vulns) there were no cases where this
situation presented itself, so output was identical to the prior
implementation.
Validated by comparing a vulns run from GitHub Actions to a local run
with the new code, and confirmed that existing v3 scores weren't
replaced when they already existed (just got adds of v2 when only v3
existed, and v2/v3 adds when no scoring existed).
Confirmed that all three CVEs mentioned in #28261 show up in feed data.
Added spot-checks for secondary CVSS scores to the feed validator tool.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Manual QA for all new/changed functionality
If the user has a software installed on their machine that happens to
match a vpp app in our catalog. When searching by vulnerable attributes
do not link that software to a vpp app. Just treat it like a non-fleet
installed application.
https://github.com/fleetdm/fleet/issues/29308
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
Temporarily fixes#29570 test failures while we figure out what desired
behavior is and determine how to get that behavior.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Added/updated automated tests
> Closes#28259
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
Still adding tests but wanted to get this up for review of the overall
"shape" of the fix
When certain things happen like installer updates we delete pending
upcoming_activities(UA) and host_software_install(HSI) entries and need
to mark setup_experience_status_results(SESR) cancelled. When this
happens if that UA/HSI are being depended on by setup experience we need
to make sure that that setup experience result eventually gets marked
failed.
I kind of went back and forth a few times on how best to do this and
avoid race conditions. One thing I tried was looking at existence of the
UA/HSI but found that naively just trying to look at that in relation to
the SESR entry seemed to have a few race conditions that were hard to
resolve. There are a few possible states here we need to account for
such as:
un-activated, totally not yet running software install cancelled
activated but not yet running on the host software install cancelled
activated and running on the host software install cancelled before
results are completely reported back
What I eventually came around to was essentially that we want to mark
the SESR cancelled in the same transaction we delete the HSI/UA in. We
then finalize it by marking it failed and sending the activity the next
time the host fetches setupm experience results. The new cancelled
status never leaves fleet. This is a bit ugly but in my testing avoided
the race conditions and works well.
Note that to actually avoid setup experience hanging entirely we still
need to fix#29357 which encompasses several cases where the unified
queue can get completely stuck for a host
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
- [ ] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [ ] Added/updated automated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Make sure fleetd is compatible with the latest released version of
Fleet (see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md)).
- [ ] Orbit runs on macOS, Linux and Windows. Check if the orbit
feature/bugfix should only apply to one platform (`runtime.GOOS`).
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
- [ ] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
# Checklist for submitter
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
## Details
This PR updates the Open Policy Agent (github.com/open-policy-agent/opa)
to version 1.4.2, and does the necessary test updates to handle the
indirect upgrade of Viper (which no longer supports YAML 1.1).
Once this is done we can also [upgrade
NFPM](https://github.com/fleetdm/fleet). I chose not to do that in this
PR to keep it to one change at a time.
