Fixes#32096
The gist of the fix is that when syncing devices from DEP we save the
migration deadline to our host_dep_assignments table. The next
enrollment, which we assume should be the migration, looks at
host_dep_assignments, sees that mdm_migration_deadline is non-Null and
mdm_migration_completed is NULL, and uses that as the signal that a
migration is in progress and skips enqueuing setup experience items. It
then marks the migration as complete which sets mdm_migration_completed
= mdm_migration_deadline. Once this is set setup experience will run as
normal unless mdm_migration_completed gets set to NULL and/or
mdm_migration_deadline gets set to a value in the future(which e.g.
would happen if the customer assigned to another MDM server then
assigned to migrate to fleet again)
DB test failure is expected here because it won't like the migration
timestamp but that is a necessary failure because this fix is going to
be backported into 4.73
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
## Database migrations
- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* New Features
* Tracks and stores Apple DEP MDM migration deadlines per device/host.
* Detects “migration in progress” during DEP sync and check-in.
* Automatically marks migration complete and skips Setup Assistant items
while migration is in progress to prevent conflicts.
* Bug Fixes
* Improved DEP compatibility by updating the protocol version and
User-Agent used for Apple’s APIs, reducing the chance of blocked or
rejected requests.
* Migrations
* Adds fields to support migration deadlines and completion status (no
action required).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Magnus Jensen <magnus@fleetdm.com>
- Added Jira and Zendesk integrations for "No team". (These are not
supported by GitOps for teams)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [x] Setting(s) is/are explicitly excluded from GitOps
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Default (No Team) responses now include limited integrations (Jira,
Zendesk).
- You can configure or clear Jira/Zendesk integrations for the Default
(No Team) settings.
- Bug Fixes
- More consistent handling of the Default (No Team) when fetching team
details.
- Improved validation to prevent conflicting automation settings between
webhooks and integrations.
- Documentation
- Clarified that Jira/Zendesk integrations aren’t supported via GitOps
or at the team level (including No Team).
- Noted that certain options (e.g., Google Calendar, Conditional Access)
aren’t supported for the Default (No Team).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
for #31518
# Details
This PR updates the jobs package with a new `GetFilteredQueuedJobs`
method which accepts a list of job names to return from the `jobs`
table, rather than returning all jobs matching the other criteria. It
also updates the `worker` package to use this new method instead of
`GetQueuedJobs`.
The purpose of this update is to allow us to add multiple workers which
process different kinds of jobs, while sharing the same `jobs` table and
framework.
I chose to add the new `GetFilteredQueuedJobs` method rather than
updating `GetQueuedJobs` because the latter is used in a bunch of tests
that would need to be updated. `GetQueuedJobs` now calls
`GetFilteredQueuedJobs` with an empty list, which is interpreted as
"return all jobs".
# Checklist for submitter
## Testing
- [X] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
There shouldn't be any functional change here, but if someone has a Jira
or Zendesk integration set up to easily test with, that'd be good.
Fixes#31444
The changes are primarily in tests. The only changes in production code
are a couple validations/checks for invalid values in:
- mysql/apple_mdm.go
- mysql/hosts.go
- mysql/queries.go
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved handling of timestamp and default values across various
features to prevent database errors and warnings.
* Enhanced validation and data consistency for Apple Business Manager
tokens and MDM profiles.
* Updated test data and logic to comply with stricter database
constraints and realistic scenarios, including date handling and field
lengths.
* **Chores**
* Updated test setups to reflect schema changes, improve data integrity,
and avoid future compatibility issues.
* Standardized SQL mode and timestamp usage in test environments.
* Refined test data for VPP apps, software installers, and device
enrollments for better reliability.
* **Tests**
* Expanded and updated tests to cover new fields, stricter validation,
and more accurate simulation of real-world conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR addresses the concern of potentially being able to release a
device before any profile is sent, and the check thinking there is no
pending. It addresses both the release worker, but also the orbit setup
experience endpoint, even though that is less likely.
_Checked the query against my host on dogfood where it took 0.1 seconds,
with the single host._
fixes: #31143
_I also ended up putting my main test in a new file
`integration_mdm_release_worker_test.go` and decided not to do fancy
setup, as there is only one test so no recurring things, and based on
our retro talk also moved the setup experience related tests inside of
`integration_mdm_dep_test.go` into their separate file
`integration_mdm_setup_experience_test.go`_
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually (No, since this one is
hard to reproduce, but instead wrote an integration test before doing
the change to verify the behaviour.)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [ ] Added/updated automated tests
- [ ] Where appropriate, automated tests simulate multiple hosts and
test for host isolation (updates to one hosts's records do not affect
another.)
- [ ] Manual QA for all new/changed functionality
> Closes https://github.com/fleetdm/fleet/issues/30678
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
> Fixes ##28738
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> Fixes#29851
> Fixes#29902
> Mainly followups from https://github.com/fleetdm/fleet/pull/30295,
plus improved integration testing
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- For database migrations:
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #28196
Demo video of the full feature: https://youtu.be/7PM41LBsnig
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #26071 and #26089
Added `macos_setup.manual_agent_install` boolean option. No validation
(pushed to another story due to complications caused by bug #28497)
Tests are failing due to vulnerability issues that are not related to
this feature. All tests were passing earlier.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #27284
This PR:
- Adds SCIM as a fallback for username during macOS end user
authentication during setup experience
- Adds SCIM/endUsers details to host details
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #24869
This subtask contains code to sign the CloudFront software installer and
bootstrap package URL using AWS SDK URL signer.
It works with the current bootstrap package delivery. For software
installers, fleetd will need to be modified to take advantage of this
URL in a future subtask (which will also include updated API contributor
docs).
My article on signed URLs, for context:
https://victoronsoftware.com/posts/cloudfront-signed-urls/
# Checklist for submitter
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
for #21309, this also fixes a few bugs found along the way
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
> Related issue: #9956
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Roberto Dip <rroperzh@gmail.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Dante Catalfamo <43040593+dantecatalfamo@users.noreply.github.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
for #19176
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
`go-kit/kit/log` was deprecated and generating warnings
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Manual QA for all new/changed functionality
for #18427
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [ ] Manual QA for all new/changed functionality
For #19016
This changes all the places where we previously assumed that certs were
hardcoded when the Fleet server started to query the database instead.
The plan is to loadtest afterwards, but as a first preemptive measure,
this adds a caching layer on top the mysql datastore.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
#18119
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Jacob Shandling <61553566+jacobshandling@users.noreply.github.com>
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#18160
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Manual QA for all new/changed functionality
#15557
Following the precedent that Lucas used for other similar PRs, the best
way to review is probably by commits.
* The first one simply copies over the files from the fork to the
monorepo
* Second one adjusts all import paths
* Third one tidies up the `go.mod` files
* Last one fixes the linter issues in the nanomdm package
# Checklist for submitter
- ~~Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~~ (not a user-visible change)
- [x] Manual QA for all new/changed functionality (ran test suite,
re-generated mocks)
I also verified that our Go test suite did run the newly moved `nanomdm`
package steps:
```
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/cryptoutil 0.003s coverage: 0.0% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/mdm 0.005s coverage: 46.2% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/service/certauth 1.320s coverage: 20.7% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/storage/file 0.007s coverage: 24.1% of statements in github.com/fleetdm/fleet/v4/...
```
