Commit graph

86 commits

Author SHA1 Message Date
Victor Lyuboslavsky
4e7c6f33a7
SyncML <Delete> Windows profiles (#42206)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33418 

Demo video: https://www.youtube.com/watch?v=gtsIYxmIOSo
Docs: https://github.com/fleetdm/fleet/pull/42269/changes

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Windows profiles now send SyncML <Delete> commands when profiles are
removed or hosts change teams, ensuring profile settings are removed
from devices like on macOS.
* Deletion is handled as a two-phase flow: pending removals are enqueued
and tracked instead of being immediately deleted.

* **Tests**
* Added/updated tests for delete-command generation, remove-status
mappings, and end-to-end removal reconciliation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-03-26 18:25:54 -05:00
Victor Lyuboslavsky
92bc1c650e
Move PostJSONWithTimeout to platform/http package and activity cleanup (#40561)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #38536

- Moved PostJSONWithTimeout to platform/http
- Created platform/errors package with only types needed by ctxerr. This
way, ctxerr did not need to import fleethttp.
- Made activity bounded context use PostJSONWithTimeout directly
- Removed some activity types from legacy code that were no longer
needed

# Checklist for submitter

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes file `38536-new-activity-bc` already present, and this is just
cleanup from that work.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **Refactor**
* Reorganized error handling utilities for improved clarity and
decoupling.
* Consolidated HTTP utilities to centralize JSON posting functionality
with timeout support.
* Simplified activity service initialization by removing unused internal
parameters.
* Cleaned up test utilities and removed webhook-related test
scaffolding.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-26 17:39:10 -06:00
Nico
e8152e53fc
Log response body in PostJSONWithTimeout error case (#40509)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
# Checklist for submitter


- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] QA'd all new/changed functionality manually
2026-02-25 15:35:29 -06:00
Victor Lyuboslavsky
913a5904c8
Move NewActivity to activity bounded context (#39521)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #38536 

This PR moves all logic to create new activities to activity bounded
context.
The old service and ActivityModule methods are not facades that route to
the new activity bounded context. The facades will be removed in a
subsequent PR.

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added webhook support for activity events with configurable endpoint
and enable/disable settings.
* Enhanced automation-initiated activity creation without requiring a
user context.
* Improved activity service architecture with centralized creation and
management.

* **Improvements**
* Refactored activity creation to use a dedicated service layer for
better separation of concerns.
* Added support for host-specific and automation-originated activities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-25 14:11:03 -06:00
Victor Lyuboslavsky
ccc36a9cb3
Finishing mysql package migration to slog (#40350)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #40054

# Checklist for submitter

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
  - Already present in previous PR

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Migrated logging to a structured, context-aware backend for clearer,
richer diagnostics and consistent log formatting.
* Introduced broader context propagation and adjusted internal
interfaces to support the new logging approach (no end-user behavior
changes).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-24 16:52:36 -06:00
Victor Lyuboslavsky
6b3bb8a961
slog migration: platform/mysql and related logic (#40072)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #40054 

# Checklist for submitter

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
  - already included in previous PR

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Refactor**
* Standardized logging infrastructure across the database and storage
layers for improved consistency and maintainability.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-19 08:27:24 -06:00
Victor Lyuboslavsky
abaeeec6b8
Change Datastore.logger type to *logging.Logger (#39938)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #38889

This is preparatory work before incrementally converting datastore/mysql
files to directly use *slog.Logger.
This will be done by using `logger.SlogLogger()` to get the underlying
`*slog.Logger`

# Checklist for submitter

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
  - Changes file already exists from previous PR

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated internal logging infrastructure to use a standardized platform
logging package across database and utility components. This
consolidates logging dependencies and improves system consistency
without affecting user-facing functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-17 15:29:52 -06:00
Victor Lyuboslavsky
988f0bbcc6
Transitioned activity bounded context to slog. (#39773)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #38889 

# Checklist for submitter

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
  - Changes file already exists

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated internal logging infrastructure across activity service
modules to enhance code consistency and maintainability.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-16 14:23:36 -06:00
Victor Lyuboslavsky
8f1cc5f90f
Fix issue where MySQL read replicas were not using TLS. (#39689)
<!-- Add the related story/sub-task/bug number, like Resolves
fleetdm/fleet#123, or remove if NA -->
**Related issue:** Resolves fleetdm/confidential#14268 

The root cause was that the poorly named `checkConfig(replicaConf)` was
mutating the config without it being obvious to developers.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Fixed an issue where MySQL read replica connections were not secured
with TLS. Read replicas now properly use TLS encryption for secure
database connections.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-11 12:08:21 -06:00
Victor Lyuboslavsky
61f635dd44
Activity bounded context: Complete read operations (#38555)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #38534

moved `/api/_version_/fleet/hosts/{id:[0-9]+}/activities` endpoint and
`MarkActivitiesAsStreamed` to activity bounded context

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added new endpoint to retrieve host-specific past activities with
pagination metadata.
  
* **Refactor**
* Refactored activity service architecture and authorization layer to
improve data provider integration and activity streaming capabilities.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-09 15:29:12 -06:00
Jonathan Katz
62051cd58a
Use tablewriter package in DumpTable util (#38801)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #
Adds `olekukonko/tablewriter` to the the mysql.DumpTable testing util,
for easier reading of tables with many columns.
Could potentially look worse in some situations, but in my opinion adds
more clarity overall.

### Example
```
    testing_utils.go:519: >> dumping table in_house_apps:
    testing_utils.go:533: [id title_id team_id global_or_team_id filename version storage_id created_at updated_at platform bundle_identifier self_service url]
    testing_utils.go:546: 1	1	1	1	foo.ipa	1.2.3	testingtesting123	2026-01-26T19:21:19Z	2026-01-26T19:21:19Z	ipados	com.foo	0
    testing_utils.go:546: 2	2	1	1	foo.ipa	1.2.3	testingtesting123	2026-01-26T19:21:19Z	2026-01-26T19:21:19Z	ios	com.foo	0
    testing_utils.go:549: << dumping table in_house_apps completed
    testing_utils.go:519: >> dumping table software_titles:
    testing_utils.go:533: [id name source extension_for bundle_identifier additional_identifier is_kernel application_id unique_identifier upgrade_code]
    testing_utils.go:546: 1	foo	ipados_apps		com.foo	2	0	NULL	com.foo	NULL
    testing_utils.go:546: 2	foo	ios_apps		com.foo	1	0	NULL	com.foo	NULL
    testing_utils.go:549: << dumping table software_titles completed
```

```
    testing_utils.go:521: >> dumping table in_house_apps:
    testing_utils.go:569:
        +----+----------+---------+-------------------+----------+---------+-------------------+----------------------+----------------------+----------+-------------------+--------------+-----+
        | id | title_id | team_id | global_or_team_id | filename | version | storage_id        | created_at           | updated_at           | platform | bundle_identifier | self_service | url |
        +----+----------+---------+-------------------+----------+---------+-------------------+----------------------+----------------------+----------+-------------------+--------------+-----+
        | 1  | 1        | 1       | 1                 | foo.ipa  | 1.2.3   | testingtesting123 | 2026-01-26T20:36:52Z | 2026-01-26T20:36:52Z | ipados   | com.foo           | 0            |     |
        | 2  | 2        | 1       | 1                 | foo.ipa  | 1.2.3   | testingtesting123 | 2026-01-26T20:36:52Z | 2026-01-26T20:36:52Z | ios      | com.foo           | 0            |     |
        +----+----------+---------+-------------------+----------+---------+-------------------+----------------------+----------------------+----------+-------------------+--------------+-----+
    testing_utils.go:552: << dumping table in_house_apps completed
    testing_utils.go:521: >> dumping table software_titles:
    testing_utils.go:569:
        +----+------+-------------+---------------+-------------------+-----------------------+-----------+----------------+-------------------+--------------+
        | id | name | source      | extension_for | bundle_identifier | additional_identifier | is_kernel | application_id | unique_identifier | upgrade_code |
        +----+------+-------------+---------------+-------------------+-----------------------+-----------+----------------+-------------------+--------------+
        | 1  | foo  | ipados_apps |               | com.foo           | 2                     | 0         | NULL           | com.foo           | NULL         |
        | 2  | foo  | ios_apps    |               | com.foo           | 1                     | 0         | NULL           | com.foo           | NULL         |
        +----+------+-------------+---------------+-------------------+-----------------------+-----------+----------------+-------------------+--------------+
    testing_utils.go:552: << dumping table software_titles completed
```
2026-01-27 09:57:53 -05:00
Victor Lyuboslavsky
089cf9a3ba
Added LoadDefaultSchema to platform testing utils to make test setup easier. (#38706)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #38234

Minor developer experience improvement for tests using MySQL.
This is a code review follow up.
2026-01-26 16:02:12 -06:00
Victor Lyuboslavsky
7deade8057
Activity bounded context: /api/latest/fleet/activities (2 of 2) (#38478)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #37806 

Removed `ds.ListActivities` from the legacy datastore and updated
code/tests to use the new activity bounded context instead.

The changes to `cron.go` and most changes to `mysql/activities_test.go`
will eventually be migrated to the activity bounded context. The current
changes are an intermediate step.

The issues tracked by https://github.com/fleetdm/fleet/issues/38234 will
be addressed in additional/parallel PRs shortly.

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
  - Done in the previous PR

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Refactor**
* Migrated activity retrieval from direct datastore calls to a
service-based architecture for improved maintainability and consistency.
* Enhanced system context handling for background automation tasks to
ensure proper authorization during scheduled operations.
* Streamlined activity recording for automated processes with dedicated
system identity tracking.

* **Tests**
* Updated test infrastructure with new helpers for activity service
integration across test suites.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
2026-01-23 07:42:09 -06:00
Magnus Jensen
8829b4e131
handle non atomic windows profiles when sending and receiving (#38332)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #37932

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information. - Added by the first sub-task for the entire
story.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2026-01-19 11:16:28 -05:00
Victor Lyuboslavsky
6019fa6d5a
Activity bounded context: /api/latest/fleet/activities (1 of 2) (#38115)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #37806 

This PR creates an activity bounded context and moves the following HTTP
endpoint (including the full vertical slice) there:
`/api/latest/fleet/activities`

NONE of the other activity functionality is moved! This is an
incremental approach starting with just 1 API/service endpoint.

A significant part of this PR is tests. This feature is now receiving
significantly more unit/integration test coverage than before.

Also, this PR does not remove the `ListActivities` datastore method in
the legacy code. That will be done in the follow up PR (part 2 of 2).

This refactoring effort also uncovered an activity/user authorization
issue: https://fleetdm.slack.com/archives/C02A8BRABB5/p1768582236611479

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Activity listing API now available with query filtering, date-range
filtering, and type-based filtering
* Pagination support for activity results with cursor-based and
offset-based options
* Configurable sorting by creation date or activity ID in ascending or
descending order
* Automatic enrichment of activity records with actor user details
(name, email, avatar)
* Role-based access controls applied to activity visibility based on
user permissions

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-19 09:07:14 -05:00
Victor Lyuboslavsky
506901443d
Moved common_mysql package to server/platform/mysql (#38017)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #37244

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Refactor**
* Internal MySQL utility package reorganized and all internal imports
updated to the new platform location; no changes to end-user
functionality or behavior.

* **Documentation**
* Added platform package documentation describing infrastructure
responsibilities and architectural boundaries to guide maintainers.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-08 13:17:19 -06:00
Victor Lyuboslavsky
bc0c7f1d13
Refactor common_mysql (#37245)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #37244

Goal: Make common_mysql package independent of domain packages so it can
be reused by future bounded contexts.

  Changes made:

  1. List options decoupling

The AppendListOptionsToSQL functions previously required
fleet.ListOptions directly. Now common_mysql defines its own interface
that describes what a list options type must provide (page number,
per-page limit, sort order, etc.). The fleet.ListOptions type implements
this interface through new getter methods. This lets any bounded context
use the SQL helpers without importing the fleet package.

  2. Error types moved

Database-specific error types like IsDuplicate and
IsChildForeignKeyError were moved from fleet package to common_mysql
where they belong. A new http/errors.go file was created for the
HTTP-specific error helpers that remain in the platform layer.

  3. Configuration restructuring

MySQL configuration types and functions were moved to
common_mysql/config.go, reducing coupling between packages.

  4. Architecture tests added

A new arch_test.go file enforces that common_mysql doesn't import domain
packages like fleet, preventing future regressions.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added cursor-based pagination support for list queries with improved
sorting capabilities including secondary order keys.

* **Bug Fixes**
* Improved database connection initialization with separate connection
management and error handling.

* **Refactor**
* Consolidated error handling interfaces and decoupled configuration
structures for better modularity.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-07 16:26:44 -06:00
Martin Angers
1253b64528
Add gitops support for in house apps (#35423) 2025-11-11 16:38:54 -05:00
Martin Angers
cab7cc15be
Initial support for in-house apps on iOS/iPadOS (#34802) 2025-10-28 08:33:58 -04:00
Jonathan Katz
c5ad64056c
32751 Change installer add/edit endpoints to read from writer db instead of reader (#33820)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32751 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [ ] QA'd all new/changed functionality manually

---------

Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
2025-10-14 16:59:01 -04:00
Victor Lyuboslavsky
a0e9d88e0d
Updated SQL modes in tests to match production. (#31445)
Fixes #31444 

The changes are primarily in tests. The only changes in production code
are a couple validations/checks for invalid values in:
- mysql/apple_mdm.go
- mysql/hosts.go
- mysql/queries.go

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved handling of timestamp and default values across various
features to prevent database errors and warnings.
* Enhanced validation and data consistency for Apple Business Manager
tokens and MDM profiles.
* Updated test data and logic to comply with stricter database
constraints and realistic scenarios, including date handling and field
lengths.

* **Chores**
* Updated test setups to reflect schema changes, improve data integrity,
and avoid future compatibility issues.
  * Standardized SQL mode and timestamp usage in test environments.
* Refined test data for VPP apps, software installers, and device
enrollments for better reliability.

* **Tests**
* Expanded and updated tests to cover new fields, stricter validation,
and more accurate simulation of real-world conditions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-08-03 08:18:13 +02:00
Juan Fernandez
1c7d2a0c8a
Removes duplicates when listing software titles on 'All teams' (#29459)
For #26375. 

When listing software titles for 'All teams', do not join against
software installers nor vpps to avoid duplicates.

Since filters related to software installers/VPP apps are no longer used
when viewing titles for 'All teams', the filter dropdown is disabled if
'All teams' is selected.
2025-06-07 10:47:53 -04:00
Martin Angers
17f1905470
Bugfix: Activate next activity on delete policy with associated software install/script exec (#29450) 2025-05-27 15:08:16 -04:00
Martin Angers
2673b83dee
IdP: resend profiles when IdP values change (#28436) 2025-04-29 14:35:37 -04:00
Martin Angers
78c0d67e78
Bugfix: re-create deleted iOS/iPadOS host entries in Fleet if it checks in again via MDM (#27231) 2025-03-26 09:33:38 -04:00
Victor Lyuboslavsky
93bdb437ac
Resend Windows profiles on change (#27308)
For #25030 

This PR includes the bug fix and tests.

It also includes the `secrets_updated_at` columns needed for story
#27351. These columns are currently unused and always NULL.

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
2025-03-20 14:43:04 -05:00
Martin Angers
8477856886 FIx conflicts 2025-02-24 14:28:34 -05:00
Sarah Gillespie
7b6e212003
Add datastore methods for host certificates feature (#26416) 2025-02-18 17:49:02 -06:00
Victor Lyuboslavsky
5ee9c0c809
Refactoring common mysql (#26367)
For #26218 

Refactoring android/mysql/testing_utils.go to remove duplication with
main mysql package.
No functional changes.

# Checklist for submitter

- [x] Manual QA for all new/changed functionality
2025-02-18 15:28:54 -06:00
Sarah Gillespie
4f954908ff
SSVL: Add labels to upload endpoint; add validations and related datastore methods (#24733) 2024-12-16 18:17:13 -06:00
Martin Angers
e5febe960d
Bugfix: create/update manual label returns outdated info when configured with a mysql replica (#23725) 2024-11-13 08:57:22 -05:00
Victor Lyuboslavsky
567803955e
NDES SCEP proxy backend (#22542)
#21955

<div>
<a href="https://www.loom.com/share/ba40b440502845d2861fd3ec7611bade">
<p>[Demo] Deploy SCEP certificates from Network Device Enrollment
Service (NDES) #21955 - Watch Video</p>
    </a>
<a href="https://www.loom.com/share/ba40b440502845d2861fd3ec7611bade">
<img style="max-width:300px;"
src="https://cdn.loom.com/sessions/thumbnails/ba40b440502845d2861fd3ec7611bade-84f2d88c9f5106c2-full-play.gif">
    </a>
  </div>

Note: A few remaining subtasks will be done in a follow-up PR. See
#22123 for a detailed list.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-10-09 13:47:27 -05:00
Victor Lyuboslavsky
5f846c8870
Merge remote-tracking branch 'origin/main' into 20320-feat-uninstall-packages 2024-09-11 09:35:04 -05:00
Jahziel Villasana-Espinoza
385da248cb
feat: validate profile with apple (#21862)
> Related issue: #17558

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-09-10 18:44:58 -04:00
Roberto Dip
05818902cd
implement OTA enrollment (#21942)
for #21019

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-09-10 16:52:17 -03:00
Victor Lyuboslavsky
7d47cd95d8
Fixed tests 2024-09-06 09:49:36 -05:00
Jahziel Villasana-Espinoza
a00559e732
feat: enable multiple ABM and VPP tokens (#21693)
> Related issue: #9956 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Roberto Dip <rroperzh@gmail.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Dante Catalfamo <43040593+dantecatalfamo@users.noreply.github.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
2024-08-29 18:51:46 -04:00
Roberto Dip
685bf58cd9
Self service vpp fixes (#21624)
for

- https://github.com/fleetdm/fleet/issues/21497
- https://github.com/fleetdm/fleet/issues/21498

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-08-28 10:47:27 -03:00
Victor Lyuboslavsky
4eb72535dc
Support for MySQL 8.4.2 (#21364)
#21270

The main change for MySQL 8.4.2 is that foreign key constraints are
stricter:
https://dev.mysql.com/doc/refman/8.4/en/server-system-variables.html#sysvar_restrict_fk_on_non_standard_key

Also, most replica-related commands have been renamed.

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- [x] Manual QA for all new/changed functionality
2024-08-16 16:32:38 +02:00
Lucas Manuel Rodriguez
ab7df5155d
Use docker compose on CI instead of docker-compose (#21017)
After this is merged I'll cherry pick to `minor-fleet-4.55.0`.
2024-08-02 18:12:36 -03:00
Tim Lee
bda812c329
Hotfix: Pass sql test dump in via stdin (#20871) 2024-07-31 05:09:52 -06:00
Roberto Dip
8d18e67ab6
prevent errors on tests that use real replicas (#20121)
For #20046, I found this while trying to get #20085 merged, as MySQL 8
tests were consistenly failing (5 times in a row) with this when
integration tests are run in parallel.

I used the `lhotari/action-upterm@v1` action to hook an SSH session into
the worker, and I noticed different errors in different runs, but all on
the same lines of this sample:

```
mysql> SELECT * FROM replication_applier_status_by_worker LIMIT 1 OFFSET 0\G
*************************** 1. row ***************************
                                           CHANNEL_NAME:
                                              WORKER_ID: 1
                                              THREAD_ID: NULL
                                          SERVICE_STATE: OFF
                                      LAST_ERROR_NUMBER: 1146
                                     LAST_ERROR_MESSAGE: Worker 1 failed executing transaction 'ANONYMOUS' at master log bin.000003, end_log_pos 25540153; Error 'Table 'server_datastore_mysql_TestTeams.host_mdm_apple_bootstrap_packages' doesn't exist' on query. Default database: 'server_datastore_mysql_TestTeams'. Query: 'TRUNCATE TABLE host_mdm_apple_bootstrap_packages'
                                   LAST_ERROR_TIMEST
```

This uses

[REPLICATE_DO_DB](https://dev.mysql.com/doc/refman/5.7/en/change-replication-filter.html)
to configure the replica at runtime to replicate only the databases that
are configured with replicas and prevent any unexpected behavior.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-07-01 11:19:02 -03:00
Victor Lyuboslavsky
7b751fa50a
Additional debug info and speculative fix for TestUpdateStatsOnReplica (#19901)
Additional debug info and speculative fix for TestUpdateStatsOnReplica
CI fails
2024-06-21 14:50:12 -05:00
Roberto Dip
1cc13a09fb
🧹 friday cleanup party: substitute deprecated import of go-kit (#19774)
`go-kit/kit/log` was deprecated and generating warnings

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Manual QA for all new/changed functionality
2024-06-17 10:27:31 -03:00
Roberto Dip
1c2ee0c949 Merge remote-tracking branch 'origin/main' into feat-save-certs 2024-05-31 09:34:48 -03:00
Victor Lyuboslavsky
f761827850
Move CalculateAggregatedPerfStatsPercentiles reads to the replica (#19206)
Move CalculateAggregatedPerfStatsPercentiles reads to the replica
#18838

I manually tested the aggregated query stats change by using a read
replica.
https://github.com/fleetdm/fleet/blob/main/tools/mysql-replica-testing/README.md
- But set `master_delay=0` due to issue
https://github.com/fleetdm/fleet/issues/19272
- Run a saved query as a live query, and see that its stats in
`aggregated_stats` table are updated.


# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-05-31 07:08:31 -05:00
Roberto Dip
545e56d288
19016 ingest certs on start (#19360)
For #19016

This changes all the places where we previously assumed that certs were
hardcoded when the Fleet server started to query the database instead.

The plan is to loadtest afterwards, but as a first preemptive measure,
this adds a caching layer on top the mysql datastore.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
2024-05-30 18:18:42 -03:00
Jahziel Villasana-Espinoza
7b2c25270a fix: broken test 2024-05-28 16:03:45 -04:00
Jahziel Villasana-Espinoza
188d773c56 fix: update integration tests to use private key 2024-05-28 13:28:49 -04:00
Martin Angers
7bb726ba8e
Create and return upcoming/past host activities for software installs (#18772) 2024-05-07 11:28:16 -04:00