<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#44723
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Strengthened validation of sorting/order parameters across many list
and cursor-based endpoints — unsupported sort keys now return explicit
errors and prevent unsafe queries.
* Labels listing: label-list pagination query name changed; ordering by
host_count is rejected when host counts are disabled (validated at
request parsing).
* **Tests**
* Added/expanded tests covering allowed order keys, rejection of unknown
keys, and pagination behavior for multiple listing APIs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
For #36087
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added script execution API supporting asynchronous and synchronous
operations with timeout handling.
* Introduced batch script execution capabilities including batch run
creation, status querying, and execution cancellation.
* Added host management API endpoints for locking, unlocking, and wiping
devices.
* Enhanced script management with create, update, delete, list, and
retrieval operations.
* Improved file download responses with proper content headers and
attachment handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#42885
Added new middleware (APIOnlyEndpointCheck) that enforces 403 for
API-only users whose request either isn't in the API endpoint catalog or
falls outside their configured per-user endpoint restrictions.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40540
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes present in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Updated internal logging infrastructure to improve consistency and
maintainability across the application.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40054
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes included in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Consolidated and standardized internal logging infrastructure across
the application by adopting a unified logging package throughout the
codebase, replacing previous external logging dependencies.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40122
# Details
* Adds deprecation warnings to `fleetctl apply`
* Adds alias conflict errors (i.e. using both new and deprecated keys in
the same spec) to `fleetctl apply`
* Adds logic around all deprecated field warnings to check the topic
first
* Disables deprecation warnings by default for `fleet serve`, `fleetctl
gitops` and `fleetctl apply`
* Enables deprecation warnings for dogfood via env var
To turn on warnings:
* In `fleet serve`, use either
`--logging_enable_topics=deprecated-field-names` or the
`FLEET_LOGGING_ENABLE_TOPICS=deprecated-field-names` env var
* In `fleetctl gitops` / `fleetctl apply` use either
`--enable-log-topics=deprecated-field-names` or
`FLEET_ENABLE_LOG_TOPICS=deprecated-field-names`
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
tested in `fleetctl apply`, `fleet serve` and `fleet gitops` that
warnings are suppressed by default and added when the appropriate env
var or CLI option is used
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](da43bf8371/server/service/endpoint_utils.go (L108))
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#37192
- Move /server/service/middleware/endpoint_utils to
/server/platform/endpointer
- Move /server/service/middleware/authzcheck to
/server/platform/middleware/authzcheck
- Move /server/service/middleware/ratelimit to
/server/platform/middleware/ratelimit
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Refactor**
* Reorganized internal endpoint utilities to a centralized platform
location for improved code organization and maintainability. No
functional changes to existing features or APIs.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Resolves#37192
Separating generic endpoint_utils middleware logic from domain-specific
business logic. New bounded contexts would share the generic logic and
implement their own domain-specific logic. The two approaches used in
this PR are:
- Use common `platform` types
- Use interfaces
In the next PR we will move `endpointer_utils`, `authzcheck` and
`ratelimit` into `platform` directory.
# Checklist for submitter
- [x] Added changes file
## Testing
- [x] Added/updated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Restructured internal error handling and context management to support
bounded context architecture.
* Improved error context collection and telemetry observability through
a provider-based mechanism.
* Decoupled licensing and authentication concerns into interfaces for
better modularity.
* **Chores**
* Updated internal package dependencies to align with new architectural
boundaries.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#36670
Refactoring `middleware/endpoint_utils` package to remove direct
dependencies on:
- fleet.Service
- android.Service
Specific changes are:
- replace AuthFunc+FleetService with AuthMiddleware
- Move the definition of handler functions to the respective services
and use a generic `CommonEndpointer[H any] struct`
Although this was discovered as part of Activity bounded context
research, this change is not directly related to bounded contexts.
In retrospect, this decoupling should have been done when creating the
Android service for ADR-0001.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Internal restructuring of endpoint handling and authentication
middleware composition to improve code maintainability and type safety.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#36287
Updated 'fleetd/certificates/<id>' and 'fleetd/certificates/<id>/status'
to authenticate using the orbit_node_key provided in the
'Authentication' header.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35214
# Details
Pursuant to the discussion in
https://fleetdm.slack.com/archives/C084F4MKYSJ/p1762352268815269, this
PR updates the `/orbit/enroll` API handler such that:
* IF end-user auth is configured for the team the host is enrolling to,
* AND the host's user has not completed authentication,
* AND the Orbit version making the enroll request does not support
prompting for end-user authentication,
* THEN the host will not be blocked from enrolling.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [ ] Added/updated automated tests
working on this, will post when done
- [X] QA'd all new/changed functionality manually
* Set up my local Fleet instance with end-user auth enabled for setup
experience on a team
* With this branch running as Fleet server, ran Orbit also on this
branch and attempted to enroll to that team
* Verified that the SSO window was opened in my browser
* With Fleet server still running this branch, switched my local working
tree to `rc-minor-fleet-v4.75.0` and ran Orbit again
* Verified that the host enrolled successfully and error messages
appeared in the fleet server logs.
For #32040.
---
Backend changes to unblock the development of the orbit and frontend
changes.
New GET and PUT APIs for setting/getting software for Linux Setup
Experience:
```
curl -k -X GET -H "Authorization: Bearer $TEST_TOKEN" https://localhost:8080/api/latest/fleet/setup_experience/linux/software?team_id=8&per_page=3000
curl -k -X PUT -H "Authorization: Bearer $TEST_TOKEN" https://localhost:8080/api/latest/fleet/setup_experience/linux/software -d '{"team_id":8,"software_title_ids":[3000, 3001, 3007]}'
```
New setup_experience/init API called by orbit to trigger the Linux setup
experience on the device:
```
curl -v -k -X POST -H "Content-Type: application/json" "https://localhost:8080/api/fleet/orbit/setup_experience/init" -d '{"orbit_node_key": "ynYEtFsvv9xZ7rX619UE8of1I28H+GCj"}'
```
Get status API to call on "My device":
```
curl -v -k -X POST "https://localhost:8080/api/latest/fleet/device/7d940b6e-130a-493b-b58a-2b6e9f9f8bfc/setup_experience/status"
```
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [X] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Added Linux support for Setup Experience alongside macOS.
- Introduced platform-specific admin APIs to configure and retrieve
Setup Experience software (macOS/Linux).
- Added device API to report Setup Experience status and an Orbit API to
initialize Setup Experience on non-macOS devices.
- Setup Experience now gates policy queries on Linux until setup is
complete.
- New activity log entry when Setup Experience software is edited
(includes platform and team).
- Documentation
- Updated audit logs reference to include the new “edited setup
experience software” event.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Android scaffold code and refactorings
- Android packages intended to be decoupled from other Fleet code
Video explaining the PR: https://www.youtube.com/watch?v=cza-35Z9Wxk
# Checklist for submitter
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
Refactoring some functionality out of the service package so it can be
reused by a different service package.
- auth middleware
- logging errors
No functional changes.
`go-kit/kit/log` was deprecated and generating warnings
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Manual QA for all new/changed functionality
#15196 This is the work of @ghernandez345 except for adding the
`ResponseController` thing in Go to override the server timeout for that
specific sync endpoint so that the calls don't timeout waiting for a
script response (the default HTTP server timeout was 90s for our
server).
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
---------
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
#8129
Apart from fixing the issue in #8129, this change also introduces UUIDs
to Fleet errors. To be able to match a returned error from the API to a
error in the Fleet logs. See
https://fleetdm.slack.com/archives/C019WG4GH0A/p1677780622769939 for
more context.
Samples with the changes in this PR:
```
curl -k -H "Authorization: Bearer $TEST_TOKEN" -H 'Content-Type:application/json' "https://localhost:8080/api/v1/fleet/sso" -d ''
{
"message": "Bad request",
"errors": [
{
"name": "base",
"reason": "Expected JSON Body"
}
],
"uuid": "a01f6e10-354c-4ff0-b96e-1f64adb500b0"
}
```
```
curl -k -H "Authorization: Bearer $TEST_TOKEN" -H 'Content-Type:application/json' "https://localhost:8080/api/v1/fleet/sso" -d 'asd'
{
"message": "Bad request",
"errors": [
{
"name": "base",
"reason": "json decoder error"
}
],
"uuid": "5f716a64-7550-464b-a1dd-e6a505a9f89d"
}
```
```
curl -k -X GET -H "Authorization: Bearer badtoken" "https://localhost:8080/api/latest/fleet/teams"
{
"message": "Authentication required",
"errors": [
{
"name": "base",
"reason": "Authentication required"
}
],
"uuid": "efe45bc0-f956-4bf9-ba4f-aa9020a9aaaf"
}
```
```
curl -k -X PATCH -H "Authorization: Bearer $TEST_TOKEN" "https://localhost:8080/api/latest/fleet/users/14" -d '{"name": "Manuel2", "password": "what", "new_password": "p4ssw0rd.12345"}'
{
"message": "Authorization header required",
"errors": [
{
"name": "base",
"reason": "Authorization header required"
}
],
"uuid": "57f78cd0-4559-464f-9df7-36c9ef7c89b3"
}
```
```
curl -k -X PATCH -H "Authorization: Bearer $TEST_TOKEN" "https://localhost:8080/api/latest/fleet/users/14" -d '{"name": "Manuel2", "password": "what", "new_password": "p4ssw0rd.12345"}'
{
"message": "Permission Denied",
"uuid": "7f0220ad-6de7-4faf-8b6c-8d7ff9d2ca06"
}
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
* add premium feature critical policies
* update documentation
* add test for premium-only field
* update old change-file
* test policies more comprehensively
* also test team policies
* PATCH returns wrong timestamp, updating test
related to #8031, this adds the following headers to HTML responses:
- Strict-Transport-Security: informs browsers that the site should only
be accessed using HTTPS, and that any future attempts to access it
using HTTP should automatically be converted to HTTPS.
- X-Frames-Options: disallows embedding the UI in other sites via
<frame>, <iframe>, <embed> or <object>, which can prevent attacks like
clickjacking.
- X-Content-Type-Options: prevents browsers from trying to guess the MIME
type which can cause browsers to transform non-executable content into
executable content.
- Referrer-Policy: prevents leaking the origin of the referrer in the
Referer.
additionally, this ensures we set `X-Content-Type-Options` for CSV and
installer responses.
This adds a new mechanism to allow us to handle compatibility issues between Orbit, Fleet Server and Fleet Desktop.
The general idea is to _always_ send a custom header of the form:
```
fleet-capabilities-header = "X-Fleet-Capabilities:" capabilities
capabilities = capability * (,)
capability = string
```
Both from the server to the clients (Orbit, Fleet Desktop) and vice-versa. For an example, see: 8c0bbdd291
Also, the following applies:
- Backwards compat: if the header is not present, assume that orbit/fleet doesn't have the capability
- The current capabilities endpoint will be removed
### Motivation
This solution is trying to solve the following problems:
- We have three independent processes communicating with each other (Fleet Desktop, Orbit and Fleet Server). Each process can be updated independently, and therefore we need a way for each process to know what features are supported by its peers.
- We originally implemented a dedicated API endpoint in the server that returned a list of the capabilities (or "features") enabled, we found this, and any other server-only solution (like API versioning) to be insufficient because:
- There are cases in which the server also needs to know which features are supported by its clients
- Clients needed to poll for changes to detect if the capabilities supported by the server change, by sending the capabilities on each request we have a much cleaner way to handling different responses.
- We are also introducing an unauthenticated endpoint to get the server features, this gives us flexibility if we need to implement different authentication mechanisms, and was one of the pitfalls of the first implementation.
Related to https://github.com/fleetdm/fleet/issues/7929
* Finish first draft of API versions
* wip
* Finalize tests
* Revert change in handler
* Remove made up version
* Update versioning with aliases
* Add changes file
* Address review comments
* Revert overupdated routes
* Expand life time of deprecated APIs
* Fix test
* Comment out problematic part of test
* Revert bad path changes
