Deprecate "team" and "query" API params (#39873)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](https://github.com/fleetdm/fleet/blob/da43bf8371695382c4af0972d5da456c6b94bdaf/server/service/endpoint_utils.go#L108)
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-02-19 19:53:32 +00:00
|
|
|
package endpointer
|
|
|
|
|
|
|
|
|
|
import (
|
|
|
|
|
"bytes"
|
|
|
|
|
"fmt"
|
|
|
|
|
"io"
|
|
|
|
|
|
|
|
|
|
"github.com/go-json-experiment/json/jsontext"
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
// AliasConflictError is returned when both the deprecated and new field names
|
|
|
|
|
// are specified in the same JSON object scope. For example, if "team_id" is
|
|
|
|
|
// renamed to "fleet_id", and a request contains both, this error is returned.
|
|
|
|
|
type AliasConflictError struct {
|
|
|
|
|
Old string
|
|
|
|
|
New string
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (e *AliasConflictError) Error() string {
|
2026-02-24 05:09:08 +00:00
|
|
|
return fmt.Sprintf("Conflicting field names: cannot specify both `%s` (deprecated) and `%s` in the same request", e.Old, e.New)
|
Deprecate "team" and "query" API params (#39873)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](https://github.com/fleetdm/fleet/blob/da43bf8371695382c4af0972d5da456c6b94bdaf/server/service/endpoint_utils.go#L108)
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-02-19 19:53:32 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// AliasRule defines a key-rename rule: the deprecated (old) key name and its
|
|
|
|
|
// replacement (new) key name. The struct's json tag uses OldKey (the current
|
|
|
|
|
// name), and renameto specifies NewKey (the target name). The rewriter
|
|
|
|
|
// accepts both names in requests: OldKey passes through as-is (with
|
|
|
|
|
// deprecation tracking) and NewKey is rewritten to OldKey for deserialization.
|
|
|
|
|
type AliasRule struct {
|
|
|
|
|
OldKey string
|
|
|
|
|
NewKey string
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// JSONKeyRewriteReader is a streaming io.Reader that handles
|
|
|
|
|
// JSON key aliasing while reading. It:
|
|
|
|
|
//
|
|
|
|
|
// - Passes through OldKey (deprecated) names as-is (the struct expects them)
|
|
|
|
|
// and tracks them in usedDeprecated for deprecation logging.
|
|
|
|
|
// - Rewrites NewKey names to OldKey so the struct can deserialize them.
|
|
|
|
|
// - Detects alias conflicts: if both OldKey and NewKey appear in the same
|
|
|
|
|
// JSON object scope, it returns an *AliasConflictError.
|
|
|
|
|
//
|
|
|
|
|
// It uses jsontext.Decoder/Encoder for token-level processing, delegating all
|
|
|
|
|
// JSON lexing (string escaping, unicode, whitespace) to the library.
|
|
|
|
|
type JSONKeyRewriteReader struct {
|
|
|
|
|
reader *bytes.Reader
|
|
|
|
|
initErr error
|
|
|
|
|
|
|
|
|
|
// Map from old (deprecated) key to its AliasRule for fast lookup.
|
|
|
|
|
oldKeyIndex map[string]AliasRule
|
|
|
|
|
// Map from new key to its AliasRule for fast lookup.
|
|
|
|
|
newKeyIndex map[string]AliasRule
|
|
|
|
|
|
|
|
|
|
// Tracks which deprecated keys have been used (old key -> true).
|
|
|
|
|
usedDeprecated map[string]bool
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// NewJSONKeyRewriteReader creates a new JSONKeyRewriteReader that wraps the
|
|
|
|
|
// given reader and applies the provided alias rules. It reads JSON tokens
|
|
|
|
|
// from src, handles bidirectional key aliasing, detects conflicts, and
|
|
|
|
|
// writes the result to an internal buffer.
|
|
|
|
|
func NewJSONKeyRewriteReader(src io.Reader, rules []AliasRule) *JSONKeyRewriteReader {
|
|
|
|
|
oldIdx := make(map[string]AliasRule, len(rules))
|
|
|
|
|
newIdx := make(map[string]AliasRule, len(rules))
|
|
|
|
|
for _, r := range rules {
|
|
|
|
|
oldIdx[r.OldKey] = r
|
|
|
|
|
newIdx[r.NewKey] = r
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
rw := &JSONKeyRewriteReader{
|
|
|
|
|
oldKeyIndex: oldIdx,
|
|
|
|
|
newKeyIndex: newIdx,
|
|
|
|
|
usedDeprecated: make(map[string]bool),
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
var buf bytes.Buffer
|
|
|
|
|
if err := rw.rewrite(src, &buf); err != nil {
|
|
|
|
|
rw.initErr = err
|
|
|
|
|
return rw
|
|
|
|
|
}
|
|
|
|
|
rw.reader = bytes.NewReader(buf.Bytes())
|
|
|
|
|
return rw
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// UsedDeprecatedKeys returns the list of deprecated key names that were
|
|
|
|
|
// encountered during reading. This should be called after the reader has been
|
|
|
|
|
// fully consumed (i.e., after json.Decoder.Decode or similar has returned),
|
|
|
|
|
// which guarantees the background goroutine has finished.
|
|
|
|
|
func (r *JSONKeyRewriteReader) UsedDeprecatedKeys() []string {
|
|
|
|
|
keys := make([]string, 0, len(r.usedDeprecated))
|
|
|
|
|
for k := range r.usedDeprecated {
|
|
|
|
|
keys = append(keys, k)
|
|
|
|
|
}
|
|
|
|
|
return keys
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Close closes the reader end of the pipe to unblock the transform goroutine
|
|
|
|
|
// if the consumer stops reading early.
|
|
|
|
|
func (r *JSONKeyRewriteReader) Close() error {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Read implements io.Reader by reading from the pipe.
|
|
|
|
|
func (r *JSONKeyRewriteReader) Read(p []byte) (int, error) {
|
|
|
|
|
if r.initErr != nil {
|
|
|
|
|
return 0, r.initErr
|
|
|
|
|
}
|
|
|
|
|
if r.reader == nil {
|
|
|
|
|
return 0, io.EOF
|
|
|
|
|
}
|
|
|
|
|
return r.reader.Read(p)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// RewriteDeprecatedKeys handles JSON key aliasing in data using
|
|
|
|
|
// the provided alias rules. It rewrites NewKey→OldKey (so the struct can
|
|
|
|
|
// deserialize), passes through OldKey as-is, and returns an error if both
|
|
|
|
|
// appear in the same scope (alias conflict) or the JSON is malformed.
|
|
|
|
|
//
|
|
|
|
|
// This is useful when a request body is captured as json.RawMessage and later
|
|
|
|
|
// decoded into a struct with `renameto` tags — the rewriter in MakeDecoder
|
|
|
|
|
// won't have seen the inner fields, so this function can be called before the
|
|
|
|
|
// deferred unmarshal.
|
2026-02-24 05:09:08 +00:00
|
|
|
func RewriteDeprecatedKeys(data []byte, rules []AliasRule) ([]byte, map[string]string, error) {
|
Deprecate "team" and "query" API params (#39873)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](https://github.com/fleetdm/fleet/blob/da43bf8371695382c4af0972d5da456c6b94bdaf/server/service/endpoint_utils.go#L108)
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-02-19 19:53:32 +00:00
|
|
|
if len(rules) == 0 || len(data) == 0 {
|
2026-02-24 05:09:08 +00:00
|
|
|
return data, nil, nil
|
Deprecate "team" and "query" API params (#39873)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](https://github.com/fleetdm/fleet/blob/da43bf8371695382c4af0972d5da456c6b94bdaf/server/service/endpoint_utils.go#L108)
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-02-19 19:53:32 +00:00
|
|
|
}
|
|
|
|
|
oldIdx := make(map[string]AliasRule, len(rules))
|
|
|
|
|
newIdx := make(map[string]AliasRule, len(rules))
|
|
|
|
|
for _, r := range rules {
|
|
|
|
|
oldIdx[r.OldKey] = r
|
|
|
|
|
newIdx[r.NewKey] = r
|
|
|
|
|
}
|
|
|
|
|
rw := &JSONKeyRewriteReader{
|
|
|
|
|
oldKeyIndex: oldIdx,
|
|
|
|
|
newKeyIndex: newIdx,
|
|
|
|
|
usedDeprecated: make(map[string]bool),
|
|
|
|
|
}
|
|
|
|
|
var buf bytes.Buffer
|
|
|
|
|
if err := rw.rewrite(bytes.NewReader(data), &buf); err != nil {
|
2026-02-24 05:09:08 +00:00
|
|
|
return nil, nil, err
|
Deprecate "team" and "query" API params (#39873)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](https://github.com/fleetdm/fleet/blob/da43bf8371695382c4af0972d5da456c6b94bdaf/server/service/endpoint_utils.go#L108)
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-02-19 19:53:32 +00:00
|
|
|
}
|
2026-02-24 05:09:08 +00:00
|
|
|
deprecatedKeysMap := make(map[string]string, len(rw.usedDeprecated))
|
|
|
|
|
for k := range rw.usedDeprecated {
|
|
|
|
|
deprecatedKeysMap[k] = rw.oldKeyIndex[k].NewKey
|
|
|
|
|
}
|
|
|
|
|
return buf.Bytes(), deprecatedKeysMap, nil
|
Deprecate "team" and "query" API params (#39873)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](https://github.com/fleetdm/fleet/blob/da43bf8371695382c4af0972d5da456c6b94bdaf/server/service/endpoint_utils.go#L108)
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-02-19 19:53:32 +00:00
|
|
|
}
|
|
|
|
|
|
2026-03-13 13:38:55 +00:00
|
|
|
// RewriteOldToNewKeys is the reverse of RewriteDeprecatedKey; it takes
|
|
|
|
|
// the rules and reverses them before translating keys.
|
|
|
|
|
// Use this in situations where a payload was rewritten from new to old keys
|
|
|
|
|
// for deserialization, but you want to return a response with the new keys
|
|
|
|
|
// for forward compatibility.
|
|
|
|
|
func RewriteOldToNewKeys(data []byte, rules []AliasRule) ([]byte, error) {
|
|
|
|
|
reversed := make([]AliasRule, len(rules))
|
|
|
|
|
for i, r := range rules {
|
|
|
|
|
reversed[i] = AliasRule{OldKey: r.NewKey, NewKey: r.OldKey}
|
|
|
|
|
}
|
|
|
|
|
result, _, err := RewriteDeprecatedKeys(data, reversed)
|
|
|
|
|
return result, err
|
|
|
|
|
}
|
|
|
|
|
|
Deprecate "team" and "query" API params (#39873)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](https://github.com/fleetdm/fleet/blob/da43bf8371695382c4af0972d5da456c6b94bdaf/server/service/endpoint_utils.go#L108)
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-02-19 19:53:32 +00:00
|
|
|
// rewrite reads tokens from src, rewrites deprecated keys, checks for alias
|
|
|
|
|
// conflicts, and writes the transformed JSON to w.
|
|
|
|
|
func (r *JSONKeyRewriteReader) rewrite(src io.Reader, w io.Writer) error {
|
|
|
|
|
dec := jsontext.NewDecoder(src, jsontext.AllowDuplicateNames(true))
|
|
|
|
|
enc := jsontext.NewEncoder(w, jsontext.AllowDuplicateNames(true))
|
|
|
|
|
|
|
|
|
|
// Stack of per-object-scope key sets for conflict detection.
|
|
|
|
|
// Pushed on '{', popped on '}'.
|
|
|
|
|
var keyScopes []map[string]bool
|
|
|
|
|
|
|
|
|
|
for {
|
|
|
|
|
tok, err := dec.ReadToken()
|
|
|
|
|
if err != nil {
|
|
|
|
|
if err == io.EOF {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
return err
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
kind := tok.Kind()
|
|
|
|
|
|
|
|
|
|
switch kind {
|
|
|
|
|
case '{':
|
|
|
|
|
keyScopes = append(keyScopes, make(map[string]bool))
|
|
|
|
|
if err := enc.WriteToken(tok); err != nil {
|
|
|
|
|
return err
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
case '}':
|
|
|
|
|
if len(keyScopes) > 0 {
|
|
|
|
|
keyScopes = keyScopes[:len(keyScopes)-1]
|
|
|
|
|
}
|
|
|
|
|
if err := enc.WriteToken(tok); err != nil {
|
|
|
|
|
return err
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
case '"':
|
|
|
|
|
// Determine if this string is an object key by checking the
|
|
|
|
|
// decoder's stack: inside an object ('{') at an odd length
|
|
|
|
|
// means we just read a key (name).
|
|
|
|
|
isKey := false
|
|
|
|
|
depth := dec.StackDepth()
|
|
|
|
|
if depth > 0 {
|
|
|
|
|
parentKind, length := dec.StackIndex(depth)
|
|
|
|
|
// length is odd after reading a name (names and values
|
|
|
|
|
// are counted separately).
|
|
|
|
|
if parentKind == '{' && length%2 == 1 {
|
|
|
|
|
isKey = true
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if isKey {
|
|
|
|
|
keyName := tok.String()
|
|
|
|
|
|
|
|
|
|
// Use OldKey as the canonical key for scope tracking.
|
|
|
|
|
// Both OldKey (pass-through) and NewKey (rewrite) resolve
|
|
|
|
|
// to the same canonical key for conflict detection.
|
|
|
|
|
|
|
|
|
|
if rule, ok := r.oldKeyIndex[keyName]; ok {
|
|
|
|
|
// This is an OldKey (deprecated name). Pass through
|
|
|
|
|
// as-is — the struct expects this name. Track it for
|
|
|
|
|
// deprecation logging.
|
|
|
|
|
canonicalKey := rule.OldKey
|
|
|
|
|
r.usedDeprecated[keyName] = true
|
|
|
|
|
|
|
|
|
|
// Conflict detection.
|
|
|
|
|
if len(keyScopes) > 0 {
|
|
|
|
|
scope := keyScopes[len(keyScopes)-1]
|
|
|
|
|
if scope[canonicalKey] {
|
|
|
|
|
return &AliasConflictError{Old: rule.OldKey, New: rule.NewKey}
|
|
|
|
|
}
|
|
|
|
|
scope[canonicalKey] = true
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Write the key as-is (old name, which the struct expects).
|
|
|
|
|
if err := enc.WriteToken(tok); err != nil {
|
|
|
|
|
return err
|
|
|
|
|
}
|
|
|
|
|
} else if rule, ok := r.newKeyIndex[keyName]; ok {
|
|
|
|
|
// This is a NewKey. Rewrite it to OldKey so the
|
|
|
|
|
// struct can deserialize it.
|
|
|
|
|
canonicalKey := rule.OldKey
|
|
|
|
|
|
|
|
|
|
// Conflict detection.
|
|
|
|
|
if len(keyScopes) > 0 {
|
|
|
|
|
scope := keyScopes[len(keyScopes)-1]
|
|
|
|
|
if scope[canonicalKey] {
|
|
|
|
|
return &AliasConflictError{Old: rule.OldKey, New: rule.NewKey}
|
|
|
|
|
}
|
|
|
|
|
scope[canonicalKey] = true
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Write the rewritten (old) key.
|
|
|
|
|
if err := enc.WriteToken(jsontext.String(canonicalKey)); err != nil {
|
|
|
|
|
return err
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
// Not an aliased key — pass through unchanged.
|
|
|
|
|
if err := enc.WriteToken(tok); err != nil {
|
|
|
|
|
return err
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
// String value — pass through unchanged.
|
|
|
|
|
if err := enc.WriteToken(tok); err != nil {
|
|
|
|
|
return err
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
default:
|
|
|
|
|
// All other tokens: [, ], numbers, bools, null — pass through.
|
|
|
|
|
if err := enc.WriteToken(tok); err != nil {
|
|
|
|
|
return err
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|