fleet/server/service/labels_test.go

550 lines
16 KiB
Go
Raw Normal View History

package service
import (
"context"
Add support for host vitals labels (#30278) # Details This PR adds support for a new label membership type, `host_vitals`. Membership for these labels is based on a database query created from user-supplied criteria. In this first iteration, the allowed criteria are very simple: a label can specify either an IdP group or IdP department, and hosts with linked users with a matching group or department. Groundwork is laid here for more complex host vitals queries, including `and` and `or` logic, different data types and different kinds of vitals (rather than just the "foreign" vitals of which IdP is an example). Note that this PR does _not_ include the cron job that will trigger membership updating, and it doesn't include ; for sake of simplicity in review that will be done in a follow-on PR. ## Basic flow ### Creating a host vitals label 1. A new label is created via the API / GitOps with membership type `host_vitals` and a `criteria` property that's a JSON blob. Currently the JSON can only contain `vital` and `value` keys (and must contain those keys) 2. The server validates that the specified `vital` exists in our [set of known host vitals](https://github.com/fleetdm/fleet/pull/30278/files#diff-b6d4c48f2624b82c2567b2b88db1de51c6b152eeb261d40acfd5b63a890839b7R418-R436). 3. The server validates that the [criteria can be parsed into a query](https://github.com/fleetdm/fleet/pull/30278/files?diff=unified&w=1#diff-4ac4cfba8bed490e8ef125a0556f5417156f805017bfe93c6e2c61aa94ba8a8cR81-R86). This also happens during GitOps dry run. 4. The label is saved (criteria is saved as JSON in the db) ### Updating membership for a host vitals label 1. The label's criteria is used to generate a query to run on the _Fleet_ db. 1. For each vital criteria, check the vital type. Currently only foreign vitals are supported. 2. For foreign vitals, add its group to a set we keep track of. 3. Add a `WHERE` clause section for the vital and value, e.g. `end_user_idp_groups = ?` 4. Once we have all the `WHERE` clauses, create the query as `SELECT %s FROM %s` + any joins contributed by foreign vitals groups + `WHERE ` + all the `WHERE` clauses we just calculated. The `%s` provide some flexibility if we want to use these queries in other contexts. 2. Delete all existing label members 3. Do an `INSERT...SELECT` using the query we calculated from the label criteria. The query will be `SELECT <label id> as label_id, hosts.id FROM hosts JOIN ...` ## Future work ### Domestic vitals These can be anything that we already store in the `hosts` table. Domestic vitals won't add any `JOIN`s to the calculated label query, and will simply be e.g. `hosts.hostname = ?` ### Custom vitals We currently support an `additional_queries` config that will cause other queries to run on hosts. The data returned from these queries is stored in a `hosts_additional` table as a JSON blob. We can use MySQL JSON functions to match values in this data, e.g. `JSON_EXTRACT(host_additional, `$.some_custom_vital`) = ?` # Checklist for submitter If some of the following don't apply, delete the relevant line. <!-- Note that API documentation changes are now addressed by the product design team. --> - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. > I'll add the changelog item when I add the cron job PR - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [X] If database migrations are included, checked table schema to confirm autoupdate - For new Fleet configuration settings - [X] Verified that the setting can be managed via GitOps, or confirmed that the setting is explicitly being excluded from GitOps. If managing via Gitops: - [X] Verified that the setting is exported via `fleetctl generate-gitops` - [X] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) - For database migrations: - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [X] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [X] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [X] Added/updated automated tests - [X] Manual QA for all new/changed functionality
2025-06-30 14:58:58 +00:00
"encoding/json"
"testing"
"time"
authz_ctx "github.com/fleetdm/fleet/v4/server/contexts/authz"
"github.com/fleetdm/fleet/v4/server/contexts/viewer"
"github.com/fleetdm/fleet/v4/server/datastore/mysql"
"github.com/fleetdm/fleet/v4/server/datastore/mysql/common_mysql/testing_utils"
"github.com/fleetdm/fleet/v4/server/fleet"
"github.com/fleetdm/fleet/v4/server/mock"
"github.com/fleetdm/fleet/v4/server/ptr"
"github.com/fleetdm/fleet/v4/server/test"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLabelsAuth(t *testing.T) {
ds := new(mock.Store)
svc, ctx := newTestService(t, ds, nil, nil)
ds.NewLabelFunc = func(ctx context.Context, lbl *fleet.Label, opts ...fleet.OptionalArg) (*fleet.Label, error) {
return lbl, nil
}
ds.SaveLabelFunc = func(ctx context.Context, lbl *fleet.Label, filter fleet.TeamFilter) (*fleet.Label, []uint, error) {
return lbl, nil, nil
}
ds.DeleteLabelFunc = func(ctx context.Context, nm string) error {
return nil
}
ds.ApplyLabelSpecsFunc = func(ctx context.Context, specs []*fleet.LabelSpec) error {
return nil
}
ds.LabelFunc = func(ctx context.Context, id uint, filter fleet.TeamFilter) (*fleet.Label, []uint, error) {
return &fleet.Label{}, nil, nil
}
ds.ListLabelsFunc = func(ctx context.Context, filter fleet.TeamFilter, opts fleet.ListOptions) ([]*fleet.Label, error) {
return nil, nil
}
2022-06-10 18:29:45 +00:00
ds.LabelsSummaryFunc = func(ctx context.Context) ([]*fleet.LabelSummary, error) {
return nil, nil
}
ds.ListHostsInLabelFunc = func(ctx context.Context, filter fleet.TeamFilter, lid uint, opts fleet.HostListOptions) ([]*fleet.Host, error) {
return nil, nil
}
ds.GetLabelSpecsFunc = func(ctx context.Context) ([]*fleet.LabelSpec, error) {
return nil, nil
}
ds.GetLabelSpecFunc = func(ctx context.Context, name string) (*fleet.LabelSpec, error) {
return &fleet.LabelSpec{}, nil
}
testCases := []struct {
name string
user *fleet.User
shouldFailWrite bool
shouldFailRead bool
}{
{
"global admin",
&fleet.User{GlobalRole: ptr.String(fleet.RoleAdmin)},
false,
false,
},
{
"global maintainer",
&fleet.User{GlobalRole: ptr.String(fleet.RoleMaintainer)},
false,
false,
},
{
"global observer",
&fleet.User{GlobalRole: ptr.String(fleet.RoleObserver)},
true,
false,
},
{
"team maintainer",
&fleet.User{Teams: []fleet.UserTeam{{Team: fleet.Team{ID: 1}, Role: fleet.RoleMaintainer}}},
Manage labels in GitOps (#27038) For #24473 # Checklist for submitter <!-- Note that API documentation changes are now addressed by the product design team. --> - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files) for more information. ## Details This PR adds the ability to manage labels via GitOps. Usage is as follows: * If a top-level `labels:` key is provided in the global YAML file provided to GitOps, then any labels in this list will be created (if using a new name) or updated (if using an existing name). * If no top-level `labels:` key is provided, no changes will be made to labels. This allows backwards-compatibility; customers won't blow away all of their labels if they don't immediately use `labels:` in their YAML Additionally, some new validation has been added so that label usage is checked prior to application. This means that when the gitops command is run, it will verify that any labels referenced elsewhere in the YAML (e.g. by software installers or mdm profiles) exist, and will bail with an error message if they don't. ## Testing **Test label deletion** 1. Add some labels via the UI 2. Run `fleetctl gitops --dry-run` with a default.yml file _without_ `labels:` in it, and verify that it doesn't say it will update or delete any labels 2. Run `fleetctl gitops` with a default.yml file _without_ `labels:` in it, and verify that it doesn't modify or remove your labels 4. Run `fleetctl gitops --dry-run` with a default.yml file with `labels:` in it and nothing underneath, and verify that it says that it will delete your labels 4. Run `fleetctl gitops` with a default.yml file with `labels:` in it and nothing underneath, and verify that it removes all your labels **Test label create/update** 1. Add a label "foo" via the UI 2. Run `fleetctl gitops --dry-run` with a default.yml file with two `labels:` in it, one named "foo" and one named "bar". Verify that the output says that one label will be created and one will be updated. 2. Run `fleetctl gitops` with a default.yml file with two `labels:` in it, one named "foo" and one named "bar". Verify that the two labels now exist in the UI with the configuration you specified. **Test label usage** 1. Add a label "foo" in the UI. 1. Run `fleetctl gitops --dry-run` with a default.yml file _without_ `labels:` in it, where a software installer or mdm profile uses the "foo" label via `labels_include_any`. Verify that the output doesn't complain about unknown labels. 1. Run `fleetctl gitops --dry-run` with a default.yml file _with_ `labels:` in it with nothing underneath, and a software installer or mdm profile uses the "foo" label via `labels_include_any`. Verify that the output complains about unknown label "foo" 1. Run `fleetctl gitops --dry-run` with a default.yml file _with_ `labels:` in it with a "foo" label defined underneath, and a software installer or mdm profile uses the "foo" label via `labels_include_any`. Verify that the output doesn't complain about unknown labels.
2025-03-19 21:35:11 +00:00
false,
false,
},
{
"team observer",
&fleet.User{Teams: []fleet.UserTeam{{Team: fleet.Team{ID: 1}, Role: fleet.RoleObserver}}},
true,
false,
},
}
for _, tt := range testCases {
t.Run(tt.name, func(t *testing.T) {
ctx := viewer.NewContext(ctx, viewer.Viewer{User: tt.user})
_, _, err := svc.NewLabel(ctx, fleet.LabelPayload{Name: t.Name(), Query: `SELECT 1`})
checkAuthErr(t, tt.shouldFailWrite, err)
_, _, err = svc.ModifyLabel(ctx, 1, fleet.ModifyLabelPayload{})
checkAuthErr(t, tt.shouldFailWrite, err)
err = svc.ApplyLabelSpecs(ctx, []*fleet.LabelSpec{})
checkAuthErr(t, tt.shouldFailWrite, err)
_, _, err = svc.GetLabel(ctx, 1)
checkAuthErr(t, tt.shouldFailRead, err)
_, err = svc.GetLabelSpecs(ctx)
checkAuthErr(t, tt.shouldFailRead, err)
_, err = svc.GetLabelSpec(ctx, "abc")
checkAuthErr(t, tt.shouldFailRead, err)
_, err = svc.ListLabels(ctx, fleet.ListOptions{})
checkAuthErr(t, tt.shouldFailRead, err)
2022-06-10 18:29:45 +00:00
_, err = svc.LabelsSummary((ctx))
checkAuthErr(t, tt.shouldFailRead, err)
_, err = svc.ListHostsInLabel(ctx, 1, fleet.HostListOptions{})
checkAuthErr(t, tt.shouldFailRead, err)
err = svc.DeleteLabel(ctx, "abc")
checkAuthErr(t, tt.shouldFailWrite, err)
err = svc.DeleteLabelByID(ctx, 1)
checkAuthErr(t, tt.shouldFailWrite, err)
})
}
}
func TestLabelsWithDS(t *testing.T) {
ds := mysql.CreateMySQLDS(t)
cases := []struct {
name string
fn func(t *testing.T, ds *mysql.Datastore)
}{
{"GetLabel", testLabelsGetLabel},
{"ListLabels", testLabelsListLabels},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
defer mysql.TruncateTables(t, ds)
c.fn(t, ds)
})
}
}
func testLabelsGetLabel(t *testing.T, ds *mysql.Datastore) {
svc, ctx := newTestService(t, ds, nil, nil)
label := &fleet.Label{
Name: "foo",
Query: "select * from foo;",
}
label, err := ds.NewLabel(ctx, label)
assert.Nil(t, err)
assert.NotZero(t, label.ID)
labelVerify, _, err := svc.GetLabel(test.UserContext(ctx, test.UserAdmin), label.ID)
assert.Nil(t, err)
assert.Equal(t, label.ID, labelVerify.ID)
Add author ID to labels (#27055) For #27035 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [x] If database migrations are included, checked table schema to confirm autoupdate - For database migrations: - [x] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [x] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [x] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [ ] Added/updated automated tests - [x] A detailed QA plan exists on the associated ticket (if it isn't there, work with the product group's QA engineer to add it) - [x] Manual QA for all new/changed functionality ## Details This PR adds an `author_id` column to the `labels` table, and adds the associated properties to the `Label` and `LabelSpec` types. When a new label is created via the UI or API, an author ID is set on the label if one can be inferred from the context. Otherwise, the author ID is set to `null`. ## Authz and Automated testing Additional backend authorization logic is introduced in a follow-on PR, https://github.com/fleetdm/fleet/pull/27089, because rconciling all of the test updates between this PR and https://github.com/fleetdm/fleet/pull/27038 was getting complicated. ## Manual Testing * Tested in the UI by creating a new label on the Hosts page * Tested via Gitops by merging this branch with https://github.com/fleetdm/fleet/pull/27038 and doing `fleetctl gitops` with a global config with `labels:` in it.
2025-03-20 21:05:16 +00:00
assert.Nil(t, label.AuthorID)
}
func testLabelsListLabels(t *testing.T, ds *mysql.Datastore) {
svc, ctx := newTestService(t, ds, nil, nil)
require.NoError(t, ds.MigrateData(context.Background()))
labels, err := svc.ListLabels(test.UserContext(ctx, test.UserAdmin), fleet.ListOptions{Page: 0, PerPage: 1000})
require.NoError(t, err)
require.Len(t, labels, 8)
2022-06-10 18:29:45 +00:00
labelsSummary, err := svc.LabelsSummary(test.UserContext(ctx, test.UserAdmin))
2022-06-10 18:29:45 +00:00
require.NoError(t, err)
require.Len(t, labelsSummary, 8)
2022-06-10 18:29:45 +00:00
}
func TestApplyLabelSpecsWithBuiltInLabels(t *testing.T) {
t.Parallel()
ds := new(mock.Store)
user := &fleet.User{
ID: 3,
Email: "foo@bar.com",
GlobalRole: ptr.String(fleet.RoleAdmin),
}
svc, ctx := newTestService(t, ds, nil, nil)
ctx = viewer.NewContext(ctx, viewer.Viewer{User: user})
name := "foo"
description := "bar"
query := "select * from foo;"
platform := ""
labelType := fleet.LabelTypeBuiltIn
labelMembershipType := fleet.LabelMembershipTypeDynamic
spec := &fleet.LabelSpec{
Name: name,
Description: description,
Query: query,
LabelType: labelType,
LabelMembershipType: labelMembershipType,
}
ds.LabelsByNameFunc = func(ctx context.Context, names []string) (map[string]*fleet.Label, error) {
return map[string]*fleet.Label{
name: {
Name: name,
Description: description,
Query: query,
Platform: platform,
LabelType: labelType,
LabelMembershipType: labelMembershipType,
},
}, nil
}
// all good
err := svc.ApplyLabelSpecs(ctx, []*fleet.LabelSpec{spec})
require.NoError(t, err)
const errorMessage = "cannot modify or add built-in label"
// not ok -- built-in label name doesn't exist
name = "not-foo"
err = svc.ApplyLabelSpecs(ctx, []*fleet.LabelSpec{spec})
assert.ErrorContains(t, err, errorMessage)
name = "foo"
// not ok -- description does not match
description = "not-bar"
err = svc.ApplyLabelSpecs(ctx, []*fleet.LabelSpec{spec})
assert.ErrorContains(t, err, errorMessage)
description = "bar"
// not ok -- query does not match
query = "select * from not-foo;"
err = svc.ApplyLabelSpecs(ctx, []*fleet.LabelSpec{spec})
assert.ErrorContains(t, err, errorMessage)
query = "select * from foo;"
// not ok -- label type does not match
labelType = fleet.LabelTypeRegular
err = svc.ApplyLabelSpecs(ctx, []*fleet.LabelSpec{spec})
assert.ErrorContains(t, err, errorMessage)
labelType = fleet.LabelTypeBuiltIn
// not ok -- label membership type does not match
labelMembershipType = fleet.LabelMembershipTypeManual
err = svc.ApplyLabelSpecs(ctx, []*fleet.LabelSpec{spec})
assert.ErrorContains(t, err, errorMessage)
labelMembershipType = fleet.LabelMembershipTypeDynamic
// not ok -- DB error
ds.LabelsByNameFunc = func(ctx context.Context, names []string) (map[string]*fleet.Label, error) {
return nil, assert.AnError
}
err = svc.ApplyLabelSpecs(ctx, []*fleet.LabelSpec{spec})
assert.ErrorIs(t, err, assert.AnError)
}
func TestLabelsWithReplica(t *testing.T) {
opts := &testing_utils.DatastoreTestOptions{DummyReplica: true}
ds := mysql.CreateMySQLDSWithOptions(t, opts)
defer ds.Close()
svc, ctx := newTestService(t, ds, nil, nil)
Add author ID to labels (#27055) For #27035 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [x] If database migrations are included, checked table schema to confirm autoupdate - For database migrations: - [x] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [x] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [x] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [ ] Added/updated automated tests - [x] A detailed QA plan exists on the associated ticket (if it isn't there, work with the product group's QA engineer to add it) - [x] Manual QA for all new/changed functionality ## Details This PR adds an `author_id` column to the `labels` table, and adds the associated properties to the `Label` and `LabelSpec` types. When a new label is created via the UI or API, an author ID is set on the label if one can be inferred from the context. Otherwise, the author ID is set to `null`. ## Authz and Automated testing Additional backend authorization logic is introduced in a follow-on PR, https://github.com/fleetdm/fleet/pull/27089, because rconciling all of the test updates between this PR and https://github.com/fleetdm/fleet/pull/27038 was getting complicated. ## Manual Testing * Tested in the UI by creating a new label on the Hosts page * Tested via Gitops by merging this branch with https://github.com/fleetdm/fleet/pull/27038 and doing `fleetctl gitops` with a global config with `labels:` in it.
2025-03-20 21:05:16 +00:00
user, err := ds.NewUser(ctx, &fleet.User{
Name: "Adminboi",
Password: []byte("p4ssw0rd.123"),
Email: "admin@example.com",
GlobalRole: ptr.String(fleet.RoleAdmin),
})
require.NoError(t, err)
ctx = viewer.NewContext(ctx, viewer.Viewer{User: user})
// create a couple hosts
h1, err := ds.NewHost(ctx, &fleet.Host{
Hostname: "host1",
HardwareSerial: uuid.NewString(),
UUID: uuid.NewString(),
Platform: "darwin",
LastEnrolledAt: time.Now(),
DetailUpdatedAt: time.Now(),
})
require.NoError(t, err)
h2, err := ds.NewHost(ctx, &fleet.Host{
Hostname: "host2",
HardwareSerial: uuid.NewString(),
UUID: uuid.NewString(),
Platform: "darwin",
LastEnrolledAt: time.Now(),
DetailUpdatedAt: time.Now(),
})
require.NoError(t, err)
// make the newly-created hosts available to the reader
opts.RunReplication()
lbl, hostIDs, err := svc.NewLabel(ctx, fleet.LabelPayload{Name: "label1", Hosts: []string{"host1", "host2"}})
require.NoError(t, err)
require.ElementsMatch(t, []uint{h1.ID, h2.ID}, hostIDs)
require.Equal(t, 2, lbl.HostCount)
Add author ID to labels (#27055) For #27035 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [x] If database migrations are included, checked table schema to confirm autoupdate - For database migrations: - [x] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [x] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [x] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [ ] Added/updated automated tests - [x] A detailed QA plan exists on the associated ticket (if it isn't there, work with the product group's QA engineer to add it) - [x] Manual QA for all new/changed functionality ## Details This PR adds an `author_id` column to the `labels` table, and adds the associated properties to the `Label` and `LabelSpec` types. When a new label is created via the UI or API, an author ID is set on the label if one can be inferred from the context. Otherwise, the author ID is set to `null`. ## Authz and Automated testing Additional backend authorization logic is introduced in a follow-on PR, https://github.com/fleetdm/fleet/pull/27089, because rconciling all of the test updates between this PR and https://github.com/fleetdm/fleet/pull/27038 was getting complicated. ## Manual Testing * Tested in the UI by creating a new label on the Hosts page * Tested via Gitops by merging this branch with https://github.com/fleetdm/fleet/pull/27038 and doing `fleetctl gitops` with a global config with `labels:` in it.
2025-03-20 21:05:16 +00:00
require.Equal(t, user.ID, *lbl.AuthorID)
// make the newly-created label available to the reader
opts.RunReplication("labels", "label_membership")
lbl, hostIDs, err = svc.ModifyLabel(ctx, lbl.ID, fleet.ModifyLabelPayload{Hosts: []string{"host1"}})
require.NoError(t, err)
require.ElementsMatch(t, []uint{h1.ID}, hostIDs)
require.Equal(t, 1, lbl.HostCount)
Add author ID to labels (#27055) For #27035 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [x] If database migrations are included, checked table schema to confirm autoupdate - For database migrations: - [x] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [x] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [x] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [ ] Added/updated automated tests - [x] A detailed QA plan exists on the associated ticket (if it isn't there, work with the product group's QA engineer to add it) - [x] Manual QA for all new/changed functionality ## Details This PR adds an `author_id` column to the `labels` table, and adds the associated properties to the `Label` and `LabelSpec` types. When a new label is created via the UI or API, an author ID is set on the label if one can be inferred from the context. Otherwise, the author ID is set to `null`. ## Authz and Automated testing Additional backend authorization logic is introduced in a follow-on PR, https://github.com/fleetdm/fleet/pull/27089, because rconciling all of the test updates between this PR and https://github.com/fleetdm/fleet/pull/27038 was getting complicated. ## Manual Testing * Tested in the UI by creating a new label on the Hosts page * Tested via Gitops by merging this branch with https://github.com/fleetdm/fleet/pull/27038 and doing `fleetctl gitops` with a global config with `labels:` in it.
2025-03-20 21:05:16 +00:00
require.Equal(t, user.ID, *lbl.AuthorID)
// reading this label without replication returns the old data as it only uses the reader
lbl, hostIDs, err = svc.GetLabel(ctx, lbl.ID)
require.NoError(t, err)
require.ElementsMatch(t, []uint{h1.ID, h2.ID}, hostIDs)
require.Equal(t, 2, lbl.HostCount)
Add author ID to labels (#27055) For #27035 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [x] If database migrations are included, checked table schema to confirm autoupdate - For database migrations: - [x] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [x] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [x] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [ ] Added/updated automated tests - [x] A detailed QA plan exists on the associated ticket (if it isn't there, work with the product group's QA engineer to add it) - [x] Manual QA for all new/changed functionality ## Details This PR adds an `author_id` column to the `labels` table, and adds the associated properties to the `Label` and `LabelSpec` types. When a new label is created via the UI or API, an author ID is set on the label if one can be inferred from the context. Otherwise, the author ID is set to `null`. ## Authz and Automated testing Additional backend authorization logic is introduced in a follow-on PR, https://github.com/fleetdm/fleet/pull/27089, because rconciling all of the test updates between this PR and https://github.com/fleetdm/fleet/pull/27038 was getting complicated. ## Manual Testing * Tested in the UI by creating a new label on the Hosts page * Tested via Gitops by merging this branch with https://github.com/fleetdm/fleet/pull/27038 and doing `fleetctl gitops` with a global config with `labels:` in it.
2025-03-20 21:05:16 +00:00
require.Equal(t, user.ID, *lbl.AuthorID)
// running the replication makes the updated data available
opts.RunReplication("labels", "label_membership")
lbl, hostIDs, err = svc.GetLabel(ctx, lbl.ID)
require.NoError(t, err)
require.ElementsMatch(t, []uint{h1.ID}, hostIDs)
require.Equal(t, 1, lbl.HostCount)
Add author ID to labels (#27055) For #27035 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [x] If database migrations are included, checked table schema to confirm autoupdate - For database migrations: - [x] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [x] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [x] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [ ] Added/updated automated tests - [x] A detailed QA plan exists on the associated ticket (if it isn't there, work with the product group's QA engineer to add it) - [x] Manual QA for all new/changed functionality ## Details This PR adds an `author_id` column to the `labels` table, and adds the associated properties to the `Label` and `LabelSpec` types. When a new label is created via the UI or API, an author ID is set on the label if one can be inferred from the context. Otherwise, the author ID is set to `null`. ## Authz and Automated testing Additional backend authorization logic is introduced in a follow-on PR, https://github.com/fleetdm/fleet/pull/27089, because rconciling all of the test updates between this PR and https://github.com/fleetdm/fleet/pull/27038 was getting complicated. ## Manual Testing * Tested in the UI by creating a new label on the Hosts page * Tested via Gitops by merging this branch with https://github.com/fleetdm/fleet/pull/27038 and doing `fleetctl gitops` with a global config with `labels:` in it.
2025-03-20 21:05:16 +00:00
require.Equal(t, user.ID, *lbl.AuthorID)
}
func TestBatchValidateLabels(t *testing.T) {
ds := new(mock.Store)
svc, ctx := newTestService(t, ds, nil, nil)
t.Run("no auth context", func(t *testing.T) {
_, err := svc.BatchValidateLabels(context.Background(), nil)
require.ErrorContains(t, err, "Authentication required")
})
authCtx := authz_ctx.AuthorizationContext{}
ctx = authz_ctx.NewContext(ctx, &authCtx)
t.Run("no auth checked", func(t *testing.T) {
_, err := svc.BatchValidateLabels(ctx, nil)
require.ErrorContains(t, err, "Authentication required")
})
// validator requires that an authz check has been performed upstream so we'll set it now for
// the rest of the tests
authCtx.SetChecked()
mockLabels := map[string]uint{
"foo": 1,
"bar": 2,
"baz": 3,
}
mockLabelIdent := func(name string, id uint) fleet.LabelIdent {
return fleet.LabelIdent{LabelID: id, LabelName: name}
}
ds.LabelIDsByNameFunc = func(ctx context.Context, names []string) (map[string]uint, error) {
res := make(map[string]uint)
if names == nil {
return res, nil
}
for _, name := range names {
if id, ok := mockLabels[name]; ok {
res[name] = id
}
}
return res, nil
}
testCases := []struct {
name string
labelNames []string
expectLabels map[string]fleet.LabelIdent
expectError string
}{
{
"no labels",
nil,
nil,
"",
},
{
"include labels",
[]string{"foo", "bar"},
map[string]fleet.LabelIdent{
"foo": mockLabelIdent("foo", 1),
"bar": mockLabelIdent("bar", 2),
},
"",
},
{
"non-existent label",
[]string{"foo", "qux"},
nil,
"some or all the labels provided don't exist",
},
{
"duplicate label",
[]string{"foo", "foo"},
map[string]fleet.LabelIdent{
"foo": mockLabelIdent("foo", 1),
},
"",
},
{
"empty slice",
[]string{},
nil,
"",
},
{
"empty string",
[]string{""},
nil,
"some or all the labels provided don't exist",
},
}
for _, tt := range testCases {
t.Run(tt.name, func(t *testing.T) {
got, err := svc.BatchValidateLabels(ctx, tt.labelNames)
if tt.expectError != "" {
require.Contains(t, err.Error(), tt.expectError)
} else {
require.NoError(t, err)
require.Equal(t, tt.expectLabels, got)
}
})
}
}
func TestNewManualLabel(t *testing.T) {
ds := new(mock.Store)
svc, ctx := newTestService(t, ds, nil, nil)
ctx = viewer.NewContext(ctx, viewer.Viewer{User: &fleet.User{GlobalRole: ptr.String(fleet.RoleAdmin)}})
ds.NewLabelFunc = func(ctx context.Context, lbl *fleet.Label, opts ...fleet.OptionalArg) (*fleet.Label, error) {
lbl.ID = 1
lbl.LabelMembershipType = fleet.LabelMembershipTypeManual
return lbl, nil
}
ds.HostIDsByIdentifierFunc = func(ctx context.Context, filter fleet.TeamFilter, hostnames []string) ([]uint, error) {
return []uint{99, 100}, nil
}
t.Run("using hostnames", func(t *testing.T) {
ds.UpdateLabelMembershipByHostIDsFunc = func(ctx context.Context, labelID uint, hostIds []uint, teamFilter fleet.TeamFilter) (*fleet.Label, []uint, error) {
require.Equal(t, uint(1), labelID)
require.Equal(t, []uint{99, 100}, hostIds)
return nil, nil, nil
}
_, _, err := svc.NewLabel(ctx, fleet.LabelPayload{
Name: "foo",
Hosts: []string{"host1", "host2"},
})
require.NoError(t, err)
})
t.Run("using IDs", func(t *testing.T) {
ds.UpdateLabelMembershipByHostIDsFunc = func(ctx context.Context, labelID uint, hostIds []uint, teamFilter fleet.TeamFilter) (*fleet.Label, []uint, error) {
require.Equal(t, uint(1), labelID)
require.Equal(t, []uint{1, 2}, hostIds)
return nil, nil, nil
}
_, _, err := svc.NewLabel(ctx, fleet.LabelPayload{
Name: "foo",
HostIDs: []uint{1, 2},
})
require.NoError(t, err)
})
}
func TestModifyManualLabel(t *testing.T) {
ds := new(mock.Store)
svc, ctx := newTestService(t, ds, nil, nil)
ctx = viewer.NewContext(ctx, viewer.Viewer{User: &fleet.User{GlobalRole: ptr.String(fleet.RoleAdmin)}})
ds.LabelFunc = func(ctx context.Context, lid uint, teamFilter fleet.TeamFilter) (*fleet.Label, []uint, error) {
return &fleet.Label{
ID: lid,
LabelMembershipType: fleet.LabelMembershipTypeManual,
}, nil, nil
}
ds.HostIDsByIdentifierFunc = func(ctx context.Context, filter fleet.TeamFilter, hostnames []string) ([]uint, error) {
return []uint{99, 100}, nil
}
ds.SaveLabelFunc = func(ctx context.Context, lbl *fleet.Label, filter fleet.TeamFilter) (*fleet.Label, []uint, error) {
return nil, nil, nil
}
t.Run("using hostnames", func(t *testing.T) {
ds.UpdateLabelMembershipByHostIDsFunc = func(ctx context.Context, labelID uint, hostIds []uint, teamFilter fleet.TeamFilter) (*fleet.Label, []uint, error) {
require.Equal(t, uint(1), labelID)
require.Equal(t, []uint{99, 100}, hostIds)
return nil, nil, nil
}
_, _, err := svc.ModifyLabel(ctx, 1, fleet.ModifyLabelPayload{
Hosts: []string{"host1", "host2"},
})
require.NoError(t, err)
})
t.Run("using IDs", func(t *testing.T) {
ds.UpdateLabelMembershipByHostIDsFunc = func(ctx context.Context, labelID uint, hostIds []uint, teamFilter fleet.TeamFilter) (*fleet.Label, []uint, error) {
require.Equal(t, uint(1), labelID)
require.Equal(t, []uint{1, 2}, hostIds)
return nil, nil, nil
}
_, _, err := svc.ModifyLabel(ctx, 1, fleet.ModifyLabelPayload{
HostIDs: []uint{1, 2},
})
require.NoError(t, err)
})
}
Add support for host vitals labels (#30278) # Details This PR adds support for a new label membership type, `host_vitals`. Membership for these labels is based on a database query created from user-supplied criteria. In this first iteration, the allowed criteria are very simple: a label can specify either an IdP group or IdP department, and hosts with linked users with a matching group or department. Groundwork is laid here for more complex host vitals queries, including `and` and `or` logic, different data types and different kinds of vitals (rather than just the "foreign" vitals of which IdP is an example). Note that this PR does _not_ include the cron job that will trigger membership updating, and it doesn't include ; for sake of simplicity in review that will be done in a follow-on PR. ## Basic flow ### Creating a host vitals label 1. A new label is created via the API / GitOps with membership type `host_vitals` and a `criteria` property that's a JSON blob. Currently the JSON can only contain `vital` and `value` keys (and must contain those keys) 2. The server validates that the specified `vital` exists in our [set of known host vitals](https://github.com/fleetdm/fleet/pull/30278/files#diff-b6d4c48f2624b82c2567b2b88db1de51c6b152eeb261d40acfd5b63a890839b7R418-R436). 3. The server validates that the [criteria can be parsed into a query](https://github.com/fleetdm/fleet/pull/30278/files?diff=unified&w=1#diff-4ac4cfba8bed490e8ef125a0556f5417156f805017bfe93c6e2c61aa94ba8a8cR81-R86). This also happens during GitOps dry run. 4. The label is saved (criteria is saved as JSON in the db) ### Updating membership for a host vitals label 1. The label's criteria is used to generate a query to run on the _Fleet_ db. 1. For each vital criteria, check the vital type. Currently only foreign vitals are supported. 2. For foreign vitals, add its group to a set we keep track of. 3. Add a `WHERE` clause section for the vital and value, e.g. `end_user_idp_groups = ?` 4. Once we have all the `WHERE` clauses, create the query as `SELECT %s FROM %s` + any joins contributed by foreign vitals groups + `WHERE ` + all the `WHERE` clauses we just calculated. The `%s` provide some flexibility if we want to use these queries in other contexts. 2. Delete all existing label members 3. Do an `INSERT...SELECT` using the query we calculated from the label criteria. The query will be `SELECT <label id> as label_id, hosts.id FROM hosts JOIN ...` ## Future work ### Domestic vitals These can be anything that we already store in the `hosts` table. Domestic vitals won't add any `JOIN`s to the calculated label query, and will simply be e.g. `hosts.hostname = ?` ### Custom vitals We currently support an `additional_queries` config that will cause other queries to run on hosts. The data returned from these queries is stored in a `hosts_additional` table as a JSON blob. We can use MySQL JSON functions to match values in this data, e.g. `JSON_EXTRACT(host_additional, `$.some_custom_vital`) = ?` # Checklist for submitter If some of the following don't apply, delete the relevant line. <!-- Note that API documentation changes are now addressed by the product design team. --> - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. > I'll add the changelog item when I add the cron job PR - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [X] If database migrations are included, checked table schema to confirm autoupdate - For new Fleet configuration settings - [X] Verified that the setting can be managed via GitOps, or confirmed that the setting is explicitly being excluded from GitOps. If managing via Gitops: - [X] Verified that the setting is exported via `fleetctl generate-gitops` - [X] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) - For database migrations: - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [X] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [X] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [X] Added/updated automated tests - [X] Manual QA for all new/changed functionality
2025-06-30 14:58:58 +00:00
func TestNewHostVitalsLabel(t *testing.T) {
ds := new(mock.Store)
svc, ctx := newTestService(t, ds, nil, nil)
ctx = viewer.NewContext(ctx, viewer.Viewer{User: &fleet.User{GlobalRole: ptr.String(fleet.RoleAdmin)}})
ds.NewLabelFunc = func(ctx context.Context, lbl *fleet.Label, opts ...fleet.OptionalArg) (*fleet.Label, error) {
return lbl, nil
}
t.Run("create host vitals label", func(t *testing.T) {
lbl, _, err := svc.NewLabel(ctx, fleet.LabelPayload{
Name: "foo",
Criteria: &fleet.HostVitalCriteria{
Vital: ptr.String("end_user_idp_group"),
Value: ptr.String("admin"),
},
})
require.NoError(t, err)
assert.Equal(t, fleet.LabelTypeRegular, lbl.LabelType)
assert.Equal(t, fleet.LabelMembershipTypeHostVitals, lbl.LabelMembershipType)
// Test parsing the criteria
query, queryValues, err := lbl.CalculateHostVitalsQuery()
require.NoError(t, err)
queryValuesJson, err := json.Marshal(queryValues)
require.NoError(t, err)
assert.Equal(t, "SELECT %s FROM %s RIGHT JOIN host_scim_user ON (hosts.id = host_scim_user.host_id) JOIN scim_users ON (host_scim_user.scim_user_id = scim_users.id) LEFT JOIN scim_user_group ON (host_scim_user.scim_user_id = scim_user_group.scim_user_id) LEFT JOIN scim_groups ON (scim_user_group.group_id = scim_groups.id) WHERE scim_groups.display_name = ? GROUP BY hosts.id", query)
Add support for host vitals labels (#30278) # Details This PR adds support for a new label membership type, `host_vitals`. Membership for these labels is based on a database query created from user-supplied criteria. In this first iteration, the allowed criteria are very simple: a label can specify either an IdP group or IdP department, and hosts with linked users with a matching group or department. Groundwork is laid here for more complex host vitals queries, including `and` and `or` logic, different data types and different kinds of vitals (rather than just the "foreign" vitals of which IdP is an example). Note that this PR does _not_ include the cron job that will trigger membership updating, and it doesn't include ; for sake of simplicity in review that will be done in a follow-on PR. ## Basic flow ### Creating a host vitals label 1. A new label is created via the API / GitOps with membership type `host_vitals` and a `criteria` property that's a JSON blob. Currently the JSON can only contain `vital` and `value` keys (and must contain those keys) 2. The server validates that the specified `vital` exists in our [set of known host vitals](https://github.com/fleetdm/fleet/pull/30278/files#diff-b6d4c48f2624b82c2567b2b88db1de51c6b152eeb261d40acfd5b63a890839b7R418-R436). 3. The server validates that the [criteria can be parsed into a query](https://github.com/fleetdm/fleet/pull/30278/files?diff=unified&w=1#diff-4ac4cfba8bed490e8ef125a0556f5417156f805017bfe93c6e2c61aa94ba8a8cR81-R86). This also happens during GitOps dry run. 4. The label is saved (criteria is saved as JSON in the db) ### Updating membership for a host vitals label 1. The label's criteria is used to generate a query to run on the _Fleet_ db. 1. For each vital criteria, check the vital type. Currently only foreign vitals are supported. 2. For foreign vitals, add its group to a set we keep track of. 3. Add a `WHERE` clause section for the vital and value, e.g. `end_user_idp_groups = ?` 4. Once we have all the `WHERE` clauses, create the query as `SELECT %s FROM %s` + any joins contributed by foreign vitals groups + `WHERE ` + all the `WHERE` clauses we just calculated. The `%s` provide some flexibility if we want to use these queries in other contexts. 2. Delete all existing label members 3. Do an `INSERT...SELECT` using the query we calculated from the label criteria. The query will be `SELECT <label id> as label_id, hosts.id FROM hosts JOIN ...` ## Future work ### Domestic vitals These can be anything that we already store in the `hosts` table. Domestic vitals won't add any `JOIN`s to the calculated label query, and will simply be e.g. `hosts.hostname = ?` ### Custom vitals We currently support an `additional_queries` config that will cause other queries to run on hosts. The data returned from these queries is stored in a `hosts_additional` table as a JSON blob. We can use MySQL JSON functions to match values in this data, e.g. `JSON_EXTRACT(host_additional, `$.some_custom_vital`) = ?` # Checklist for submitter If some of the following don't apply, delete the relevant line. <!-- Note that API documentation changes are now addressed by the product design team. --> - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. > I'll add the changelog item when I add the cron job PR - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [X] If database migrations are included, checked table schema to confirm autoupdate - For new Fleet configuration settings - [X] Verified that the setting can be managed via GitOps, or confirmed that the setting is explicitly being excluded from GitOps. If managing via Gitops: - [X] Verified that the setting is exported via `fleetctl generate-gitops` - [X] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) - For database migrations: - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [X] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [X] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). - [X] Added/updated automated tests - [X] Manual QA for all new/changed functionality
2025-06-30 14:58:58 +00:00
assert.Equal(t, `["admin"]`, string(queryValuesJson))
})
}