Elgato_dark/fleet
1
0
Fork 0
mirror of https://github.com/fleetdm/fleet synced 2026-05-23 08:58:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 111

Merge branch 'main' into 14415

Browse source
This commit is contained in:
Jacob Shandling 2023-12-11 13:52:45 -08:00
commit d2359675d4
225 changed files with 17948 additions and 1860 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.github/ISSUE_TEMPLATE/bug-report.md vendored
View file

@ -29,5 +29,5 @@ N/A
<!-- If this is a performance issue, follow these steps to generate and attach a debug archive: https://fleetdm.com/docs/using-fleet/monitoring-fleet#debugging-performance-issues -->
<!-- ### 🛠️ To fix ### -->
<!-- ### 🛠️ To fix -->
<!-- If this bug requires additional product design work, uncomment the heading above and add instructions to fix, Figma link, etc. here once design changes are settled. -->

6
.github/ISSUE_TEMPLATE/story.md vendored
View file

@ -7,9 +7,9 @@ assignees: ''
---
> **This issue's remaining effort can be completed in ≤1 sprint. It will be valuable even if nothing else ships.**
>
> It is [planned and ready](https://fleetdm.com/handbook/company/development-groups#making-changes) to implement. It is on the proper kanban board.
<!-- **This issue's remaining effort can be completed in ≤1 sprint. It will be valuable even if nothing else ships.**
It is [planned and ready](https://fleetdm.com/handbook/company/development-groups#making-changes) to implement. It is on the proper kanban board. -->
## Goal

4
.github/workflows/golangci-lint.yml vendored
View file

@ -62,3 +62,7 @@ jobs:
# version changes
go install github.com/golangci/golangci-lint/cmd/golangci-lint@411e0bbbd3096aa0ee2b924160629bdf2bc81d40 # v1.54.2
make lint-go
- name: Run cloner-check tool
run: |
go run ./tools/cloner-check/main.go -check

6
CHANGELOG.md
View file

@ -1,3 +1,9 @@
## Fleet 4.41.1 (Dec 7, 2023)
### Bug fix
* Fixed logging of results for scheduled queries configured outside of Fleet when `server_settings.query_reports_disabled` is set to `true`.
## Fleet 4.41.0 (Nov 28, 2023)
### Changes

12
Makefile
View file

@ -1,4 +1,4 @@
.PHONY: build clean clean-assets e2e-reset-db e2e-serve e2e-setup changelog db-reset db-backup db-restore
.PHONY: build clean clean-assets e2e-reset-db e2e-serve e2e-setup changelog db-reset db-backup db-restore check-go-cloner update-go-cloner
export GO111MODULE=on
@ -188,6 +188,16 @@ deps-js:
deps-go:
go mod download
# check that the generated files in tools/cloner-check/generated_files match
# the current version of the cloneable structures.
check-go-cloner:
go run ./tools/cloner-check/main.go --check
# update the files in tools/cloner-check/generated_files with the current
# version of the cloneable structures.
update-go-cloner:
go run ./tools/cloner-check/main.go --update
migration:
go run github.com/fleetdm/goose/cmd/goose -dir server/datastore/mysql/migrations/tables create $(name)
gofmt -w server/datastore/mysql/migrations/tables/*_$(name)*.go

41
articles/expeditioners-podcast-with-marcus-ransom.md Normal file
View file

@ -0,0 +1,41 @@
# ExpedITioners Podcast
## Marcus Ransom: The positive future of collaboration between vendors and Apple for enterprise
<iframe allow="autoplay *; encrypted-media *; fullscreen *; clipboard-write" frameborder="0" height="175" style="width:100%;max-width:660px;overflow:hidden;background:transparent;" sandbox="allow-forms allow-popups allow-same-origin allow-scripts allow-storage-access-by-user-activation allow-top-navigation-by-user-activation" src="https://embed.podcasts.apple.com/us/podcast/marcus-ransom-the-positive-future-of/id1641183838?i=1000638225150"></iframe>
Listen to the episode on [Apple](https://podcasts.apple.com/us/podcast/marcus-ransom-the-positive-future-of/id1641183838?i=1000638225150), [Spotify](https://open.spotify.com/episode/1DcqQhWvrrBTgGVJINvm0T?si=c0tw9fzCTxywp-6WpbZHJA), or [PodBean](https://expeditioners.podbean.com/e/marcus-ransom-the-positive-future-of-collaboration-between-vendors-and-apple-for-enterprise/).
### Show notes: 
We're joined by Marcus Ransom Sales Engineer at Jamf and one of the hosts of the Mac Admins podcast. In this episode, Zach and Marcus talk about the exciting future of Apple for enterprise and the MacAdmin community that supports it.
### Topics discussed:
- Marcus’ introduction to the Mac admin/IT world.
- Opportunities with the future of Apple products
- Changes throughout the history of the MacAdmin community.
- Integrating MacOS devices across every ecosystem.
- Frequent challenges and opportunities seen across the industry.
- Enabling developers to build the tools your company needs for its customers.
- Thoughts on the future of Mac IT.
- Apple instituting actionable and useful feedback from vendors.
- The importance of sharing information across the industry and community.
### Resources mentioned:
- [Xworld Australia ](https://auc.edu.au/xworld/about/)
- [MacAdmins Slack](https://www.macadmins.org/)
- [MacAdmins podcast](https://podcast.macadmins.org/)
- [MacAdmins Foundation](https://www.macadmins.org/about-the-mac-admins-foundation)
### Where to get in touch:
- [LinkedIn](https://www.linkedin.com/in/marcusransom/)
<meta name="category" value="podcasts">
<meta name="authorGitHubUsername" value="zwass">
<meta name="authorFullName" value="Zach Wasserman">
<meta name="publishedOn" value="2023-12-11">
<meta name="articleTitle" value="ExpedITioners podcast with Marcus Ransom">
<meta name="articleImageUrl" value="../website/assets/images/articles/expeditioners-podcast-ep7-1600x900@2x.jpg">

1
changes/13034-host-mdm-idp-email Normal file
View file

@ -0,0 +1 @@
- Added capability to look up hosts based on IdP email.

1
changes/14778-agent-option Normal file
View file

@ -0,0 +1 @@
- Fixes a validation bug that allowed the agent options `overrides.platform` field to be set to `null`.

1
changes/14779-fix-software-deadlock Normal file
View file

@ -0,0 +1 @@
* Fix possible deadlocks when deleting `software` during data ingestion (found during load test).

2
changes/14920-device-health Normal file
View file

@ -0,0 +1,2 @@
- Adds a `GET /hosts/{id}/health` endpoint which reports back some key data about host health such
as vulnerable software and failing policies.

1
changes/15146-filter-query-reports-by-user Normal file
View file

@ -0,0 +1 @@
- query reports now only show results for hosts the user has permission to

1
changes/15168-scheduled-outside-fleet
View file

@ -1 +0,0 @@
* Fixed logging of results for scheduled queries configured outside of Fleet, when `server_settings.query_reports_disabled` is set to `true`.

4
changes/15229-list-software-versions Normal file
View file

@ -0,0 +1,4 @@
- Added `GET software/versions` endpoint to list and filter software versions.
- Added `GET software/versions/{id}` endpoint to get a specific software version.
- Deprecated `GET software` endpoint.
- Deprecated `GET software/{id}` endpoint.

1
changes/15430-change-script-timeout-error-code-to-408 Normal file
View file

@ -0,0 +1 @@
- POST /api/v1/fleet/scripts/run/sync timeout (longer than 60 seconds) will now return error code: 408 instead of 504

1
changes/15472-performance-stat-types Normal file
View file

@ -0,0 +1 @@
Changed query performance statistics to uint64 to match osquery reports.

2
changes/issue-14959-remove-feature-flag Normal file
View file

@ -0,0 +1,2 @@
* Removed the `FLEET_DEV_MDM_ENABLED` feature flag that allowed enabling Windows MDM. The feature flag is not used anymore, and Windows MDM can be enabled without it.

1
changes/issue-14979-improve-safety-checks-for-cached-items Normal file
View file

@ -0,0 +1 @@
* Improved safety and efficiency of implementation of cacheable data.

1
changes/issue-15345-filter-hosts-by-software Normal file
View file

@ -0,0 +1 @@
* Added ability to filter hosts by `software_version_id` and `software_title_id` for the "list hosts", "count hosts" and "get hosts report in CSV" endpoints.

1
changes/issue-15406-fleetctl-get-software Normal file
View file

@ -0,0 +1 @@
* Updated `fleetctl get software` to list software titles, and add optional `--versions` flag to list software versions.

1
changes/issue-15438-firehose-addon Normal file
View file

@ -0,0 +1 @@
* Update firehose delivery addon to use latest module version, this includes breaking changes to previous configurations as the default prefixes have been changed to natively support time-partitioned Athena table creation

2
charts/fleet/Chart.yaml
View file

@ -8,7 +8,7 @@ version: v6.0.1
home: https://github.com/fleetdm/fleet
sources:
- https://github.com/fleetdm/fleet.git
appVersion: v4.41.0
appVersion: v4.41.1
dependencies:
- name: mysql
condition: mysql.enabled

2
charts/fleet/values.yaml
View file

@ -2,7 +2,7 @@
# All settings related to how Fleet is deployed in Kubernetes
hostName: fleet.localhost
replicas: 3 # The number of Fleet instances to deploy
imageTag: v4.41.0 # Version of Fleet to deploy
imageTag: v4.41.1 # Version of Fleet to deploy
podAnnotations: {} # Additional annotations to add to the Fleet pod
serviceAccountAnnotations: {} # Additional annotations to add to the Fleet service account
resources:

4
cmd/fleet/serve.go
View file

@ -553,8 +553,8 @@ the way that the Fleet server works.
wstepCertManager microsoft_mdm.CertManager
)
// Configuring WSTEP certs if Windows MDM feature flag is enabled
if configpkg.IsMDMFeatureFlagEnabled() && config.MDM.IsMicrosoftWSTEPSet() {
// Configuring WSTEP certs
if config.MDM.IsMicrosoftWSTEPSet() {
_, crtPEM, keyPEM, err := config.MDM.MicrosoftWSTEP()
if err != nil {
initFatal(err, "validate Microsoft WSTEP certificate and key")

20
cmd/fleetctl/apply_test.go
View file

@ -14,7 +14,6 @@ import (
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"testing"
"time"
@ -902,8 +901,6 @@ func mobileconfigForTest(name, identifier string) []byte {
}
func TestApplyAsGitOps(t *testing.T) {
t.Setenv("FLEET_DEV_MDM_ENABLED", "1")
enqueuer := new(nanomdm_mock.Storage)
license := &fleet.LicenseInfo{Tier: fleet.TierPremium, Expiration: time.Now().Add(24 * time.Hour)}
@ -3338,17 +3335,6 @@ spec:
`, macSetupFile),
wantErr: `macOS MDM isn't turned on.`,
},
{
desc: "app config enable windows mdm without feature flag",
spec: `
apiVersion: v1
kind: config
spec:
mdm:
windows_enabled_and_configured: true
`,
wantErr: `422 Validation Failed: cannot enable Windows MDM without the feature flag explicitly enabled`,
},
{
desc: "app config enable windows mdm without WSTEP",
spec: `
@ -3370,12 +3356,6 @@ spec:
license := &fleet.LicenseInfo{Tier: fleet.TierPremium, Expiration: time.Now().Add(24 * time.Hour)}
for _, c := range cases {
t.Run(c.desc, func(t *testing.T) {
// bit hacky, but since the env var is temporary while Windows MDM is in beta,
// didn't want to add a field to the test cases just for this.
if strings.Contains(c.desc, "WSTEP") {
t.Setenv("FLEET_DEV_MDM_ENABLED", "1")
}
_, ds := runServerWithMockedDS(t, &service.TestServerOpts{License: license})
setupDS(ds)
filename := writeTmpYml(t, c.spec)

137
cmd/fleetctl/get.go
View file

@ -1213,12 +1213,16 @@ func getSoftwareCommand() *cli.Command {
return &cli.Command{
Name: "software",
Aliases: []string{"s"},
Usage: "List software",
Usage: "List software titles",
Flags: []cli.Flag{
&cli.UintFlag{
Name: teamFlagName,
Usage: "Only list software of hosts that belong to the specified team",
},
&cli.BoolFlag{
Name: "versions",
Usage: "List all software versions",
},
jsonFlag(),
yamlFlag(),
configFlag(),
@ -1242,49 +1246,104 @@ func getSoftwareCommand() *cli.Command {
query.Set("team_id", strconv.FormatUint(uint64(teamID), 10))
}
software, err := client.ListSoftware(query.Encode())
if err != nil {
return fmt.Errorf("could not list software: %w", err)
if c.Bool("versions") {
return printSoftwareVersions(c, client, query)
}
if len(software) == 0 {
log(c, "No software found")
return nil
}
if c.Bool(jsonFlagName) || c.Bool(yamlFlagName) {
spec := specGeneric{
Kind: "software",
Version: "1",
Spec: software,
}
err = printSpec(c, spec)
if err != nil {
return err
}
return nil
}
// Default to printing as table
data := [][]string{}
for _, s := range software {
data = append(data, []string{
s.Name,
s.Version,
s.Source,
s.GenerateCPE,
fmt.Sprint(len(s.Vulnerabilities)),
})
}
columns := []string{"Name", "Version", "Source", "CPE", "# of CVEs"}
printTable(c, columns, data)
return nil
return printSoftwareTitles(c, client, query)
},
}
}
func printSoftwareVersions(c *cli.Context, client *service.Client, query url.Values) error {
software, err := client.ListSoftwareVersions(query.Encode())
if err != nil {
return fmt.Errorf("could not list software versions: %w", err)
}
if len(software) == 0 {
log(c, "No software versions found")
return nil
}
if c.Bool(jsonFlagName) || c.Bool(yamlFlagName) {
spec := specGeneric{
Kind: "software",
Version: "1",
Spec: software,
}
err = printSpec(c, spec)
if err != nil {
return err
}
return nil
}
// Default to printing as table
data := [][]string{}
for _, s := range software {
data = append(data, []string{
s.Name,
s.Version,
s.Source,
fmt.Sprintf("%d vulnerabilities", len(s.Vulnerabilities)),
fmt.Sprint(s.HostsCount),
})
}
columns := []string{"Name", "Version", "Type", "Vulnerabilities", "Hosts"}
printTable(c, columns, data)
return nil
}
func printSoftwareTitles(c *cli.Context, client *service.Client, query url.Values) error {
software, err := client.ListSoftwareTitles(query.Encode())
if err != nil {
return fmt.Errorf("could not list software titles: %w", err)
}
if len(software) == 0 {
log(c, "No software titles found")
return nil
}
if c.Bool(jsonFlagName) || c.Bool(yamlFlagName) {
spec := specGeneric{
Kind: "software_title",
Version: "1",
Spec: software,
}
err = printSpec(c, spec)
if err != nil {
return err
}
return nil
}
// Default to printing as table
data := [][]string{}
for _, s := range software {
vulns := make(map[string]bool)
for _, ver := range s.Versions {
if ver.Vulnerabilities != nil {
for _, vuln := range *ver.Vulnerabilities {
vulns[vuln] = true
}
}
}
data = append(data, []string{
s.Name,
fmt.Sprintf("%d versions", s.VersionsCount),
s.Source,
fmt.Sprintf("%d vulnerabilities", len(vulns)),
fmt.Sprint(s.HostsCount),
})
}
columns := []string{"Name", "Versions", "Type", "Vulnerabilities", "Hosts"}
printTable(c, columns, data)
return nil
}
func getMDMAppleCommand() *cli.Command {
return &cli.Command{
Name: "mdm-apple",

198
cmd/fleetctl/get_test.go
View file

@ -602,7 +602,163 @@ func TestGetConfig(t *testing.T) {
})
}
func TestGetSoftware(t *testing.T) {
func TestGetSoftwareTitles(t *testing.T) {
_, ds := runServerWithMockedDS(t, &service.TestServerOpts{
License: &fleet.LicenseInfo{
Tier: fleet.TierPremium,
Expiration: time.Now().Add(24 * time.Hour),
},
})
var gotTeamID *uint
ds.ListSoftwareTitlesFunc = func(ctx context.Context, opt fleet.SoftwareTitleListOptions) ([]fleet.SoftwareTitle, int, *fleet.PaginationMetadata, error) {
gotTeamID = opt.TeamID
return []fleet.SoftwareTitle{
{
Name: "foo",
Source: "chrome_extensions",
HostsCount: 2,
VersionsCount: 3,
Versions: []fleet.SoftwareVersion{
{
Version: "0.0.1",
Vulnerabilities: &fleet.SliceString{"cve-123-456-001", "cve-123-456-002"},
},
{
Version: "0.0.2",
Vulnerabilities: &fleet.SliceString{"cve-123-456-001"},
},
{
Version: "0.0.3",
Vulnerabilities: &fleet.SliceString{"cve-123-456-003"},
},
},
},
{
Name: "bar",
Source: "deb_packages",
HostsCount: 0,
VersionsCount: 1,
Versions: []fleet.SoftwareVersion{
{
Version: "0.0.3",
Vulnerabilities: nil,
},
},
},
}, 0, nil, nil
}
expected := `+------+------------+-------------------+-------------------+-------+
| NAME | VERSIONS | TYPE | VULNERABILITIES | HOSTS |
+------+------------+-------------------+-------------------+-------+
| foo | 3 versions | chrome_extensions | 3 vulnerabilities | 2 |
+------+------------+-------------------+-------------------+-------+
| bar | 1 versions | deb_packages | 0 vulnerabilities | 0 |
+------+------------+-------------------+-------------------+-------+
`
expectedYaml := `---
apiVersion: "1"
kind: software_title
spec:
- hosts_count: 2
id: 0
name: foo
source: chrome_extensions
browser: ""
versions:
- id: 0
version: 0.0.1
vulnerabilities:
- cve-123-456-001
- cve-123-456-002
- id: 0
version: 0.0.2
vulnerabilities:
- cve-123-456-001
- id: 0
version: 0.0.3
vulnerabilities:
- cve-123-456-003
versions_count: 3
- hosts_count: 0
id: 0
name: bar
source: deb_packages
browser: ""
versions:
- id: 0
version: 0.0.3
versions_count: 1
`
expectedJson := `
{
"kind": "software_title",
"apiVersion": "1",
"spec": [
{
"id": 0,
"name": "foo",
"source": "chrome_extensions",
"browser": "",
"hosts_count": 2,
"versions_count": 3,
"versions": [
{
"id": 0,
"version": "0.0.1",
"vulnerabilities": [
"cve-123-456-001",
"cve-123-456-002"
]
},
{
"id": 0,
"version": "0.0.2",
"vulnerabilities": [
"cve-123-456-001"
]
},
{
"id": 0,
"version": "0.0.3",
"vulnerabilities": [
"cve-123-456-003"
]
}
]
},
{
"id": 0,
"name": "bar",
"source": "deb_packages",
"browser": "",
"hosts_count": 0,
"versions_count": 1,
"versions": [
{
"id": 0,
"version": "0.0.3"
}
]
}
]
}
`
assert.Equal(t, expected, runAppForTest(t, []string{"get", "software"}))
assert.YAMLEq(t, expectedYaml, runAppForTest(t, []string{"get", "software", "--yaml"}))
assert.JSONEq(t, expectedJson, runAppForTest(t, []string{"get", "software", "--json"}))
runAppForTest(t, []string{"get", "software", "--json", "--team", "999"})
require.NotNil(t, gotTeamID)
assert.Equal(t, uint(999), *gotTeamID)
}
func TestGetSoftwareVersions(t *testing.T) {
_, ds := runServerWithMockedDS(t)
foo001 := fleet.Software{
@ -623,17 +779,21 @@ func TestGetSoftware(t *testing.T) {
return []fleet.Software{foo001, foo002, foo003, bar003}, nil
}
expected := `+------+---------+-------------------+--------------------------+-----------+
| NAME | VERSION | SOURCE | CPE | # OF CVES |
+------+---------+-------------------+--------------------------+-----------+
| foo | 0.0.1 | chrome_extensions | somecpe | 2 |
+------+---------+-------------------+--------------------------+-----------+
| foo | 0.0.2 | chrome_extensions | | 0 |
+------+---------+-------------------+--------------------------+-----------+
| foo | 0.0.3 | chrome_extensions | someothercpewithoutvulns | 0 |
+------+---------+-------------------+--------------------------+-----------+
| bar | 0.0.3 | deb_packages | | 0 |
+------+---------+-------------------+--------------------------+-----------+
ds.CountSoftwareFunc = func(ctx context.Context, opt fleet.SoftwareListOptions) (int, error) {
return 4, nil
}
expected := `+------+---------+-------------------+-------------------+-------+
| NAME | VERSION | TYPE | VULNERABILITIES | HOSTS |
+------+---------+-------------------+-------------------+-------+
| foo | 0.0.1 | chrome_extensions | 2 vulnerabilities | 0 |
+------+---------+-------------------+-------------------+-------+
| foo | 0.0.2 | chrome_extensions | 0 vulnerabilities | 0 |
+------+---------+-------------------+-------------------+-------+
| foo | 0.0.3 | chrome_extensions | 0 vulnerabilities | 0 |
+------+---------+-------------------+-------------------+-------+
| bar | 0.0.3 | deb_packages | 0 vulnerabilities | 0 |
+------+---------+-------------------+-------------------+-------+
`
expectedYaml := `---
@ -644,6 +804,7 @@ spec:
id: 0
name: foo
source: chrome_extensions
browser: ""
version: 0.0.1
vulnerabilities:
- cve: cve-321-432-543
@ -662,6 +823,7 @@ spec:
id: 0
name: foo
source: chrome_extensions
browser: ""
version: 0.0.3
vulnerabilities: null
- bundle_identifier: bundle
@ -669,6 +831,7 @@ spec:
id: 0
name: bar
source: deb_packages
browser: ""
version: 0.0.3
vulnerabilities: null
`
@ -683,6 +846,7 @@ spec:
"name": "foo",
"version": "0.0.1",
"source": "chrome_extensions",
"browser": "",
"generated_cpe": "somecpe",
"vulnerabilities": [
{
@ -710,6 +874,7 @@ spec:
"name": "foo",
"version": "0.0.3",
"source": "chrome_extensions",
"browser": "",
"generated_cpe": "someothercpewithoutvulns",
"vulnerabilities": null
},
@ -719,6 +884,7 @@ spec:
"version": "0.0.3",
"bundle_identifier": "bundle",
"source": "deb_packages",
"browser": "",
"generated_cpe": "",
"vulnerabilities": null
}
@ -726,11 +892,11 @@ spec:
}
`
assert.Equal(t, expected, runAppForTest(t, []string{"get", "software"}))
assert.YAMLEq(t, expectedYaml, runAppForTest(t, []string{"get", "software", "--yaml"}))
assert.JSONEq(t, expectedJson, runAppForTest(t, []string{"get", "software", "--json"}))
assert.Equal(t, expected, runAppForTest(t, []string{"get", "software", "--versions"}))
assert.YAMLEq(t, expectedYaml, runAppForTest(t, []string{"get", "software", "--versions", "--yaml"}))
assert.JSONEq(t, expectedJson, runAppForTest(t, []string{"get", "software", "--versions", "--json"}))
runAppForTest(t, []string{"get", "software", "--json", "--team", "999"})
runAppForTest(t, []string{"get", "software", "--versions", "--json", "--team", "999"})
require.NotNil(t, gotTeamID)
assert.Equal(t, uint(999), *gotTeamID)
}

16
codecov.yml
View file

@ -1,16 +1,22 @@
coverage:
status:
project: false
patch: false
project:
default:
target: auto # aim for the same coverage as the base of the PR
threshold: 1% # allow a decrease of up to 1%
patch:
default:
target: auto # aim for the same coverage as the base of the PR
threshold: 1% # allow a decrease of up to 1%
flag_management:
default_rules:
carryforward: true
statuses:
- type: project
informational: true
informational: false # change this to true if you do not want to enforce covergae requirement
- type: patch
informational: true
informational: false # change this to true if you do not want to enforce covergae requirement
individual_flags:
- name: backend
paths:
@ -21,4 +27,4 @@ flag_management:
- orbit/
- name: frontend
paths:
- frontend/
- frontend/

14
docs/Configuration/agent-configuration.md
View file

@ -445,20 +445,6 @@ overrides:
- "last_modified"
```
## Command line flags
> Requires Fleet v4.22.0 or later and Orbit v1.3.0 or later**
In the `command_line_flags` key, you can update the osquery flags of your Orbit enrolled agents.
```yaml
agent_options:
config:
overrides:
command_line_flags:
enable_file_events: true
```
## Update agent options in Fleet UI
<!-- Heading is kept so that the link from the Fleet UI still works -->

33
docs/Contributing/API-for-contributors.md
View file

@ -2708,6 +2708,39 @@ If the Fleet instance is provided required parameters to complete setup.
## Scripts
### Run script asynchronously
_Available in Fleet Premium_
Creates a script execution request and returns the execution identifier to retrieve results at a later time.
`POST /api/v1/fleet/scripts/run`
#### Parameters
| Name | Type | In | Description |
| ---- | ------- | ---- | -------------------------------------------- |
| host_id | integer | body | **Required**. The ID of the host to run the script on. |
| script_id | integer | body | The ID of the existing saved script to run. Only one of either `script_id` or `script_contents` can be included in the request; omit this parameter if using `script_contents`. |
| script_contents | string | body | The contents of the script to run. Only one of either `script_id` or `script_contents` can be included in the request; omit this parameter if using `script_id`. |
> Note that if both `script_id` and `script_contents` are included in the request, this endpoint will respond with an error.
#### Example
`POST /api/v1/fleet/scripts/run`
##### Default response
`Status: 202`
```json
{
"host_id": 1227,
"execution_id": "e797d6c6-3aae-11ee-be56-0242ac120002"
}
```
### Batch-apply scripts
_Available in Fleet Premium_

246
docs/Contributing/Releasing-Fleet.md
View file

@ -1,162 +1,168 @@
# Releasing Fleet
## Release process
This section outlines the release process at Fleet. The current release cadence is once every three weeks. Patch versions are released as needed.
This section outlines the release process at Fleet after all EMs have certified that they are ready for release.
The current release cadence is once every three weeks. Patch versions are released as needed.
#### Release candidate process
Major, minor, and patch releases go through the same release candidate and smoke testing process prior to release.
1. The release ritual or patch release DRI creates a release candidate branch. If this is a major or minor version, they prepend the branch name with `prepare-v`. If it is a patch version, they prepend the branch name with `patch-v`. For example, `prepare-v4.0.0` or `patch-v4.0.1`.
> When a `prepare-*` or `patch-*` branch is pushed, the [Docker publish Action](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-snapshot-fleet.yaml) will be invoked to push a container image for smoke testing with `fleetctl preview` (eg. `fleetctl preview --tag patch-fleet-v4.3.1`).
2. The release DRI pushes the branch to github.com/fleetdm/fleet:
```
git push origin prepare-fleet-v4.0.0
```
3. The release DRI checks in the GitHub UI that Actions ran successfully for this branch.
4. The release DRI adds a comment to the related smoke testing issue [created during the freeze ritual](https://fleetdm.com/handbook/engineering#create-release-qa-issue) confirming that the release candidate is ready for testing.
5. QA is conducted on the release candidate following the steps in the smoke testing issue. If smoke testing passes successfully, the EM for each relevant product group adds a comment to the issue certifying that their product group has completed smoke testing.
6. When the smoke testing issue has been certified, the release DRI publishes the release.
### Preparing a minor or major release
### Prepare a new version of Fleet
Note: Please prefix versions with `fleet-v` (e.g., `fleet-v4.0.0`) in git tags, Helm charts, and NPM configs.
1. Update the [CHANGELOG](https://github.com/fleetdm/fleet/blob/main/CHANGELOG.md) with the changes you made since the last
Fleet release. Use `make changelog` to pull the changes files into `CHANGELOG.md`, then manually
edit. When editing, order the most relevant/important changes at the time and try to make the
tone and syntax of the written language match throughout the document. `make changelog` will stage all changes
file entries for deletion with the commit.
1. Update the [CHANGELOG](https://github.com/fleetdm/fleet/blob/main/CHANGELOG.md) with the changes you made since the last Fleet release. Use `make changelog` to pull the change files into `CHANGELOG.md`, then manually edit. When editing, order the most relevant/important changes at the top and make sure each line is in the past tense. `make changelog` will stage all change file entries for deletion with the commit.
Add a "Performance" section below the list of changes. This section should summarize the number of
hosts that the Fleet server can handle, call out if this number has
changed since the last release, and list the infrastructure used in the load testing environment.
2. Update version numbers in the relevant files:
Update version numbers in the relevant files:
- [fleetctl package.json](https://github.com/fleetdm/fleet/blob/main/tools/fleetctl-npm/package.json) (do not yet `npm publish`)
- [Helm chart.yaml](https://github.com/fleetdm/fleet/blob/main/charts/fleet/Chart.yaml) and [values file](https://github.com/fleetdm/fleet/blob/main/charts/fleet/values.yaml)
- Terraform variables ([AWS](https://github.com/fleetdm/fleet/blob/main/infrastructure/dogfood/terraform/aws/variables.tf)/[GCP](https://github.com/fleetdm/fleet/blob/main/infrastructure/dogfood/terraform/gcp/variables.tf))
- [Kubernetes `deployment.yml` example file](https://github.com/fleetdm/fleet/blob/main/docs/Deploy/Deploying-Fleet-on-Kubernetes.md)
- All Terraform (*.tf) files referencing the previous version of Fleet.
- [fleetctl package.json](https://github.com/fleetdm/fleet/blob/main/tools/fleetctl-npm/package.json) (do not yet `npm publish`)
- [Helm chart.yaml](https://github.com/fleetdm/fleet/blob/main/charts/fleet/Chart.yaml) and [values file](https://github.com/fleetdm/fleet/blob/main/charts/fleet/values.yaml)
- Terraform variables ([AWS](https://github.com/fleetdm/fleet/blob/main/infrastructure/dogfood/terraform/aws/variables.tf)/[GCP](https://github.com/fleetdm/fleet/blob/main/infrastructure/dogfood/terraform/gcp/variables.tf))
- [Kubernetes `deployment.yml` example file](https://github.com/fleetdm/fleet/blob/main/docs/Deploy/Deploying-Fleet-on-Kubernetes.md)
Commit these changes via Pull Request and pull the changes on the `main` branch locally.
Commit these changes via Pull Request and pull the changes on the `main` branch locally. Check that
`HEAD` of the `main` branch points to the commit with these changes.
### Prepare a minor or major release
2. Create release candidate branch and conduct smoke testing as documented above.
1. Complete the steps above to [prepare a new version of Fleet](#prepare-a-new-version-of-fleet).
3. Tag and push the new release in Git:
```sh
git tag fleet-v<VERSION>
git push origin fleet-v<VERSION>
```
2. Create a new branch. Minor or major release branches should be prefixed with `prepare-`. In this example we are creating `v4.3.0`:
Note that `origin` may be `upstream` depending on your `git remote` configuration. The intent here
is to push the new tag to the `github.com/fleetdm/fleet` repository.
```sh
git checkout main
git checkout --branch prepare-fleet-v4.3.0
```
After the tag is pushed, GitHub Actions will automatically begin building the new release.
3. [Create release candidate](#create-release-candidate).
***
4. [Complete release QA](#complete-release-qa).
Wait while GitHub Actions creates and uploads the artifacts.
5. Tag and push the new release:
***
```sh
git tag fleet-v<VERSION>
git push origin fleet-v<VERSION>
```
When the Actions Workflow has been completed:
Note that `origin` may be `upstream` depending on your `git remote` configuration. The intent here
is to push the new tag to the `github.com/fleetdm/fleet` repository.
4. Edit the draft release on the [GitHub releases page](https://github.com/fleetdm/fleet/releases).
Use the version number as the release title. Use the below template for the release description
(replace items in <> with the appropriate values):
```md
### Changes
After the tag is pushed, GitHub Actions will automatically begin building the new release.
<COPY FROM CHANGELOG>
***
### Upgrading
Wait while GitHub Actions creates and uploads the artifacts.
Please visit our [update guide](https://fleetdm.com/docs/deploying/upgrading-fleet) for upgrade instructions.
***
### Documentation
When the Actions Workflow has been completed, [publish the new version of Fleet](#publish-a-new-version-of-fleet).
Documentation for Fleet is available at [fleetdm.com/docs](https://fleetdm.com/docs).
### Binary Checksum
**SHA256**
<COPY FROM checksums.txt>
```
When editing is complete, publish the release.
5. Publish the new version of `fleetctl` on NPM. Run `npm publish` in the
[fleetctl-npm](https://github.com/fleetdm/fleet/tree/main/tools/fleetctl-npm) directory. Note that NPM does not allow replacing a
package without creating a new version number. Take care to get things correct before running
`npm publish`!
> If releasing a "prerelease" of Fleet, run `npm publish --tag prerelease`. This way, you can
> publish a prerelease of fleetctl while the most recent fleetctl npm package, available for public
> download, is still the latest _official_ release.
6. Deploy the new version to Fleet's internal dogfood instance: https://fleetdm.com/handbook/engineering#deploying-to-dogfood.
7. In the #g-infra Slack channel, notify the @infrastructure-oncall of the release. This way, the @infrastructure-oncall individual can deploy the new version.
8. Announce the release in the #general channel.
9. Announce the release in the #fleet channel of [osquery
Slack](https://fleetdm.com/slack) and
update the channel's topic with the link to this release. Using `@here` requires admin
permissions, so typically this announcement will be done by `@zwass`.
Announce the release via blog post (on Medium) and Twitter (linking to blog post).
### Preparing a patch release
### Prepare a patch release
A patch release is required when a critical bug is found. Critical bugs are defined in [our handbook](https://fleetdm.com/handbook/quality#critical-bugs).
#### Process
1. Complete the steps above to [prepare a new version of Fleet](#prepare-a-new-version-of-fleet).
1. The DRI for release testing/QA notifies the [directly responsible individual (DRI) for creating the patch release branch](https://fleetdm.com/handbook/engineering#rituals) to create the new branch, starting from the git tag of the prior release. Patch branches should be prefixed with `patch-`. In this example we are creating `4.3.1`:
```sh
git checkout fleet-v4.3.0
git checkout --branch patch-fleet-v4.3.1
```
2. Create a new branch, starting from the git tag of the prior release. Patch branches should be prefixed with `patch-`. In this example we are creating `v4.3.1`:
```sh
git checkout fleet-v4.3.0
git checkout --branch patch-fleet-v4.3.1
```
2. The DRI for creating the patch release branch cherry picks the necessary commits into the new branch:
```sh
git cherry-pick d34db33f
```
3. Cherry picks the necessary commits from `main` into the new branch:
```sh
git cherry-pick d34db33f
```
3. The DRI for creating the patch release branch pushes the branch to github.com/fleetdm/fleet:
> Make sure to cherry-pick the commit containing changelog and version number updates.
4. **Important!** Any migrations that are not cherry-picked in a patch must have a _later_ timestamp than migrations that were cherry-picked. If there are new migrations that were not cherry-picked, verify that those migrations have later timestamps. If they do not, submit a new Pull Request to increase the timestamps and ensure that migrations are run in the appropriate order.
5. [Create release candidate](#create-release-candidate).
6. [Complete release QA](#complete-release-qa).
7. Tag and push the new release in Git:
```sh
git tag fleet-v-v4.3.1
git push origin fleet-v-4.3.1
```
Note that `origin` may be `upstream` depending on your `git remote` configuration. The intent here
is to push the new tag to the `github.com/fleetdm/fleet` repository.
After the tag is pushed, GitHub Actions will automatically begin building the new release.
***
Wait while GitHub Actions creates and uploads the artifacts.
***
When the Actions Workflow has been completed, [publish the new version of Fleet](#publish-a-new-version-of-fleet).
### Create release candidate
1. Push a branch containing new commits to [fleetdm/fleet](https://github.com/fleetdm/fleet) that begins with `prepare-*` or `patch-*`.
```sh
git push origin patch-fleet-v4.3.1
```
When a `patch-*` branch is pushed, the [Docker publish
Action](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-snapshot-fleet.yaml) will
be invoked to push a container image for QA with `fleetctl preview` (eg. `fleetctl preview --tag patch-fleet-v4.3.1`).
> When a `prepare-*` or `patch-*` branch is pushed, the [Docker publish Action](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-snapshot-fleet.yaml) will run and create a container image for QA with `fleetctl preview` (eg. `fleetctl preview --tag patch-fleet-v4.3.1`).
4. The patch release DRI checks in the GitHub UI that Actions ran successfully for this branch.
2. Check the [Docker Publish GitHub action](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-snapshot-fleet.yaml) to confirm it completes successfully for this branch.
5. The patch release DRI notifies the [DRI for release testing/QA](https://fleetdm.com/handbook/product#rituals) that the branch is available for completing [smoke tests](https://github.com/fleetdm/fleet/blob/main/.github/ISSUE_TEMPLATE/smoke-tests.md).
3. Create a [Release QA](https://github.com/fleetdm/fleet/blob/main/.github/ISSUE_TEMPLATE/smoke-tests.md) issue. Populate the version and browsers, and assign to the QA person leading the release. Add the appropriate [product group label](https://fleetdm.com/handbook/company/product-groups), and `:release` label, so that it appears on the product group's release board.
6. The DRI for release testing/QA makes sure the standard release instructions at the top of this document are followed. Be sure that modifications to the changelog and config files are commited _on the `patch-*` branch_.
4. Notify QA that the release candidate is ready for (release QA)[#complete-release-qa].
7. The DRI for release testing/QA notifies the [DRI for the release ritual](https://fleetdm.com/handbook/engineering#rituals) that the patch release is ready. The DRI for the release ritual releases the patch.
### Complete release QA
8. The DRI for creating the patch release branch cherry-picks the commit containing the changelog updates into a new branch, and merges that commit into `main` through a Pull Request.
1. Move the release QA issue into the "In progress" column on the release board.
9. **Important!** The DRI for creating the patch release branch manually checks the database migrations. Any migrations that are not cherry-picked in a patch must have a _later_ timestamp than migrations that were cherry-picked. If there are new migrations that were not cherry-picked, verify that those migrations have later timestamps. If they do not, submit a new Pull Request to increase the timestamps and ensure that migrations are run in the appropriate order.
2. Complete each item listed in the release QA issue.
3. If bugs are found, file bug tickets and notify your EM.
4. When all items are completed with no bugs remaining, move the issue to the "Ready for release" column on the release board and notify your EM.
### Publish a new version of Fleet
1. Edit the draft release on the [GitHub releases page](https://github.com/fleetdm/fleet/releases).Use the version number as the release title. Use the below template for the release description
(replace items in <> with the appropriate values):
```md
### Changes
<COPY FROM CHANGELOG>
### Upgrading
Please visit our [update guide](https://fleetdm.com/docs/deploying/upgrading-fleet) for upgrade instructions.
### Documentation
Documentation for Fleet is available at [fleetdm.com/docs](https://fleetdm.com/docs).
### Binary Checksum
**SHA256**
<COPY FROM checksums.txt>
```
When editing is complete, publish the release.
2. Publish the new version of `fleetctl` on NPM. Run `npm publish` in the [fleetctl-npm](https://github.com/fleetdm/fleet/tree/main/tools/fleetctl-npm) directory. Note that NPM does not allow replacing a package without creating a new version number. Take care to get things correct before running `npm publish`!
> If releasing a "prerelease" of Fleet, run `npm publish --tag prerelease`. This way, you can publish a prerelease of fleetctl while the most recent fleetctl npm package, available for public download, is still the latest _official_ release.
3. Deploy the new version to Fleet's internal dogfood instance: https://fleetdm.com/handbook/engineering#deploying-to-dogfood.
4. In the #help-infrastructure Slack channel, notify the @infrastructure-oncall of the release. The @infrastructure-oncall will schedule time to upgrade our managed cloud to the new version.
5. Announce the release in the #fleet channel of [osquery Slack](https://fleetdm.com/slack) by updating the channel topic with the link to this release.
6. Announce the release in the #general channel by copying and pasting the osquery Slack channel topic.
<meta name="pageOrderInSection" value="500">
<meta name="description" value="Learn how new versions of Fleet are tested and released.">
<meta name="description" value="Learn how new versions of Fleet are created, tested, and released.">

202
docs/REST API/rest-api.md
View file

@ -699,7 +699,7 @@ Retrieves a list of the non expired carves. Carve contents remain available for
Retrieves the specified carve.
`GET /api/v1/fleet/carves/{id}`
`GET /api/v1/fleet/carves/:id`
#### Parameters
@ -738,7 +738,7 @@ Retrieves the specified carve.
Retrieves the specified carve block. This endpoint retrieves the data that was carved.
`GET /api/v1/fleet/carves/{id}/block/{block_id}`
`GET /api/v1/fleet/carves/:id/block/:block_id`
#### Parameters
@ -1394,7 +1394,7 @@ Delete all global enroll secrets.
Returns the valid team enroll secrets.
`GET /api/v1/fleet/teams/{id}/secrets`
`GET /api/v1/fleet/teams/:id/secrets`
#### Parameters
@ -1425,7 +1425,7 @@ None.
Replaces all existing team enroll secrets.
`PATCH /api/v1/fleet/teams/{id}/secrets`
`PATCH /api/v1/fleet/teams/:id/secrets`
#### Parameters
@ -1623,7 +1623,7 @@ Returns a list of the active invitations in Fleet.
Delete the specified invite from Fleet.
`DELETE /api/v1/fleet/invites/{id}`
`DELETE /api/v1/fleet/invites/:id`
#### Parameters
@ -1633,7 +1633,7 @@ Delete the specified invite from Fleet.
#### Example
`DELETE /api/v1/fleet/invites/{id}`
`DELETE /api/v1/fleet/invites/123`
##### Default response
@ -1644,7 +1644,7 @@ Delete the specified invite from Fleet.
Verify the specified invite.
`GET /api/v1/fleet/invites/{token}`
`GET /api/v1/fleet/invites/:token`
#### Parameters
@ -1654,7 +1654,7 @@ Verify the specified invite.
#### Example
`GET /api/v1/fleet/invites/{token}`
`GET /api/v1/fleet/invites/abcdef012456789`
##### Default response
@ -1693,7 +1693,7 @@ Verify the specified invite.
### Update invite
`PATCH /api/v1/fleet/invites/{id}`
`PATCH /api/v1/fleet/invites/:id`
#### Parameters
@ -2180,7 +2180,7 @@ Returns the count of all hosts organized by status. `online_count` includes all
Returns the information of the specified host.
`GET /api/v1/fleet/hosts/{id}`
`GET /api/v1/fleet/hosts/:id`
#### Parameters
@ -2421,7 +2421,7 @@ Returns the information of the specified host.
Returns the information of the host specified using the `uuid`, `osquery_host_id`, `hostname`, or
`node_key` as an identifier
`GET /api/v1/fleet/hosts/identifier/{identifier}`
`GET /api/v1/fleet/hosts/identifier/:identifier`
#### Parameters
@ -2628,7 +2628,7 @@ Returns a subset of information about the host specified by `token`. To get all
This is the API route used by the **My device** page in Fleet desktop to display information about the host to the end user.
`GET /api/v1/fleet/device/{token}`
`GET /api/v1/fleet/device/:token`
##### Parameters
@ -2826,7 +2826,7 @@ This is the API route used by the **My device** page in Fleet desktop to display
Deletes the specified host from Fleet. Note that a deleted host will fail authentication with the previous node key, and in most osquery configurations will attempt to re-enroll automatically. If the host still has a valid enroll secret, it will re-enroll successfully.
`DELETE /api/v1/fleet/hosts/{id}`
`DELETE /api/v1/fleet/hosts/:id`
#### Parameters
@ -2847,7 +2847,7 @@ Deletes the specified host from Fleet. Note that a deleted host will fail authen
Flags the host details, labels and policies to be refetched the next time the host checks in for distributed queries. Note that we cannot be certain when the host will actually check in and update the query results. Further requests to the host APIs will indicate that the refetch has been requested through the `refetch_requested` field on the host object.
`POST /api/v1/fleet/hosts/{id}/refetch`
`POST /api/v1/fleet/hosts/:id/refetch`
#### Parameters
@ -2992,7 +2992,7 @@ Retrieves a host's Google Chrome profile information which can be used to link a
Requires [Fleetd](https://fleetdm.com/docs/using-fleet/fleetd), the osquery manager from Fleet. Fleetd can be built with [fleetctl](https://fleetdm.com/docs/using-fleet/adding-hosts#osquery-installer).
`GET /api/v1/fleet/hosts/{id}/device_mapping`
`GET /api/v1/fleet/hosts/:id/device_mapping`
#### Parameters
@ -3032,7 +3032,7 @@ Retrieves a host's MDM enrollment status and MDM server URL.
If the host exists but is not enrolled to an MDM server, then this API returns `null`.
`GET /api/v1/fleet/hosts/{id}/mdm`
`GET /api/v1/fleet/hosts/:id/mdm`
#### Parameters
@ -3124,7 +3124,7 @@ Currently supported only on macOS.
Retrieves a host's MDM enrollment status, MDM server URL, and Munki version.
`GET /api/v1/fleet/hosts/{id}/macadmins`
`GET /api/v1/fleet/hosts/:id/macadmins`
#### Parameters
@ -3536,7 +3536,7 @@ Creates a dynamic label.
Modifies the specified label. Note: Label queries and platforms are immutable. To change these, you must delete the label and create a new label.
`PATCH /api/v1/fleet/labels/{id}`
`PATCH /api/v1/fleet/labels/:id`
#### Parameters
@ -3587,7 +3587,7 @@ Modifies the specified label. Note: Label queries and platforms are immutable. T
Returns the specified label.
`GET /api/v1/fleet/labels/{id}`
`GET /api/v1/fleet/labels/:id`
#### Parameters
@ -3776,7 +3776,7 @@ Returns a list of all the labels in Fleet.
Returns a list of the hosts that belong to the specified label.
`GET /api/v1/fleet/labels/{id}/hosts`
`GET /api/v1/fleet/labels/:id/hosts`
#### Parameters
@ -3872,7 +3872,7 @@ If `mdm_id`, `mdm_name`, `mdm_enrollment_status`, `os_settings`, or `os_settings
Deletes the label specified by name.
`DELETE /api/v1/fleet/labels/{name}`
`DELETE /api/v1/fleet/labels/:name`
#### Parameters
@ -3893,7 +3893,7 @@ Deletes the label specified by name.
Deletes the label specified by ID.
`DELETE /api/v1/fleet/labels/id/{id}`
`DELETE /api/v1/fleet/labels/id/:id`
#### Parameters
@ -4061,7 +4061,7 @@ List all configuration profiles for macOS hosts enrolled to Fleet's MDM that are
### Download custom macOS setting (configuration profile)
`GET /api/v1/fleet/mdm/apple/profiles/{profile_id}`
`GET /api/v1/fleet/mdm/apple/profiles/:profile_id`
#### Parameters
@ -4113,7 +4113,7 @@ solely on the response status code returned by this endpoint.
### Delete custom macOS setting (configuration profile)
`DELETE /api/v1/fleet/mdm/apple/profiles/{profile_id}`
`DELETE /api/v1/fleet/mdm/apple/profiles/:profile_id`
#### Parameters
@ -4292,6 +4292,8 @@ This endpoint returns the results for a specific custom MDM command.
}
```
> Note: If the server has not yet received a result for a command, it will return an empty object (`{}`).
### List custom MDM commands
> `GET /api/v1/fleet/mdm/apple/commands` API endpoint is deprecated as of Fleet 4.40. It is maintained for backward compatibility. Please use the new API endpoint below. See old API endpoint docs [here](https://github.com/fleetdm/fleet/blob/ee02782eaf84c121256d73abc20b949d31bf2e57/docs/REST%20API/rest-api.md#list-custom-mdm-commands).
@ -4487,7 +4489,7 @@ None.
### Turn off MDM for a host
`PATCH /api/v1/fleet/mdm/hosts/{id}/unenroll`
`PATCH /api/v1/fleet/mdm/hosts/:id/unenroll`
#### Parameters
@ -4557,7 +4559,7 @@ _Available in Fleet Premium_
Get information about a bootstrap package that was uploaded to Fleet.
`GET /api/v1/fleet/mdm/apple/bootstrap/{team_id}/metadata`
`GET /api/v1/fleet/mdm/apple/bootstrap/:team_id/metadata`
#### Parameters
@ -4595,7 +4597,7 @@ _Available in Fleet Premium_
Delete a team's bootstrap package.
`DELETE /api/v1/fleet/mdm/apple/bootstrap/{team_id}`
`DELETE /api/v1/fleet/mdm/apple/bootstrap/:team_id`
#### Parameters
@ -4779,7 +4781,7 @@ _Available in Fleet Premium_
Delete an EULA file.
`DELETE /api/v1/fleet/mdm/apple/setup/eula/{token}`
`DELETE /api/v1/fleet/mdm/apple/setup/eula/:token`
#### Parameters
@ -4801,7 +4803,7 @@ _Available in Fleet Premium_
Download an EULA file
`GET /api/v1/fleet/mdm/apple/setup/eula/{token}`
`GET /api/v1/fleet/mdm/apple/setup/eula/:token`
#### Parameters
@ -4937,7 +4939,7 @@ For example, a policy might ask “Is Gatekeeper enabled on macOS devices?“ Th
### Get policy by ID
`GET /api/v1/fleet/global/policies/{id}`
`GET /api/v1/fleet/global/policies/:id`
#### Parameters
@ -5117,7 +5119,7 @@ Where `query_id` references an existing `query`.
### Edit policy
`PATCH /api/v1/fleet/global/policies/{policy_id}`
`PATCH /api/v1/fleet/global/policies/:id`
#### Parameters
@ -5226,13 +5228,13 @@ Team policies work the same as policies, but at the team level.
### List team policies
`GET /api/v1/fleet/teams/{id}/policies`
`GET /api/v1/fleet/teams/:id/policies`
#### Parameters
| Name | Type | In | Description |
| ------------------ | ------- | ---- | ------------------------------------------------------------------------------------------------------------- |
| id | integer | url | Required. Defines what team ID to operate on |
| id | integer | path | **Required.** Defines what team ID to operate on |
| page | integer | query | Page number of the results to fetch. |
| per_page | integer | query | Results per page. |
#### Example
@ -5305,11 +5307,12 @@ Team policies work the same as policies, but at the team level.
### Count team policies
`GET /api/v1/fleet/team/{team_id}/policies/count`
`GET /api/v1/fleet/team/:team_id/policies/count`
#### Parameters
| Name | Type | In | Description |
| ------------------ | ------- | ---- | ------------------------------------------------------------------------------------------------------------- |
| team_id | integer | path | **Required.** Defines what team ID to operate on
| query | string | query | Search query keywords. Searchable fields include `name`. |
#### Example
@ -5330,14 +5333,14 @@ Team policies work the same as policies, but at the team level.
### Get team policy by ID
`GET /api/v1/fleet/teams/{team_id}/policies/{id}`
`GET /api/v1/fleet/teams/:team_id/policies/:policy_id`
#### Parameters
| Name | Type | In | Description |
| ------------------ | ------- | ---- | ------------------------------------------------------------------------------------------------------------- |
| team_id | integer | url | Defines what team ID to operate on |
| id | integer | path | **Required.** The policy's ID. |
| team_id | integer | path | **Required.** Defines what team ID to operate on |
| policy_id | integer | path | **Required.** The policy's ID. |
#### Example
@ -5373,13 +5376,13 @@ Team policies work the same as policies, but at the team level.
The semantics for creating a team policy are the same as for global policies, see [Add policy](#add-policy).
`POST /api/v1/fleet/teams/{team_id}/policies`
`POST /api/v1/fleet/teams/:id/policies`
#### Parameters
| Name | Type | In | Description |
| ---------- | ------- | ---- | ------------------------------------ |
| team_id | integer | url | Defines what team ID to operate on. |
| id | integer | path | Defines what team ID to operate on. |
| name | string | body | The query's name. |
| query | string | body | The query in SQL. |
| description | string | body | The query's description. |
@ -5435,13 +5438,13 @@ Either `query` or `query_id` must be provided.
### Remove team policies
`POST /api/v1/fleet/teams/{team_id}/policies/delete`
`POST /api/v1/fleet/teams/:team_id/policies/delete`
#### Parameters
| Name | Type | In | Description |
| -------- | ------- | ---- | ------------------------------------------------- |
| team_id | integer | url | Defines what team ID to operate on |
| team_id | integer | path | **Required.** Defines what team ID to operate on |
| ids | list | body | **Required.** The IDs of the policies to delete. |
#### Example
@ -5468,7 +5471,7 @@ Either `query` or `query_id` must be provided.
### Edit team policy
`PATCH /api/v1/fleet/teams/{team_id}/policies/{policy_id}`
`PATCH /api/v1/fleet/teams/:team_id/policies/:policy_id`
#### Parameters
@ -5651,7 +5654,7 @@ Returns a list of global queries or team queries.
Returns the query specified by ID.
`GET /api/v1/fleet/queries/{id}`
`GET /api/v1/fleet/queries/:id`
#### Parameters
@ -5714,7 +5717,7 @@ Returns the query specified by ID.
Returns the query report specified by ID.
`GET /api/v1/fleet/queries/{id}/report`
`GET /api/v1/fleet/queries/:id/report`
#### Parameters
@ -5871,7 +5874,7 @@ Creates a global query or team query.
Modifies the query specified by ID.
`PATCH /api/v1/fleet/queries/{id}`
`PATCH /api/v1/fleet/queries/:id`
#### Parameters
@ -5944,7 +5947,7 @@ Modifies the query specified by ID.
Deletes the query specified by name.
`DELETE /api/v1/fleet/queries/{name}`
`DELETE /api/v1/fleet/queries/:name`
#### Parameters
@ -5955,7 +5958,7 @@ Deletes the query specified by name.
#### Example
`DELETE /api/v1/fleet/queries/{name}`
`DELETE /api/v1/fleet/queries/foo`
##### Default response
@ -5966,7 +5969,7 @@ Deletes the query specified by name.
Deletes the query specified by ID.
`DELETE /api/v1/fleet/queries/id/{id}`
`DELETE /api/v1/fleet/queries/id/:id`
#### Parameters
@ -6257,7 +6260,7 @@ None.
> The schedule API endpoints are deprecated as of Fleet 4.35. They are maintained for backwards compatibility.
> Please use the [queries](#queries) endpoints, which as of 4.35 have attributes such as `interval` and `platform` that enable scheduling.
`PATCH /api/v1/fleet/global/schedule/{id}`
`PATCH /api/v1/fleet/global/schedule/:id`
#### Parameters
@ -6313,7 +6316,7 @@ None.
> The schedule API endpoints are deprecated as of Fleet 4.35. They are maintained for backwards compatibility.
> Please use the [queries](#queries) endpoints, which as of 4.35 have attributes such as `interval` and `platform` that enable scheduling.
`DELETE /api/v1/fleet/global/schedule/{id}`
`DELETE /api/v1/fleet/global/schedule/:id`
#### Parameters
@ -6347,7 +6350,7 @@ This allows you to easily configure scheduled queries that will impact a whole t
> The schedule API endpoints are deprecated as of Fleet 4.35. They are maintained for backwards compatibility.
> Please use the [queries](#queries) endpoints, which as of 4.35 have attributes such as `interval` and `platform` that enable scheduling.
`GET /api/v1/fleet/teams/{id}/schedule`
`GET /api/v1/fleet/teams/:id/schedule`
#### Parameters
@ -6427,7 +6430,7 @@ This allows you to easily configure scheduled queries that will impact a whole t
> The schedule API endpoints are deprecated as of Fleet 4.35. They are maintained for backwards compatibility.
> Please use the [queries](#queries) endpoints, which as of 4.35 have attributes such as `interval` and `platform` that enable scheduling.
`POST /api/v1/fleet/teams/{id}/schedule`
`POST /api/v1/fleet/teams/:id/schedule`
#### Parameters
@ -6485,7 +6488,7 @@ This allows you to easily configure scheduled queries that will impact a whole t
> The schedule API endpoints are deprecated as of Fleet 4.35. They are maintained for backwards compatibility.
> Please use the [queries](#queries) endpoints, which as of 4.35 have attributes such as `interval` and `platform` that enable scheduling.
`PATCH /api/v1/fleet/teams/{team_id}/schedule/{scheduled_query_id}`
`PATCH /api/v1/fleet/teams/:team_id/schedule/:scheduled_query_id`
#### Parameters
@ -6542,7 +6545,7 @@ This allows you to easily configure scheduled queries that will impact a whole t
> The schedule API endpoints are deprecated as of Fleet 4.35. They are maintained for backwards compatibility.
> Please use the [queries](#queries) endpoints, which as of 4.35 have attributes such as `interval` and `platform` that enable scheduling.
`DELETE /api/v1/fleet/teams/{team_id}/schedule/{scheduled_query_id}`
`DELETE /api/v1/fleet/teams/:team_id/schedule/:scheduled_query_id`
#### Parameters
@ -6563,8 +6566,7 @@ This allows you to easily configure scheduled queries that will impact a whole t
## Scripts
- [Run script asynchronously](#run-script-asynchronously)
- [Run script synchronously](#run-script-synchronously)
- [Run script](#run-script)
- [Get script result](#get-script-result)
- [Upload a script](#upload-a-script)
- [Delete a script](#delete-a-script)
@ -6572,44 +6574,11 @@ This allows you to easily configure scheduled queries that will impact a whole t
- [Get or download a script](#get-or-download-a-script)
- [Get script details by host](#get-script-details-by-host)
### Run script asynchronously
### Run script
_Available in Fleet Premium_
Creates a script execution request and returns the execution identifier to retrieve results at a later time.
`POST /api/v1/fleet/scripts/run`
#### Parameters
| Name | Type | In | Description |
| ---- | ------- | ---- | -------------------------------------------- |
| host_id | integer | body | **Required**. The ID of the host to run the script on. |
| script_id | integer | body | The ID of the existing saved script to run. Only one of either `script_id` or `script_contents` can be included in the request; omit this parameter if using `script_contents`. |
| script_contents | string | body | The contents of the script to run. Only one of either `script_id` or `script_contents` can be included in the request; omit this parameter if using `script_id`. |
> Note that if both `script_id` and `script_contents` are included in the request, this endpoint will respond with an error.
#### Example
`POST /api/v1/fleet/scripts/run`
##### Default response
`Status: 202`
```json
{
"host_id": 1227,
"execution_id": "e797d6c6-3aae-11ee-be56-0242ac120002"
}
```
### Run script synchronously
_Available in Fleet Premium_
Creates a script execution request and waits for a result to return (up to a 1 minute timeout).
Execute a script and see script results (1 minute timeout).
`POST /api/v1/fleet/scripts/run/sync`
@ -6658,7 +6627,7 @@ Gets the result of a script that was executed.
#### Example
`GET /api/v1/fleet/scripts/results/{execution_id}`
`GET /api/v1/fleet/scripts/results/:execution_id`
##### Default Response
@ -6738,7 +6707,7 @@ _Available in Fleet Premium_
Deletes an existing script.
`DELETE /api/v1/fleet/scripts/{id}`
`DELETE /api/v1/fleet/scripts/:id`
#### Parameters
@ -6805,7 +6774,7 @@ _Available in Fleet Premium_
_Available in Fleet Premium_
`GET /api/v1/fleet/scripts/{id}`
`GET /api/v1/fleet/scripts/:id`
#### Parameters
@ -6857,7 +6826,7 @@ echo "hello"
_Available in Fleet Premium_
`GET /api/v1/fleet/hosts/{id}/scripts`
`GET /api/v1/fleet/hosts/:id/scripts`
#### Parameters
@ -6868,7 +6837,7 @@ _Available in Fleet Premium_
#### Example
`GET /api/v1/fleet/hosts/{id}/scripts`
`GET /api/v1/fleet/hosts/123/scripts`
##### Default response
@ -6924,7 +6893,7 @@ _Available in Fleet Premium_
Returns the session information for the session specified by ID.
`GET /api/v1/fleet/sessions/{id}`
`GET /api/v1/fleet/sessions/:id`
#### Parameters
@ -6952,7 +6921,7 @@ Returns the session information for the session specified by ID.
Deletes the session specified by ID. When the user associated with the session next attempts to access Fleet, they will be asked to log in.
`DELETE /api/v1/fleet/sessions/{id}`
`DELETE /api/v1/fleet/sessions/:id`
#### Parameters
@ -7026,6 +6995,17 @@ Deletes the session specified by ID. When the user associated with the session n
}
],
"hosts_count": 1
},
{
"id": 2,
"name": "1Password – Password Manager",
"version": "2.3.7",
"source": "chrome_extensions",
"extension_id": "aeblfdkhhhdcdjpifhhbdiojplfjncoa",
"browser": "chrome",
"generated_cpe": "cpe:2.3:a:1password:1password:2.3.7:*:*:*:*:chrome:*:*",
"vulnerabilities": null,
"hosts_count": 2
}
]
}
@ -7338,7 +7318,7 @@ _Available in Fleet Premium_
_Available in Fleet Premium_
`GET /api/v1/fleet/teams/{id}`
`GET /api/v1/fleet/teams/:id`
#### Parameters
@ -7490,7 +7470,7 @@ _Available in Fleet Premium_
_Available in Fleet Premium_
`PATCH /api/v1/fleet/teams/{id}`
`PATCH /api/v1/fleet/teams/:id`
#### Parameters
@ -7669,7 +7649,7 @@ _Available in Fleet Premium_
_Available in Fleet Premium_
`POST /api/v1/fleet/teams/{id}/agent_options`
`POST /api/v1/fleet/teams/:id/agent_options`
#### Parameters
@ -7758,7 +7738,7 @@ _Available in Fleet Premium_
_Available in Fleet Premium_
`DELETE /api/v1/fleet/teams/{id}`
`DELETE /api/v1/fleet/teams/:id`
#### Parameters
@ -8169,7 +8149,7 @@ By default, the user will be forced to reset its password upon first login.
Returns all information about a specific user.
`GET /api/v1/fleet/users/{id}`
`GET /api/v1/fleet/users/:id`
#### Parameters
@ -8229,7 +8209,7 @@ Returns all information about a specific user.
### Modify user
`PATCH /api/v1/fleet/users/{id}`
`PATCH /api/v1/fleet/users/:id`
#### Parameters
@ -8337,7 +8317,7 @@ Returns all information about a specific user.
Delete the specified user from Fleet.
`DELETE /api/v1/fleet/users/{id}`
`DELETE /api/v1/fleet/users/:id`
#### Parameters
@ -8358,7 +8338,7 @@ Delete the specified user from Fleet.
The selected user is logged out of Fleet and required to reset their password during the next attempt to log in. This also revokes all active Fleet API tokens for this user. Returns the user object.
`POST /api/v1/fleet/users/{id}/require_password_reset`
`POST /api/v1/fleet/users/:id/require_password_reset`
#### Parameters
@ -8369,7 +8349,7 @@ The selected user is logged out of Fleet and required to reset their password du
#### Example
`POST /api/v1/fleet/users/{id}/require_password_reset`
`POST /api/v1/fleet/users/123/require_password_reset`
##### Request body
@ -8404,7 +8384,7 @@ The selected user is logged out of Fleet and required to reset their password du
Returns a list of the user's sessions in Fleet.
`GET /api/v1/fleet/users/{id}/sessions`
`GET /api/v1/fleet/users/:id/sessions`
#### Parameters
@ -8444,7 +8424,7 @@ None.
Deletes the selected user's sessions in Fleet. Also deletes the user's API token.
`DELETE /api/v1/fleet/users/{id}/sessions`
`DELETE /api/v1/fleet/users/:id/sessions`
#### Parameters
@ -8522,7 +8502,7 @@ Returns information about the current state of the database; valid keys are:
- `innodb-status`: returns InnoDB status information.
- `process-list`: returns running processes (queries, etc).
`GET /debug/db/{key}`
`GET /debug/db/:key`
#### Parameters
@ -8534,7 +8514,7 @@ Returns runtime profiling data of the server in the format expected by `go tools
Valid keys are: `cmdline`, `profile`, `symbol` and `trace`.
`GET /debug/pprof/{key}`
`GET /debug/pprof/:key`
#### Parameters
None.

94
docs/Using Fleet/MDM-commands.md
View file

@ -1,28 +1,27 @@
# Commands
In Fleet you can run MDM commands to take some action on your macOS hosts, like restart the host, remotely.
If a host is offline when you run a command, the host will run the command the next time it comes online.
In Fleet you can run MDM commands to take action on your macOS and Windows hosts, like restarting the host, remotely.
## Custom commands
You can run custom commands and view a specific command's results using the `fleetctl` command-line interface.
To run a custom command, we will do the following steps:
1. Create a `.xml` with the request payload
2. Choose a target host
3. Run the command using `fleetctl`
4. View our command's results using `fleetctl`
### Step 1: create a `.xml` file
### Step 1: Create an XML file
You can run any command supported by Apple's MDM protocol as a custom command in Fleet. To see the list of possible commands, head to [Apple's Commands and Queries documentation](https://developer.apple.com/documentation/devicemanagement/commands_and_queries).
You can run any command supported by [Apple's MDM protocol](https://developer.apple.com/documentation/devicemanagement/commands_and_queries) or [Microsoft's MDM protocol](https://learn.microsoft.com/en-us/windows/client-management/mdm/).
> The "Erase a device" and "Lock a device" commands are only available in Fleet Premium
> The lock and wipe commands are only available in Fleet Premium
Each command has example request payloads in XML format. For example, if we want to restart a host, we'll use the "Restart a Device" request payload documented by Apple [here](https://developer.apple.com/documentation/devicemanagement/restart_a_device#3384428).
For example, to restart a macOS host, we'll use the "Restart a Device" command documented by Apple [here](https://developer.apple.com/documentation/devicemanagement/restart_a_device#3384428).
To run the "Restart a device" command, we'll need to create a `restart-device.xml` file locally and copy and paste the request payload into this `.xml` file:
First, we'll need to create a `restart-device.xml` file locally with this payload:
```xml
<?xml version="1.0" encoding="UTF-8"?>
@ -34,90 +33,69 @@ To run the "Restart a device" command, we'll need to create a `restart-device.xm
<key>RequestType</key>
<string>RestartDevice</string>
</dict>
<key>CommandUUID</key>
<string>0001_RestartDevice</string>
</dict>
</plist>
```
### Step 2: choose a target host
To restart a Windows host, we'll use the "Reboot" command documented by Microsoft [here](https://learn.microsoft.com/en-us/windows/client-management/mdm/reboot-csp).
To run a command, we need to specify a target host by hostname. Commands can only be run on a single host in Fleet.
The `restart-device.xml` file will have this payload instead:
To find a host's hostname, choose the "Fleet UI" or "fleetctl" method and follow the steps below.
```xml
<Exec>
<Item>
<Target>
<LocURI>./Device/Vendor/MSFT/Reboot/RebootNow</LocURI>
</Target>
<Meta>
<Format xmlns="syncml:metinf">null</Format>
<Type>text/plain</Type>
</Meta>
<Data></Data>
</Item>
</Exec>
```
Fleet UI:
### Step 2: Choose a target host
1. Head to the **Hosts** page in Fleet and find your target host.
2. Make sure the **Hostname** column is visible (select **Edit columns** if not) and find your host's hostname. You'll need this hostname to run the command.
> A host must be enrolled to Fleet and have MDM turned on to run a command against it.
`fleetctl` CLI:
To run a command, we need to specify a target host by hostname.
1. Run the `fleetctl get hosts --mdm` command to get a list of hosts that are enrolled to Fleet and have MDM turned on.
2. Find your host's hostname. You'll need this hostname to run the command.
2. Find your target host's hostname. You'll need this hostname to run the command.
### Step 3: run the command
### Step 3: Run the command
1. Run the `fleetctl mdm run-command --payload=restart-device.xml --host=hostname ` command.
> Replace the --payload and --host flags with your `.xml` file and hostname respectively.
2. Look at the on-screen information. In the output you'll see the command required to see results. Be sure to copy this command. If you don't, it will be difficult to view command results later.
> Replace the --payload and --host flags with your XML file and hostname respectively.
2. Look at the on-screen information. In the output you'll see the command to see results.
### Step 4: View the command's results
1. Run the `fleetctl get mdm-command-results --id=<insert-command-id>`
2. Look at the on-screen information.
Example output:
```sh
$ fleetctl get mdm-command-results -id 333af7f8-b9a4-4f62-bfb2-f7488fbade21
+--------------------------------------+----------------------+----------------+--------------+---------------------+---------------------------------------------------------+
| ID | TIME | TYPE | STATUS | HOSTNAME | RESULTS |
+--------------------------------------+----------------------+----------------+--------------+---------------------+---------------------------------------------------------+
| 333af7f8-b9a4-4f62-bfb2-f7488fbade21 | 2023-04-04T21:29:29Z | RestartDevice | Acknowledged | xyz-macbook-air.lan | <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE |
| | | | | | plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" |
| | | | | | "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> |
| | | | | | <plist version="1.0"> <dict> <key>CommandUUID</key> |
| | | | | | <string>333af7f8-b9a4-4f62-bfb2-f7488fbade21</string> |
| | | | | | <key>Status</key> |
| | | | | | <string>Acknowledged</string> <key>UDID</key> |
| | | | | | <string>3A529CD6-2154-55EA-9AB7-EB13A43D9F5E</string> |
| | | | | | </dict> </plist> |
+--------------------------------------+----------------------+----------------+--------------+---------------------+---------------------------------------------------------+
```
## List recent commands
You can view the list of the 1,000 latest commands using "fleetctl":
You can view a list of the 1,000 latest commands:
1. Run `fleetctl get mdm-commands`
2. View the list of latest commands, most recent first, along with the timestamp, targeted hostname, command type, execution status and command ID.
Example output:
The command ID can be used to view command results as documented in [step 4 of the previous section](#step-4-view-the-commands-results).
```sh
$ fleetctl get mdm-commands
+--------------------------------------+----------------------+--------------------------+--------------+------------------------+
| ID | TIME | TYPE | STATUS | HOSTNAME |
+--------------------------------------+----------------------+--------------------------+--------------+------------------------+
| 024fb3b9-cd8a-40a6-8dd7-6c155f488fd1 | 2023-04-12T18:19:10Z | RestartDevice | Acknowledged | iMac-Pro.local |
+--------------------------------------+----------------------+--------------------------+--------------+------------------------+
| 87dc6325-8bc0-4fc8-9a2f-3901c535456e | 2023-04-12T18:15:01Z | DeviceLock | Acknowledged | iMac-Pro.local |
+--------------------------------------+----------------------+--------------------------+--------------+------------------------+
```
The possible statuses for macOS hosts are the following:
The command ID can be used to view command results as documented in [step 4 of the previous section](#step-4-view-the-commands-results). The possible status values are:
* Pending: the command has yet to run on the host. The host will run the command the next time it comes online.
* NotNow: the host responded with "NotNow" status via the MDM protocol: the host received the command, but couldn’t execute it. The host will try to run the command the next time it comes online.
* Acknowledged: the host responded with "Acknowledged" status via the MDM protocol: the host processed the command successfully.
* Error: the host responded with "Error" status via the MDM protocol: an error occurred. Run the `fleetctl get mdm-command-results --id=<insert-command-id` to view the error.
* CommandFormatError: the host responded with "CommandFormatError" status via the MDM protocol: a protocol error occurred, which can result from a malformed command. Run the `fleetctl get mdm-command-results --id=<insert-command-id` to view the error.
The possible statuses for Windows hosts are documented in Microsoft's documentation [here](https://learn.microsoft.com/en-us/windows/client-management/oma-dm-protocol-support#syncml-response-status-codes).
<meta name="pageOrderInSection" value="1507">
<meta name="title" value="Commands">
<meta name="description" value="Learn how to run custom MDM commands on macOS hosts using Fleet.">
<meta name="description" value="Learn how to run custom MDM commands on hosts using Fleet.">
<meta name="navSection" value="Device management">

4
docs/Using Fleet/MDM-macOS-setup.md
View file

@ -225,8 +225,8 @@ Set Fleet to be the MDM for all future Macs purchased via Apple or an authorized
1. Log in to [Apple Business Manager](https://business.apple.com)
2. Click your profile icon in the bottom left
3. Click **Preferences**
4. Click **MDM Server Assignment**
5. Switch Macs to the new Fleet instance.
4. Click **MDM Server Assignment** and click **Edit** next to **Default Server Assignment**.
5. Switch **Mac** to Fleet.
### Step 6: set the default team for hosts enrolled via ABM

54
docs/Using Fleet/Scripts.md
View file

@ -10,16 +10,7 @@ PowerShell scripts are supported on Windows. Other types of scripts are not supp
Script execution is disabled by default. Continue reading to learn how to enable scripts.
## Execute a script
You can execute a script using the `fleetctl` command-line interface.
To execute a script, we will do the following steps:
1. Enable script execution
2. Write a script
3. Run the script
### Step 1: Enable script execution
## Enable scripts
If you use Fleet's macOS MDM features, scripts are automatically enabled for macOS hosts that have MDM turned on. You're set!
@ -31,39 +22,28 @@ If you don't use MDM features, to enable scripts, we'll deploy a fleetd agent wi
Learn more about generating a fleetd agent and deploying it [here](./enroll-hosts.md).
### Step 2: Write a script
## Execute a script
As an example, we'll write a shell script for a macOS host that downloads a Fleet wallpaper and set the host's wallpaper to it.
You can execute a script in the Fleet UI, with Fleet API, or with the fleetctl command-line interface (CLI).
To run the script, we'll need to create a `set-wallpaper-to-fleet.sh` file locally and copy and paste this script into this `.sh` file:
Fleet UI:
1. In Fleet, head to the **Controls > Scripts** tab and upload your script.
2. Head to the **Hosts** page and select the host you want to run the script on.
3. On your target host's host details page, select the **Scripts** tab and select **Actions** to run the script.
> Currently, you can only run scripts on macOS and Windows hosts in the Fleet UI. To run a script on a Linux host, use the Fleet API or fleetctl CLI.
Fleet API: API documentation is [here](https://fleetdm.com/docs/rest-api/rest-api#run-script)
fleetctl CLI:
```sh
wallpaper="/tmp/wallpaper.png"
curl --fail https://fleetdm.com/images/wallpaper-cloud-city-1920x1080.png -o $wallpaper
osascript -e 'tell application "Finder" to set desktop picture to POSIX file "'"$wallpaper"'"'
fleetctl run-script --script-path=/path/to/script --host=hostname
```
### Step 3: Run the script
1. Run this fleetctl command:
```sh
fleetctl run-script --script-path=set-wallpaper-to-fleet.sh --host=hostname
```
> Replace --host flag with your target host's hostname.
2. Look at the on-screen information. In the output you'll see the script's exit code and output.
Each time a Fleet user runs a script an entry is created in [Fleet's activity feed](./Audit-logs.md#type-code-ran-script-code).
## Security considerations
Script execution can only be enabled by someone with root access to the host.
Turning MDM on for a macOS host or pushing a new fleetd agent qualify as root access.
<meta name="pageOrderInSection" value="1508">
<meta name="title" value="Scripts">
<meta name="description" value="Learn how to execute a custom script on macOS, Windows, and Linux hosts in Fleet.">

1
docs/Using Fleet/enroll-hosts.md
View file

@ -108,7 +108,6 @@ In the Google Admin console:
- [Signing fleetd installer](#signing-fleetd-installer)
- [Generating Windows installers using local WiX toolset](#generating-windows-installers-using-local-wix-toolset)
- [fleetd configuration options](#fleetd-configuration-options)
- [Enroll hosts with plain osquery](#enroll-hosts-with-plain-osquery)
### Grant full disk access to osquery on macOS

10403
ee/cis/win-11/cis-policy-queries.yml Normal file
View file

File diff suppressed because it is too large Load diff

41
ee/fleetd-chrome/src/tables/system_info.ts
View file

@ -33,12 +33,21 @@ export default class TableSystemInfo extends Table {
// @ts-expect-error @types/chrome doesn't yet have instanceID.
const uuid = (await chrome.instanceID.getID()) as string;
let devMode = false;
if (!chrome.enterprise) {
const { installType } = await chrome.management.getSelf();
devMode = installType === "development";
}
// TODO should it default to UUID or should Fleet handle it somehow?
let hostname = "";
try {
// @ts-expect-error @types/chrome doesn't yet have the deviceAttributes Promise API.
hostname = (await chrome.enterprise.deviceAttributes.getDeviceHostname()) as string;
if (!devMode) {
// @ts-expect-error @types/chrome doesn't yet have the deviceAttributes Promise API.
hostname = (await chrome.enterprise.deviceAttributes.getDeviceHostname()) as string;
} else {
hostname = uuid;
}
} catch (err) {
console.warn("get hostname:", err);
warningsArray.push({
@ -49,8 +58,13 @@ export default class TableSystemInfo extends Table {
let hwSerial = "";
try {
// @ts-expect-error @types/chrome doesn't yet have the deviceAttributes Promise API.
hwSerial = (await chrome.enterprise.deviceAttributes.getDeviceSerialNumber()) as string;
if (!devMode) {
// @ts-expect-error @types/chrome doesn't yet have the deviceAttributes Promise API.
hwSerial = (await chrome.enterprise.deviceAttributes.getDeviceSerialNumber()) as string;
} else {
// We leave it blank. The host will be identified by UUID instead.
hwSerial = "";
}
} catch (err) {
console.warn("get serial number:", err);
warningsArray.push({
@ -62,13 +76,18 @@ export default class TableSystemInfo extends Table {
let hwVendor = "",
hwModel = "";
try {
// This throws "Not allowed" error if
// https://chromeenterprise.google/policies/?policy=EnterpriseHardwarePlatformAPIEnabled is
// not configured to enabled for the device.
// @ts-expect-error @types/chrome doesn't yet have the deviceAttributes Promise API.
const platformInfo = await chrome.enterprise.hardwarePlatform.getHardwarePlatformInfo();
hwVendor = platformInfo.manufacturer;
hwModel = platformInfo.model;
if (!devMode) {
// This throws "Not allowed" error if
// https://chromeenterprise.google/policies/?policy=EnterpriseHardwarePlatformAPIEnabled is
// not configured to enabled for the device.
// @ts-expect-error @types/chrome doesn't yet have the deviceAttributes Promise API.
const platformInfo = await chrome.enterprise.hardwarePlatform.getHardwarePlatformInfo();
hwVendor = platformInfo.manufacturer;
hwModel = platformInfo.model;
} else {
hwVendor = "dev-hardware_vendor";
hwModel = "dev-hardware_model";
}
} catch (err) {
console.warn("get platform info:", err);
warningsArray.push({

4
frontend/interfaces/config.ts
View file

@ -207,10 +207,6 @@ export interface IConfig {
};
};
mdm: IMdmConfig;
/** This is the flag that determines if the windwos mdm feature flag is enabled.
TODO: WINDOWS FEATURE FLAG: remove when windows MDM is released. Only used for windows MDM dev currently.
*/
mdm_enabled?: boolean;
}
export interface IWebhookSettings {

32
frontend/interfaces/host.ts
View file

@ -84,6 +84,38 @@ export interface IDeviceUser {
source: string;
}
const DEVICE_USER_SOURCE_TO_DISPLAY: { [key: string]: string } = {
google_chrome_profiles: "Google Chrome",
mdm_idp_accounts: "identity provider",
} as const;
const getDeviceUserSourceForDisplay = (s: string): string => {
return DEVICE_USER_SOURCE_TO_DISPLAY[s] || s;
};
const getDeviceUserForDisplay = (d: IDeviceUser): IDeviceUser => {
return { ...d, source: getDeviceUserSourceForDisplay(d.source) };
};
/*
* mapDeviceUsersForDisplay is a helper function that takes an array of device users and returns a
* new array of device users with the source field mapped to a more user-friendly value. It also
* ensures that the identity provider account is always the first device user in the array.
*/
export const mapDeviceUsersForDisplay = (
deviceMapping: IDeviceUser[]
): IDeviceUser[] => {
const newDeviceMapping: IDeviceUser[] = [];
deviceMapping.forEach((d) => {
if (d.source === "mdm_idp_accounts") {
newDeviceMapping.unshift(getDeviceUserForDisplay(d));
} else {
newDeviceMapping.push(getDeviceUserForDisplay(d));
}
});
return newDeviceMapping;
};
export interface IDeviceMappingResponse {
device_mapping: IDeviceUser[];
}

6
frontend/pages/ManageControlsPage/OSSettings/cards/DiskEncryption/DiskEncryption.tsx
View file

@ -106,11 +106,7 @@ const DiskEncryption = ({
return "If turned on, hosts' disk encryption keys will be stored in Fleet. ";
}
const isWindowsFeatureFlagEnabled = config?.mdm_enabled ?? false;
const dynamicText = isWindowsFeatureFlagEnabled
? " and “BitLocker” on Windows"
: "";
return `Also known as “FileVault” on macOS${dynamicText}. If turned on, hosts' disk encryption keys will be stored in Fleet. `;
return `Also known as “FileVault” on macOS and “BitLocker” on Windows. If turned on, hosts' disk encryption keys will be stored in Fleet. `;
};
return (

13
frontend/pages/ManageControlsPage/OSSettings/cards/DiskEncryption/components/DiskEncryptionTable/DiskEncryptionTable.tsx
View file

@ -20,8 +20,6 @@ interface IDiskEncryptionTableProps {
}
const DiskEncryptionTable = ({ currentTeamId }: IDiskEncryptionTableProps) => {
const { config } = useContext(AppContext);
const {
data: diskEncryptionStatusData,
error: diskEncryptionStatusError,
@ -34,15 +32,8 @@ const DiskEncryptionTable = ({ currentTeamId }: IDiskEncryptionTableProps) => {
}
);
// TODO: WINDOWS FEATURE FLAG: remove this when windows feature flag is removed.
// this is used to conditianlly show "View all hosts" link in table cells.
const windowsFeatureFlagEnabled = config?.mdm_enabled ?? false;
const tableHeaders = generateTableHeaders(windowsFeatureFlagEnabled);
const tableData = generateTableData(
windowsFeatureFlagEnabled,
diskEncryptionStatusData,
currentTeamId
);
const tableHeaders = generateTableHeaders();
const tableData = generateTableData(diskEncryptionStatusData, currentTeamId);
if (diskEncryptionStatusError) {
return <DataError />;

34
frontend/pages/ManageControlsPage/OSSettings/cards/DiskEncryption/components/DiskEncryptionTable/DiskEncryptionTableConfig.tsx
View file

@ -32,7 +32,6 @@ interface ICellProps {
};
row: {
original: {
includeWindows: boolean;
status: IStatusCellValue;
teamId: number;
};
@ -94,18 +93,13 @@ const defaultTableHeaders: IDataColumn[] = [
return (
<div className="disk-encryption-table__aggregate-table-data">
<TextCell value={aggregateCount} formatter={(val) => <>{val}</>} />
{/* TODO: WINDOWS FEATURE FLAG: remove this conditional when windows mdm
is released. the view all UI will show in the windows column when we
release the feature. */}
{!original.includeWindows && (
<ViewAllHostsLink
className="view-hosts-link"
queryParams={{
[HOSTS_QUERY_PARAMS.DISK_ENCRYPTION]: original.status.value,
team_id: original.teamId,
}}
/>
)}
<ViewAllHostsLink
className="view-hosts-link"
queryParams={{
[HOSTS_QUERY_PARAMS.DISK_ENCRYPTION]: original.status.value,
team_id: original.teamId,
}}
/>
</div>
);
},
@ -144,14 +138,8 @@ const windowsTableHeader: IDataColumn[] = [
},
];
// TODO: WINDOWS FEATURE FLAG: return all headers when windows feature flag is removed.
export const generateTableHeaders = (
includeWindows: boolean
): IDataColumn[] => {
return includeWindows
? [...defaultTableHeaders, ...windowsTableHeader]
: defaultTableHeaders;
return defaultTableHeaders;
export const generateTableHeaders = (): IDataColumn[] => {
return [...defaultTableHeaders, ...windowsTableHeader];
};
const STATUS_CELL_VALUES: Record<DiskEncryptionStatus, IStatusCellValue> = {
@ -215,9 +203,6 @@ const STATUS_ORDER = [
] as const;
export const generateTableData = (
// TODO: WINDOWS FEATURE FLAG: remove includeWindows when windows feature flag is removed.
// This is used to conditionally show "View all hosts" link in table cells.
includeWindows: boolean,
data?: IDiskEncryptionSummaryResponse,
currentTeamId?: number
) => {
@ -227,7 +212,6 @@ export const generateTableData = (
status: DiskEncryptionStatus,
statusAggregate: IDiskEncryptionStatusAggregate
) => ({
includeWindows,
status: STATUS_CELL_VALUES[status],
macosHosts: statusAggregate.macos,
windowsHosts: statusAggregate.windows,

8
frontend/pages/admin/IntegrationsPage/cards/AutomaticEnrollment/components/AppleBusinessManagerSection/AppleBusinessManagerSection.tsx
View file

@ -234,11 +234,9 @@ const AppleBusinessManagerSection = ({
<div className={baseClass}>
<h2>Apple Business Manager</h2>
{isLoadingMdmAppleBm ? <Spinner /> : renderAppleBMInfo()}
{config?.mdm_enabled && (
<WindowsAutomaticEnrollmentCard
viewDetails={navigateToWindowsAutomaticEnrollment}
/>
)}
<WindowsAutomaticEnrollmentCard
viewDetails={navigateToWindowsAutomaticEnrollment}
/>
{showEditTeamModal && (
<EditTeamModal
onCancel={toggleEditTeamModal}

11
frontend/pages/admin/IntegrationsPage/cards/MdmSettings/MdmSettings.tsx
View file

@ -63,13 +63,10 @@ const MdmSettings = ({ router }: IMdmSettingsProps) => {
turnOnMacOSMdm={navigateToMacOSMdm}
viewDetails={navigateToMacOSMdm}
/>
{/* TODO: remove conditional rendering when windows MDM is released. */}
{config?.mdm_enabled && (
<WindowsMdmCard
turnOnWindowsMdm={navigateToWindowsMdm}
editWindowsMdm={navigateToWindowsMdm}
/>
)}
<WindowsMdmCard
turnOnWindowsMdm={navigateToWindowsMdm}
editWindowsMdm={navigateToWindowsMdm}
/>
</>
)}
</div>

6
frontend/pages/admin/IntegrationsPage/cards/MdmSettings/WindowsMdmPage/WindowsMdmPage.tsx
View file

@ -111,12 +111,6 @@ interface IWindowsMdmPageProps {
const WindowsMdmPage = ({ router }: IWindowsMdmPageProps) => {
const { config } = useContext(AppContext);
// TODO: remove when windows MDM is fully released. This is a temporary redirect
// when the feature is not enabled.
if (!config?.mdm_enabled) {
router.replace(PATHS.ADMIN_INTEGRATIONS_MDM);
}
const isWindowsMdmEnabled =
config?.mdm?.windows_enabled_and_configured ?? false;

8
frontend/pages/hosts/ManageHostsPage/ManageHostsPage.tsx
View file

@ -1367,9 +1367,9 @@ const ManageHostsPage = ({
const emptyState = () => {
const emptyHosts: IEmptyTableProps = {
graphicName: "empty-hosts",
header: "Devices will show up here once they’re added to Fleet.",
header: "Hosts will show up here once they’re added to Fleet.",
info:
"Expecting to see devices? Try again in a few seconds as the system catches up.",
"Expecting to see hosts? Try again in a few seconds as the system catches up.",
};
if (includesFilterQueryParam) {
delete emptyHosts.graphicName;
@ -1377,8 +1377,8 @@ const ManageHostsPage = ({
emptyHosts.info =
"Expecting to see new hosts? Try again in a few seconds as the system catches up.";
} else if (canEnrollHosts) {
emptyHosts.header = "Add your devices to Fleet";
emptyHosts.info = "Generate an installer to add your own devices.";
emptyHosts.header = "Add your hosts to Fleet";
emptyHosts.info = "Generate an installer to add your own hosts.";
emptyHosts.primaryButton = (
<Button variant="brand" onClick={toggleAddHostsModal} type="button">
Add hosts

10
frontend/pages/hosts/details/_styles.scss
View file

@ -106,6 +106,16 @@
color: $core-fleet-black;
font-weight: $bold;
}
&__data {
.device-mapping {
&__source {
color: $ui-fleet-black-75;
}
&__more {
color: $ui-fleet-black-50;
}
}
}
}
.list {

70
frontend/pages/hosts/details/cards/About/About.tsx
View file

@ -5,13 +5,33 @@ import { HumanTimeDiffWithFleetLaunchCutoff } from "components/HumanTimeDiffWith
import TooltipWrapper from "components/TooltipWrapper";
import CustomLink from "components/CustomLink";
import { IHostMdmData, IMunkiData, IDeviceUser } from "interfaces/host";
import {
IHostMdmData,
IMunkiData,
IDeviceUser,
mapDeviceUsersForDisplay,
} from "interfaces/host";
import {
DEFAULT_EMPTY_CELL_VALUE,
MDM_STATUS_TOOLTIP,
} from "utilities/constants";
import { COLORS } from "styles/var/colors";
const getDeviceUserTipContent = (deviceMapping: IDeviceUser[]) => {
if (deviceMapping.length === 0) {
return [];
}
const format = (d: IDeviceUser) =>
d.source ? `${d.email} (${d.source})` : d.email;
return deviceMapping.slice(1).map((d) => (
<span key={format(d)}>
{format(d)}
<br />
</span>
));
};
interface IAboutProps {
aboutData: { [key: string]: any };
deviceMapping?: IDeviceUser[];
@ -123,34 +143,38 @@ const About = ({
return null;
}
const numUsers = deviceMapping.length;
const tooltipText = deviceMapping.map((d) => (
<span key={Math.random().toString().slice(2)}>
{d.email}
<br />
</span>
));
let displayPrimaryUser: React.ReactNode = DEFAULT_EMPTY_CELL_VALUE;
const newDeviceMapping = mapDeviceUsersForDisplay(deviceMapping);
if (newDeviceMapping[0]) {
const { email, source } = newDeviceMapping[0];
if (!source) {
displayPrimaryUser = email;
} else {
displayPrimaryUser = (
<>
{email}{" "}
<span className="device-mapping__source">{`(${source})`}</span>
</>
);
}
}
return (
<div className="info-grid__block">
<span className="info-grid__header">Used by</span>
<span className="info-grid__data">
{numUsers > 1 ? (
<>
<span data-tip data-for="device_mapping" className="tooltip">
{`${numUsers} users`}
</span>
<ReactTooltip
effect="solid"
backgroundColor={COLORS["tooltip-bg"]}
id="device_mapping"
data-html
>
<span className={`tooltip__tooltip-text`}>{tooltipText}</span>
</ReactTooltip>
</>
{newDeviceMapping.length > 1 ? (
<TooltipWrapper
tipContent={getDeviceUserTipContent(newDeviceMapping)}
>
{displayPrimaryUser}
<span className="device-mapping__more">{` +${
newDeviceMapping.length - 1
} more`}</span>
</TooltipWrapper>
) : (
deviceMapping[0].email || DEFAULT_EMPTY_CELL_VALUE
displayPrimaryUser
)}
</span>
</div>

4
frontend/services/entities/scripts.ts
View file

@ -75,7 +75,7 @@ export interface IHostScriptsResponse {
/**
* Request body for POST /scripts/run
*
* https://fleetdm.com/docs/rest-api/rest-api#run-script-asynchronously
* https://github.com/fleetdm/fleet/blob/main/docs/Contributing/API-for-contributors.md#run-script-asynchronously
*/
export interface IScriptRunRequest {
host_id: number;
@ -86,7 +86,7 @@ export interface IScriptRunRequest {
/**
* Response body for POST /scripts/run
*
* https://fleetdm.com/docs/rest-api/rest-api#run-script-asynchronously
* https://github.com/fleetdm/fleet/blob/main/docs/Contributing/API-for-contributors.md#run-script-asynchronously
*/
export interface IScriptRunResponse {
host_id: number;

77
handbook/ceo.md
View file

@ -27,35 +27,48 @@ From time to time, you will need to schedule an interview between a candidate an
- Add candidate's [LinkedIn url](https://www.linkedin.com/search/results/all/?keywords=people) on the first bullet for Mike.
3. Set the Google Calendar description of the calendar event to: `Agenda: URL_FOR_NEW_COPY_OF_FINAL_INTERVIEW_DOC`
### Program the CEO to do something
1. If necessary or if unsure, immediately direct message the CEO on Slack to clarify priority level, timing, and level of effort. (For example, whether to schedule 30m or 60m to complete in full, or 30m planning as an iterative step.)
2. If there is not room on the calendar to schedule this soon enough with both Mike and Sam as needed (erring on the side of sooner), then either immediately direct message the CEO with a backup plan, or if it can obviously wait, then discuss at the next roundup.
3. Create a calendar event with a Zoom meeting for the CEO and Apprentice. Keep the title short. For the description, keep it very brief and use this template:
```
Agenda:
1. Apprentice: Is there enough context for you (CEO) to accomplish this?
2. Apprentice: Is this still a priority for you (CEO) to do.. right now? Or should it be "someday/maybe"?
3. Apprentice: Is there enough time for you (CEO) to do this live? (Right now during this meeting?)
4. Apprentice: What are the next steps after you (CEO) complete this?
5. Apprentice: LINK_TO_DOC_OR_ISSUE
```
> Keep calendar event titles short so they are readable at a glance. Please include any other info via link, so that information is not duplicated or lost in the calendar.
### Process the CEO's calendar
Time management for the CEO is essential. The Apprentice processes the CEO's calendar by checking for and correcting any double-booking (e.g. two meetings scheduled for overlapping times that the CEO needs to attend) or new meetings added.
Prioritizing in order of importance:
- Travel and personal commitments
- External meetings
- Board meetings
- Priority design reviews
- E-group
- ❌ Reserved until day of
- Customer or prospect calls _(up to the first 4 hours per week)_
- Design reviews
- Puppet show
_Please do not move external meetings, travel, and personal commitments without asking the CEO first._
[Meeting agenda prep](https://docs.google.com/document/d/1gH3IRRgptrqSYzBFy-77g98JROTL8wqrazJIMkp-Gb4/edit#heading=h.i7mkhr6m123r) is especially important to help the CEO focus and transition quickly in and between meetings.
In the notes document include:
1. LinkedIn profile link of all outside participants
2. Screen-shot of LinkedIn profile pic
3. Company name (in doc title and file name)
4. Correct date (20XX-XX-XX in doc title and file name)
5. Context that helps the CEO to understand the purpose of the meeting at a glance from:
- CEO's email
- LinkedIn messages (careful not to mark things as read!)
- Google Drive
Be sure to do this from Mike's browser so as to not lock him out of any meeting docs.
Time management for the CEO is essential. The Apprentice processes the CEO's calendar multiple times per day.
- **Clear any unexpected new events or double-bookings.** Look for any new double-bookings, invites that haven't been accepted, or other events you don't recognize.
1. Double-book temporarily with a "UNCONFIRMED" calendar block so that the CEO ignores it and doesn't spend time trying to figure out what it is.
2. Go to the organizer (or nearest fleetie who's not the CEO):
- Get full context on what the CEO should know as to the purpose of the meeting and why the organizer thinks it is helpful or necessary for the CEO to attend.
- Remind the organizer with [this link to the handbook that all CEO events have times chosen by Sam before booking](https://fleetdm.com/handbook/company/communications#schedule-time-with-the-ceo).
3. Bring prepped discussion item about this proposed event to the next CEO roundup, including the purpose of the event and why it is helpful or necessary for the CEO to attend (according to the person requesting the CEO's attendance). The CEO will decide whether to attend.
4. Delete the "UNCONFIRMED" block if the meeting is confirmed, or otherwise work with the organizer to pick a new time or let them know the decision.
- **Prepare the agenda for any newly-added meetings**: [Meeting agenda prep](https://docs.google.com/document/d/1gH3IRRgptrqSYzBFy-77g98JROTL8wqrazJIMkp-Gb4/edit#heading=h.i7mkhr6m123r) is especially important to help the CEO focus and transition quickly in and between meetings.
- In the notes document include:
1. LinkedIn profile link of all outside participants
2. Screen-shot of LinkedIn profile pic
3. Company name (in doc title and file name)
4. Correct date (20XX-XX-XX in doc title and file name)
5. Context that helps the CEO to understand the purpose of the meeting at a glance from:
- CEO's email
- LinkedIn messages (careful not to mark things as read!)
- Google Drive
- Be sure to do this from the CEO's browser so as to not lock him out of any meeting docs.
### Process the CEO's inbox
- The Apprentice to the CEO is [responsible](https://fleetdm.com/handbook/company/why-this-way#why-direct-responsibility) for [processing all email traffic](https://docs.google.com/document/d/1gH3IRRgptrqSYzBFy-77g98JROTL8wqrazJIMkp-Gb4/edit#heading=h.i7mkhr6m123r) prior to CEO review.
@ -200,7 +213,7 @@ Goal: No one else is currently LinkedIn connecting with community Slack particip
### Process and backup Sid agenda
Every two weeks, our CEO Mike has a meeting with Sid Sijbrandij. The CEO uses dedicated (blocked, recurring) time to prepare for this meeting earlier in the week.
30 minutes After each meeting (to allow all parties to collect action items), the Apprentice makes a copy of the "💻 Sid : Mike(Fleet)" doc and renames it "YYYY-MM-DD Backup of 💻 Sid : Mike(Fleet)". Then moves the backup version into the [(¶¶) Sid archive](https://drive.google.com/drive/u/0/folders/1SP6J-F6M5engq5ivV0Sv3tq8nNIwYFcq)
30 minutes After each meeting (to allow all parties to collect action items), the Apprentice makes a copy of the "💻 Sid : Mike(Fleet)" doc and renames it "YYYY-MM-DD Backup of 💻 Sid : Mike(Fleet)". Then moves the backup version into the [(¶¶) Sid archive](https://drive.google.com/drive/folders/1izVfIBt2nr4APlkm36E6DJg1k1PDjmae)
Then process the backup Sid agenda by:
- Leaving google doc comments assigning all Fleet TODOs to correct Fleeties.
@ -259,10 +272,10 @@ It's not enough to just "delete" a recording of a meeting in Gong. Instead, use
## Rituals
- Note: Some rituals are especially time-sensitive and require attention multiple times per day (⏰). Set reminders for the following times (CT):
- 9:30 AM /before start of business
- 12:30 PM /beginning of "reserved block"
- 6:30 PM /post-mortem days meetings
- Note: Some rituals (⏰) are especially time-sensitive and require attention multiple times (3+) per day. Set reminders for the following times (CT):
- 9:30 AM _(/before first meeting)_
- 12:30 PM CT _(/beginning of "reserved block")_
- 6:30 PM CT _(/after last meeting, before roundup / Japan calls)_
<rituals :rituals="rituals['handbook/company/ceo.rituals.yml']"></rituals>

2
handbook/company/README.md
View file

@ -134,7 +134,7 @@ Fleet raised its Series A funding round. The world now has at least 1.65 millio
## Org chart
To provide clarity about decision-making, [responsibility](https://fleetdm.com/handbook/company/why-this-way#why-direct-responsibility), and resources, everyone at Fleet has a manager, and [every manager](https://fleetdm.com/handbook/company#management) has direct reports. Fleet's organizational chart is accessible company-wide as a sub-tab in ["🧑‍🚀 Fleeties" (private google doc)](https://docs.google.com/spreadsheets/d/1OSLn-ZCbGSjPusHPiR5dwQhheH1K8-xqyZdsOe9y7qc/edit#gid=0). On the other sub-tabs, you can also check out a world map of where everyone is located, hiring stats, and fun facts about each team member.
- 🔦 [Business Operations](https://fleetdm.com/handbook/business-operations): The Business Operations department is directly responsible for these traditional functions: People, Finance, Legal, IT, and Revenue Operations (RevOps).
- 🔦 [Business Operations](https://fleetdm.com/handbook/business-operations): The Business Operations department is directly responsible for these traditional functions: People, Finance, tax, compliance, Legal, and IT.
- 🏹 [Customer Success](https://fleetdm.com/handbook/customer-success): The customer success department is directly responsible for ensuring that customers and community members of Fleet achieve their desired outcomes with Fleet products and services.
- 🐋 [Sales](https://fleetdm.com/handbook/sales): The Sales department is directly responsible for attaining the revenue goals of Fleet and helping customers deliver on their objectives.
- 🫧 [Demand](https://fleetdm.com/handbook/demand): The Demand department is directly responsible for growing awareness of Fleet and nurturing the community through participation in events, conversations, and other programs.

29
handbook/company/pricing-features-table.yml
View file

@ -5,7 +5,7 @@
- industryName: Device health
friendlyName: Automate device health
description: Automatically report system health issues using webhooks or integrations, to notify or quarantine outdated or misconfigured systems that are at higher risk of vulnerabilities or theft.
documentationUrl: https://fleetdm.com/docs/rest-api/rest-api#get-host
documentationUrl: https://fleetdm.com/docs/using-fleet/automations#automations
screenshotSrc:
tier: Free
productCategories: [Endpoint operations]
@ -213,7 +213,7 @@
- industryName: Detection engineering
friendlyName: # Ship logs to your data lake and comopare with known bad binary hashes or capture behavioral data and build custom detections (e.g. using a framework like MITRE)
description:
documentationUrl:
documentationUrl: https://fleetdm.com/docs/using-fleet/log-destinations
tier: Free
dri: mikermcneil
usualDepartment: Security
@ -227,7 +227,7 @@
- industryName: Threat hunting
friendlyName: # TODO: live query
description:
documentationUrl:
documentationUrl: https://fleetdm.com/queries
tier: Free
dri: mikermcneil
usualDepartment: Security
@ -278,6 +278,7 @@
- industryName: Agent auto-update
friendlyName: Keep agents and extensions up to date
descrption: Keep agents and extensions up to date by loading code from Fleet's free update registry.
documentationUrl: https://fleetdm.com/docs/using-fleet/fleetd
tier: Free
productCategories: [Endpoint operations]
usualDepartment: IT
@ -288,6 +289,7 @@
tier: Free
productCategories: [Endpoint operations]
usualDepartment: IT
documentationUrl: https://fleetdm.com/docs/using-fleet/fleetd
waysToUse:
- description: Build scripts for Ansible deployments
moreInfoUrl: https://www.youtube.com/watch?v=qflUfLQCnwY&list=PL6-FgoWOoK2YUR4ADGsxTSL3onb-GzCnM&index=4
@ -300,6 +302,7 @@
# ╚═╝╩ ╩ ╩ ╚═╝╩ ╩ ╩╝╚╝╚═╝ ╩ ╩ ╩╩═╝╩═╝╩ ╩ ╩ ╩╚═╝╝╚╝
- industryName: Batch installation (Chef, Ansible, Puppet, MDM)
friendlyName: Install agents over the air
documentationUrl: https://fleetdm.com/docs/using-fleet/fleetd
tier: Free
productCategories: [Endpoint operations]
usualDepartment: IT
@ -308,6 +311,7 @@
# ╩╚═╚═╝╩ ╩╚═╝ ╩ ╚═╝ ╚═╝╚═╝ ╩ ╩ ╩╝╚╝╚═╝╚═╝
- industryName: Remote settings
description: Configure agent options remotely, over the air. (Includes osquery config, and osquery startup flags.). Fleetd startup flags coming soon (2023-12-31) #customer-blanco
documentationUrl: https://fleetdm.com/docs/configuration/agent-configuration
moreInfoUrl: https://github.com/fleetdm/fleet/issues/13825
tier: Free
productCategories: [Endpoint operations]
@ -317,8 +321,9 @@
# ╚╝ ╩ ╩╩╚═╩╩ ╩╚═╝╩═╝╚═╝ ╚═╝╝╚╝╩╚═╚═╝╩═╝╩═╝╩ ╩╚═╝╝╚╝ ╩
- industryName: Variable enrollment
description: Enroll hosts in different groups using different enrollment secrets and/or installers per-baseline.
documentationUrl: https://fleetdm.com/docs/configuration/configuration-files#teams
tier: Premium
productCategories: [Endpoint operations,Device management]
productCategories: [Endpoint operations, Device management]
usualDepartment: IT
# ╔═╗╦═╗╦╦ ╦╔═╗╔╦╗╔═╗ ╦ ╦╔═╗╔╦╗╔═╗╔╦╗╔═╗ ╦═╗╔═╗╔═╗╦╔═╗╔╦╗╦═╗╦ ╦
# ╠═╝╠╦╝║╚╗╔╝╠═╣ ║ ║╣ ║ ║╠═╝ ║║╠═╣ ║ ║╣ ╠╦╝║╣ ║ ╦║╚═╗ ║ ╠╦╝╚╦╝
@ -326,6 +331,7 @@
- industryName: Private update registry
friendlyName: Update agents from a secret URL
description: Load agent code from a secret URL that you manage.
documentationUrl: https://fleetdm.com/docs/using-fleet/update-agents
tier: Premium
productCategories: [Endpoint operations]
usualDepartment: Security
@ -335,6 +341,7 @@
- industryName: Custom tables
friendlyName: Add tables to osquery with extensions
description: Install osquery extensions over the air. # (GitOptional)
documentationUrl: https://fleetdm.com/docs/configuration/agent-configuration#extensions
moreInfoUrl: https://github.com/trailofbits/osquery-extensions/blob/3df2b72ad78549e25344c79dbc9bce6808c4d92a/README.md#extensions
tier: Premium
productCategories: [Endpoint operations]
@ -357,6 +364,7 @@
# ╚═╝╚═╝╩ ╩╩ ╩╩ ╩╝╚╝═╩╝ ╩═╝╩╝╚╝╚═╝ ╩ ╚═╝╚═╝╩═╝ └─ ╚═╝╩═╝╩ ─┘
- industryName: Command line tool (CLI)
friendlyName: fleetctl
documentationUrl: https://fleetdm.com/docs/using-fleet/fleetctl-cli
productCategories: [Endpoint operations,Device management]
usualDepartment: IT
tier: Free
@ -456,6 +464,7 @@
tier: Premium
- industryName: Self-managed
friendlyName: Host it yourself
documentationUrl: https://fleetdm.com/docs/deploy/introduction
productCategories: [Endpoint operations,Device management,Vulnerability management]
tier: Free
buzzwords: [Self-hosted]
@ -468,9 +477,11 @@
tier: Premium
- industryName: Interactive MDM migration # « end-user initiated MDM migration, with interactive UI
tier: Premium
documentationUrl: ""
usualDepartment: IT
productCategories: [Device management]
- industryName: Remotely enforce OS settings
documentationUrl: https://fleetdm.com/docs/using-fleet/mdm-custom-macos-settings
tier: Free
usualDepartment: IT
waysToUse:
@ -484,22 +495,27 @@
productCategories: [Device management]
- industryName: Self service
description: Provide resolution instructions for end users through Fleet Desktop that suggest how an end user can fix a posture issue themselves.
documentationUrl: https://fleetdm.com/docs/using-fleet/fleet-desktop
tier: Premium
usualDepartment: IT
productCategories: [Device management]
- industryName: User-initiated enrollment of macOS computers
documentationUrl: https://fleetdm.com/docs/using-fleet/mdm-migration-guide#migrate-manually-enrolled-hosts
tier: Free
usualDepartment: IT
productCategories: [Device management]
- industryName: Low-level MDM commands for macOS and Windows (e.g. remote restart)
documentationUrl: https://fleetdm.com/docs/using-fleet/mdm-commands
tier: Free
usualDepartment: IT
productCategories: [Device management]
- industryName: Native macOS update reminders
documentationUrl: https://fleetdm.com/docs/using-fleet/mdm-macos-updates
tier: Free
usualDepartment: IT
productCategories: [Device management]
- industryName: Zero-touch setup for macOS computers
documentationUrl: https://fleetdm.com/docs/using-fleet/mdm-macos-setup-experience
tier: Premium
usualDepartment: IT
productCategories: [Device management]
@ -509,6 +525,7 @@
- description: Customize the out-of-the-box setup experience for your end users.
- description: Require end users to authenticate with your identity provider (IdP) and agree to an end user license agreement (EULA) before they can use their new workstation
- industryName: Enforce OS updates
documentationUrl: https://fleetdm.com/docs/using-fleet/mdm-macos-updates
tier: Premium
usualDepartment: IT
productCategories: [Device management,Vulnerability management]
@ -516,10 +533,12 @@
- description: Enforce macOS updates via Nudge.
- description: Automatically update Windows after the end user reaches a deadline. Coming soon (2023-12-30) #Customer-preston
- industryName: Encrypt macOS hard disks with FileVault
documentationUrl: https://fleetdm.com/docs/using-fleet/mdm-disk-encryption
tier: Premium
usualDepartment: IT
productCategories: [Device management]
- industryName: Remotely lock and wipe macOS computers
documentationUrl: https://fleetdm.com/docs/using-fleet/mdm-commands
tier: Premium
usualDepartment: IT
productCategories: [Device management]
@ -531,6 +550,7 @@
usualDepartment: IT
productCategories: [Device management]
- industryName: Puppet module
documentationUrl: https://fleetdm.com/docs/using-fleet/puppet-module
friendlyName: Map macOS settings to computers with Puppet module
tier: Premium
usualDepartment: IT
@ -584,6 +604,7 @@
usualDepartment: IT
tier: Premium
- industryName: Versionable queries and config (GitOps)
documentationUrl: https://fleetdm.com/guides/using-github-actions-to-apply-configuration-profiles-with-fleet#basic-article
tier: Free
productCategories: [Endpoint operations,Device management,Vulnerability management]
usualDepartment: IT

2
handbook/company/product-groups.md
View file

@ -218,7 +218,7 @@ Anyone in the product group can initiate an air guitar session.
5. Document: Summarize the learnings, decisions, and next steps in the user story issue.
6. Decide: Bring the issue to a design review to determine an outcome:
6. Decide: Assign the issue to the Head of Product Design to determine an outcome:
1. Move forward with the formal drafting process leading to engineering.
2. Keep it open for future consideration.
3. Discard if it is invalidated through the process.

12
handbook/product/README.md
View file

@ -65,10 +65,10 @@ When starting a new draft:
> As drafting occurs, inevitably, the requirements will change. The main description of the issue should be the single source of truth for the problem to be solved and the required outcome. The product manager is responsible for keeping the main description of the issue up-to-date. Comments and other items can and should be kept in the issue for historical record-keeping.
### Ensure product user-story is complete
Once the draft has been approved, it moves to the "Settled" column on the drafting board.
### Ensure story drafting is complete
Once a story has gone through design and is considered "Settled", it moves to the "Settled" column on the drafting board and assign to the Engineering Manager (EM).
Before assigning an engineering manager to [estimate](https://fleetdm.com/handbook/engineering#sprint-ceremonies) a user story, the product designer ensures the product section of the user story [checklist](https://github.com/fleetdm/fleet/issues/new?assignees=&labels=story&projects=&template=story.md&title=) is complete.
Before assigning an EM to [estimate](https://fleetdm.com/handbook/engineering#sprint-ceremonies) a user story, the product designer ensures the product section of the user story [checklist](https://github.com/fleetdm/fleet/issues/new?assignees=&labels=story&projects=&template=story.md&title=) is complete.
Once a bug has gone through design and is considered "Settled", the designer removes the `:product` label and moves the issue to the 'Sprint backlog' column on the "Bugs" board and assigns the group engineering manager.
@ -158,8 +158,8 @@ The following highlights should be considered when deciding if we promote a feat
explains why the feature is advertised as "beta" and tracking the feature's progress towards advertising the feature as "stable."
- The feature will be advertised as "beta" in the documentation on fleetdm.com/docs, release notes, release blog posts, and Twitter.
### Maintain compatibility with current versions of operating systems and CIS benchmarks
Fleet's product offerings depend on the capabilities of other platforms. This requires the ongoing attention of the product and engineering teams to ensure that we are up-to-date with new capabilities and that our existing capabilities continue to function. The first step to staying up-to-date with Fleet's partners is to know when the partner platform changes.
### Maintain current versions
Fleet's product depends on the capabilities of other platforms.
Every week, a member of the product team looks up whether there is:
1. a new major or minor version of [macOS](https://support.apple.com/en-us/HT201260)
@ -270,7 +270,7 @@ Please see [handbook/product#revise-a-draft-currently-in-development](https://fl
Please see [handbook/product#outside-contributions](https://fleetdm.com/handbook/product#outside-contributions)
##### Prioritizing bugs
handbook/product#correctly-prioritize-a-bug](https://fleetdm.com/handbook/product#correctly-prioritize-a-bug)
Please see [handbook/product#correctly-prioritize-a-bug](https://fleetdm.com/handbook/product#correctly-prioritize-a-bug)
##### Writing user stories
Please see [handbook/product#write-a-user-story](https://fleetdm.com/handbook/product#write-a-user-story)

2
infrastructure/dogfood/terraform/aws/variables.tf
View file

@ -56,7 +56,7 @@ variable "database_name" {
variable "fleet_image" {
description = "the name of the container image to run"
default = "fleetdm/fleet:v4.41.0"
default = "fleetdm/fleet:v4.41.1"
}
variable "software_inventory" {

2
infrastructure/dogfood/terraform/gcp/variables.tf
View file

@ -68,5 +68,5 @@ variable "redis_mem" {
}
variable "image" {
default = "fleet:v4.41.0"
default = "fleet:v4.41.1"
}

6
infrastructure/loadtesting/terraform/readme.md
View file

@ -29,7 +29,7 @@ terraform apply -var tag=hosts-5k-test -var fleet_containers=5 -var db_instance_
1. arm64 (M1/M2/etc) Mac Only: run `helpers/setup-darwin_arm64.sh` to build terraform plugins that lack arm64 builds in the registry. Alternatively, you can use the amd64 terraform binary, which works with Rosetta 2.
1. Log into AWS SSO on `loadtesting` via `aws sso login`. (If you have multiple profiles, export the `AWS_PROFILE` variable.) For configuration, see `infrastructure/sso` folder's readme in the `confidential` private repo.
1. Initialize your terraform environment with `terraform init`.
1. Select a workspace for your test: `terraform workspace new WORKSPACE-NAME; terraform workspace select WORKSPACE-NAME`. Ensure your `WORKSPACE-NAME` is less than or equal to 17 characters and contains only alphanumeric characters and hyphens, as it is used to generate names for AWS resources.
1. Select a workspace for your test: `terraform workspace new WORKSPACE-NAME; terraform workspace select WORKSPACE-NAME`. Ensure your `WORKSPACE-NAME` is less than or equal to 17 characters and contains only lowercase alphanumeric characters and hyphens, as it is used to generate names for AWS resources.
1. Apply terraform with your branch name with `terraform apply -var tag=BRANCH_NAME` and type `yes` to approve execution of the plan. This takes a while to complete (many minutes, > ~30m). Note that for a few minutes after `terraform apply`, the Fleet instances may be failing to start with a permission issue (to read a database secret), but this should resolve automatically after a bit and ECS will begin to start the Fleet instances, but they may still fail due to missing database migrations (this will show up in the instances' logs). At this point you can move on to the next step.
1. Run database migrations (see [Running migrations](#running-migrations)). You will get 500 errors and your containers will not run if you do not do this. After running this step, you might need to wait a few minutes until the environment is up and running.
1. Perform your tests (see [Running a loadtest](#running-a-loadtest)). Your deployment will be available at `https://WORKSPACE-NAME.loadtest.fleetdm.com`. Reach out to the infrastructure team to get the credentials to log in.
@ -115,9 +115,9 @@ terraform apply -var tag=BRANCH_NAME -var loadtest_containers=XXX -target=aws_ec
#### Using a release tag instead of a branch
Since the tag name on Dockerhub doesn't match the tag name on GitHub, this presents a special use case when wanting to deploy a release tag. In this case, you can use the optional `-var github_branch` in order to specify the separate tag. For example, you would use the following to deploy a loadtest of version 4.28.0:
Since the tag name on Dockerhub doesn't match the tag name on GitHub, this presents a special use case when wanting to deploy a release tag. In this case, you can use the optional `-var git_branch` in order to specify the separate tag. For example, you would use the following to deploy a loadtest of version 4.28.0:
`terraform apply -var tag=v4.28.0 -var github_branch=fleet-v4.28.0 -var loadtest_containers=8`
`terraform apply -var tag=v4.28.0 -var git_branch=fleet-v4.28.0 -var loadtest_containers=8`
#### General Troubleshooting

2
infrastructure/sandbox/JITProvisioner/jitprovisioner.tf
View file

@ -206,7 +206,7 @@ resource "random_uuid" "jitprovisioner" {
# Use the local to make the trigger work.
locals {
fleet_tag = "v4.41.0"
fleet_tag = "v4.41.1"
}
resource "null_resource" "standard-query-library" {

2
infrastructure/sandbox/PreProvisioner/lambda/deploy_terraform/main.tf
View file

@ -165,7 +165,7 @@ resource "helm_release" "main" {
set {
name = "imageTag"
value = "v4.41.0"
value = "v4.41.1"
}
set {

4
mdm_profiles/password_policy.mobileconfig
View file

@ -30,7 +30,7 @@
<key>maxInactivity</key>
<integer>15</integer>
<key>minLength</key>
<integer>11</integer>
<integer>10</integer>
<key>requireAlphanumeric</key>
<true/>
</dict>
@ -38,7 +38,7 @@
<key>PayloadDescription</key>
<string>Configures our Macs to require passwords that are 10 character long</string>
<key>PayloadDisplayName</key>
<string>Enforce password length (11 characters)</string>
<string>Enforce password length (10 characters)</string>
<key>PayloadIdentifier</key>
<string>com.github.erikberglund.ProfileCreator.F7CF282E-D91B-44E9-922F-A719634F9C8E</string>
<key>PayloadOrganization</key>

1
orbit/changes/14176-orbit-retries Normal file
View file

@ -0,0 +1 @@
* Add backoff functionality to download `fleetd` updates. With this update, `fleetd` is going to retry 3 times and then wait 24 hours to try again.

1
orbit/changes/15539-removing-duplicate-device_token-calls Normal file
View file

@ -0,0 +1 @@
Fixing fleetd to NOT make unnecessary duplicate call to orbit/device_token endpoint.

1
orbit/changes/15541-reduce-agent-calls-to-orbit-config-endpoint Normal file
View file

@ -0,0 +1 @@
Reducing the number of fleetd calls to fleet/orbit/config endpoint by caching the config for 3 seconds.

48
orbit/cmd/orbit/orbit.go
View file

@ -34,6 +34,7 @@ import (
"github.com/fleetdm/fleet/v4/orbit/pkg/update/filestore"
"github.com/fleetdm/fleet/v4/pkg/certificate"
"github.com/fleetdm/fleet/v4/pkg/file"
retrypkg "github.com/fleetdm/fleet/v4/pkg/retry"
"github.com/fleetdm/fleet/v4/pkg/secure"
"github.com/fleetdm/fleet/v4/server/fleet"
"github.com/fleetdm/fleet/v4/server/service"
@ -404,21 +405,44 @@ func main() {
g.Add(updateRunner.Execute, updateRunner.Interrupt)
osquerydLocalTarget, err := updater.Get("osqueryd")
if err != nil {
return fmt.Errorf("get osqueryd target: %w", err)
}
osquerydPath = osquerydLocalTarget.ExecPath
if c.Bool("fleet-desktop") {
fleetDesktopLocalTarget, err := updater.Get("desktop")
// if getting any of the targets fails, keep on
// retrying, the `updater.Get` method has built-in backoff functionality.
//
// NOTE: it used to be the case that we would return an
// error on the first attempt here, causing orbit to
// restart. This was changed to have control over
// how/when we want to retry to download the packages.
err = retrypkg.Do(func() error {
osquerydLocalTarget, err := updater.Get("osqueryd")
if err != nil {
return fmt.Errorf("get desktop target: %w", err)
log.Info().Err(err).Msg("get osqueryd target failed")
return fmt.Errorf("get osqueryd target: %w", err)
}
if runtime.GOOS == "darwin" {
desktopPath = fleetDesktopLocalTarget.DirPath
} else {
desktopPath = fleetDesktopLocalTarget.ExecPath
osquerydPath = osquerydLocalTarget.ExecPath
if c.Bool("fleet-desktop") {
fleetDesktopLocalTarget, err := updater.Get("desktop")
if err != nil {
log.Info().Err(err).Msg("get desktop target failed")
return fmt.Errorf("get desktop target: %w", err)
}
if runtime.GOOS == "darwin" {
desktopPath = fleetDesktopLocalTarget.DirPath
} else {
desktopPath = fleetDesktopLocalTarget.ExecPath
}
}
return nil
},
// retry every 5 minutes to not flood the logs,
// but actual pings to the remote server are
// handled by `updater.Get`
retrypkg.WithInterval(5*time.Minute),
)
if err != nil {
// this should never happen because `retry.Do` is
// executed without a defined number of max attempts
return fmt.Errorf("getting targets after retry: %w", err)
}
} else {
log.Info().Msg("running with auto updates disabled")

4
orbit/pkg/token/readwriter.go
View file

@ -53,10 +53,6 @@ func (rw *ReadWriter) Rotate() error {
}
id := uuid.String()
if err := rw.Write(id); err != nil {
return fmt.Errorf("writing token: %w", err)
}
attempts := 3
interval := 5 * time.Second
err = retry.Do(func() error {

6
orbit/pkg/update/nudge_test.go
View file

@ -12,6 +12,7 @@ import (
"github.com/fleetdm/fleet/v4/orbit/pkg/constant"
"github.com/fleetdm/fleet/v4/pkg/optjson"
"github.com/fleetdm/fleet/v4/pkg/retry"
"github.com/fleetdm/fleet/v4/server/fleet"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
@ -55,8 +56,9 @@ func (s *nudgeTestSuite) TestNudgeConfigFetcherAddNudge() {
t := s.T()
tmpDir := t.TempDir()
updater := &Updater{
client: s.client,
opt: Options{Targets: make(map[string]TargetInfo), RootDirectory: tmpDir},
client: s.client,
opt: Options{Targets: make(map[string]TargetInfo), RootDirectory: tmpDir},
retryer: retry.NewLimitedWithCooldown(3, 1*time.Second),
}
runner := &Runner{updater: updater, localHashes: make(map[string][]byte)}
interval := time.Second

43
orbit/pkg/update/update.go
View file

@ -15,12 +15,14 @@ import (
"runtime"
"strings"
"sync"
"time"
"github.com/fleetdm/fleet/v4/orbit/pkg/build"
"github.com/fleetdm/fleet/v4/orbit/pkg/constant"
"github.com/fleetdm/fleet/v4/orbit/pkg/platform"
"github.com/fleetdm/fleet/v4/pkg/certificate"
"github.com/fleetdm/fleet/v4/pkg/fleethttp"
"github.com/fleetdm/fleet/v4/pkg/retry"
"github.com/fleetdm/fleet/v4/pkg/secure"
"github.com/rs/zerolog/log"
"github.com/theupdateframework/go-tuf/client"
@ -40,9 +42,10 @@ const (
// Updater supports updating plain executables and
// .tar.gz compressed executables.
type Updater struct {
opt Options
client *client.Client
mu sync.Mutex
opt Options
client *client.Client
retryer *retry.LimitedWithCooldown
mu sync.Mutex
}
// Options are the options that can be provided when creating an Updater.
@ -171,6 +174,9 @@ func NewUpdater(opt Options) (*Updater, error) {
updater := &Updater{
opt: opt,
client: tufClient,
// per product spec, retry up to three consecutive times, then
// wait 24 hours to try again.
retryer: retry.NewLimitedWithCooldown(3, 24*time.Hour),
}
if err := updater.initializeDirectories(); err != nil {
@ -315,6 +321,37 @@ func (u *Updater) Targets() (data.TargetFiles, error) {
// Get downloads (if it doesn't exist) a target and returns its local information.
func (u *Updater) Get(target string) (*LocalTarget, error) {
meta, err := u.Lookup(target)
if err != nil {
return nil, err
}
// use the specific target hash as the key for retries. This allows new
// updates to the TUF server to be downloaded immediately without
// having to wait the cooldown.
key := target
if _, metaHash, err := selectHashFunction(meta); err == nil {
key = fmt.Sprintf("%x", metaHash)
}
var localTarget *LocalTarget
err = u.retryer.Do(key, func() error {
var err error
localTarget, err = u.get(target)
return err
})
if err != nil {
var rErr *retry.ExcessRetriesError
if errors.As(err, &rErr) {
return nil, fmt.Errorf("skipped getting target: %w", err)
}
return nil, fmt.Errorf("getting target: %w", err)
}
return localTarget, nil
}
func (u *Updater) get(target string) (*LocalTarget, error) {
if target == "" {
return nil, errors.New("target is required")
}

81
pkg/retry/limited_with_cooldown.go Normal file
View file

@ -0,0 +1,81 @@
package retry
import (
"fmt"
"sync"
"time"
)
// ExcessRetriesError is returned when the number of retries for a specific
// hash exceeds the maximum allowed limit and the cooldown period has not yet
// elapsed.
//
// It indicates that the function associated with the hash will not
// be retried until the cooldown period is over.
type ExcessRetriesError struct {
nextRetry time.Duration
}
func (e *ExcessRetriesError) Error() string {
return fmt.Sprintf("max retries exceeded, retrying again in %v", e.nextRetry)
}
// LimitedWithCooldown manages function calls with a limit on retries and a
// cooldown period.
// It tracks retries and wait times for each unique hash key.
//
// NOTE: we also have a backoff package widely used in the repo
// (github.com/cenkalti/backoff/v4), but I'm implementing a custom struct due to:
// - the very specific product requirements (retry n consecutive times per
// hash, then wait for a defined cooldown.)
// - this retry mechanism is used by the `fleetd` updater and needs to be
// thread safe.
type LimitedWithCooldown struct {
maxRetries int // maxRetries is the maximum number of retries allowed before entering cooldown.
cooldown time.Duration // cooldown is the duration to wait before resetting the retry count.
retries map[string]int // retries tracks the number of retries for each hash key.
wait map[string]time.Time // wait tracks the start of the cooldown period for each hash key.
mu sync.Mutex // mu is used to ensure thread safety.
}
// NewLimitedWithCooldown creates a new instance of LimitedWithCooldown.
// - maxRetries specifies the maximum number of retries allowed for a function call.
// - cooldown specifies the duration to wait before allowing retries again.
func NewLimitedWithCooldown(maxRetries int, cooldown time.Duration) *LimitedWithCooldown {
return &LimitedWithCooldown{
maxRetries: maxRetries,
cooldown: cooldown,
retries: map[string]int{},
wait: map[string]time.Time{},
}
}
// Do executes the provided function fn associated with the given hash.
// It applies the retry and cooldown logic based on previous attempts with the same hash.
// - hash is a unique identifier for the function call context.
// - fn is the function to be executed; it must return an error to indicate success or failure.
//
// If the retries exceed maxRetries and the cooldown period has not passed, ErrRetriesExceeded is returned.
// If fn executes successfully, the retry count and wait time for the hash are reset.
// Returns an error if fn fails and has not exceeded the max retries or cooldown period.
func (t *LimitedWithCooldown) Do(hash string, fn func() error) error {
t.mu.Lock()
defer t.mu.Unlock()
if t.retries[hash] >= t.maxRetries &&
time.Since(t.wait[hash]) <= t.cooldown {
return &ExcessRetriesError{nextRetry: time.Until(t.wait[hash])}
}
if err := fn(); err != nil {
t.retries[hash]++
if t.retries[hash] >= t.maxRetries {
t.wait[hash] = time.Now()
}
return err
}
t.retries[hash] = 0
t.wait[hash] = time.Time{}
return nil
}

112
pkg/retry/limited_with_cooldown_test.go Normal file
View file

@ -0,0 +1,112 @@
package retry
import (
"errors"
"testing"
"time"
"github.com/stretchr/testify/require"
)
func TestNewLimitedWithCooldown(t *testing.T) {
lwc := NewLimitedWithCooldown(3, 1*time.Hour)
require.NotNil(t, lwc)
require.Equal(t, 3, lwc.maxRetries)
require.Equal(t, 1*time.Hour, lwc.cooldown)
}
func TestLimitedWithCooldwonDo(t *testing.T) {
t.Run("success", func(t *testing.T) {
lwc := NewLimitedWithCooldown(3, 1*time.Hour)
hash := "foo"
err := lwc.Do(hash, func() error {
return nil
})
require.NoError(t, err)
require.Equal(t, 0, lwc.retries[hash])
require.True(t, lwc.wait[hash].IsZero())
})
t.Run("fail and retry", func(t *testing.T) {
lwc := NewLimitedWithCooldown(3, 1*time.Hour)
hash := "foo"
// failures followed by a success
for i := 0; i < 2; i++ {
err := lwc.Do(hash, func() error {
return errors.New("failure")
})
require.Error(t, err)
require.Equal(t, i+1, lwc.retries[hash])
}
err := lwc.Do(hash, func() error {
return nil
})
require.NoError(t, err)
require.Equal(t, 0, lwc.retries[hash])
require.True(t, lwc.wait[hash].IsZero())
})
t.Run("exceed max retries", func(t *testing.T) {
lwc := NewLimitedWithCooldown(3, 1*time.Hour)
hash := "foo"
for i := 0; i < 3; i++ {
err := lwc.Do(hash, func() error {
return errors.New("failure")
})
require.Error(t, err)
}
err := lwc.Do(hash, func() error {
return nil
})
var rErr *ExcessRetriesError
require.ErrorAs(t, err, &rErr)
})
t.Run("cooldown period", func(t *testing.T) {
lwc := NewLimitedWithCooldown(3, 1*time.Millisecond)
hash := "foo"
for i := 0; i < 3; i++ {
err := lwc.Do(hash, func() error {
return errors.New("failure")
})
require.Error(t, err)
}
time.Sleep(2 * time.Millisecond)
err := lwc.Do(hash, func() error {
return nil
})
require.NoError(t, err)
require.Equal(t, 0, lwc.retries[hash])
require.True(t, lwc.wait[hash].IsZero())
})
t.Run("multiple hashes", func(t *testing.T) {
lwc := NewLimitedWithCooldown(3, 1*time.Hour)
hash1 := "hash1"
hash2 := "hash2"
// Fail for hash1
err1 := lwc.Do(hash1, func() error {
return errors.New("failure")
})
require.Error(t, err1)
// Succeed for hash2
err2 := lwc.Do(hash2, func() error {
return nil
})
require.NoError(t, err2)
require.Equal(t, 1, lwc.retries[hash1])
require.Equal(t, 0, lwc.retries[hash2])
})
}

8
schema/osquery_fleet_schema.json
View file

@ -1389,7 +1389,9 @@
"evented": false,
"cacheable": false,
"notes": "",
"examples": "List installed Atom packages and their version.\n```\nSELECT name, version, description FROM atom_packages;\n```",
"examples": [
"select * from atom_packages"
],
"columns": [
{
"name": "name",
@ -1452,10 +1454,10 @@
"notes": "",
"hidden": false,
"required": false,
"index": true,
"requires_user_context": true
"index": true
}
],
"hidden": true,
"fleetRepoUrl": "https://github.com/fleetdm/fleet/blob/main/schema/tables/atom_packages.yml"
},
{

15
server/authz/policy.rego
View file

@ -261,6 +261,21 @@ allow {
action == write
}
# Allow read for host health for global admin/maintainer, team admins, observer.
allow {
object.type == "host_health"
subject.global_role == [admin, maintainer, observer][_]
action == read
}
# Allow read for host health for team admin/maintainer, team admins, observer.
allow {
object.type == "host_health"
team_role(subject, object.team_id) == [admin, maintainer, observer][_]
action == read
}
##
# Labels
##

21
server/authz/policy_test.go
View file

@ -2087,3 +2087,24 @@ func TestJSONToInterfaceUser(t *testing.T) {
assert.Equal(t, json.Number("42"), subject["teams"].([]interface{})[1].(map[string]interface{})["id"])
}
}
func TestHostHealth(t *testing.T) {
t.Parallel()
hostHealth := &fleet.HostHealth{TeamID: ptr.Uint(1)}
runTestCases(t, []authTestCase{
{user: nil, object: hostHealth, action: read, allow: false},
{user: test.UserGitOps, object: hostHealth, action: read, allow: false},
{user: test.UserTeamGitOpsTeam1, object: hostHealth, action: read, allow: false},
{user: test.UserTeamGitOpsTeam2, object: hostHealth, action: read, allow: false},
{user: test.UserAdmin, object: hostHealth, action: read, allow: true},
{user: test.UserTeamAdminTeam1, object: hostHealth, action: read, allow: true},
{user: test.UserTeamAdminTeam2, object: hostHealth, action: read, allow: false},
{user: test.UserObserver, object: hostHealth, action: read, allow: true},
{user: test.UserTeamObserverTeam1, object: hostHealth, action: read, allow: true},
{user: test.UserTeamObserverTeam2, object: hostHealth, action: read, allow: false},
{user: test.UserMaintainer, object: hostHealth, action: read, allow: true},
{user: test.UserTeamMaintainerTeam1, object: hostHealth, action: read, allow: true},
{user: test.UserTeamMaintainerTeam2, object: hostHealth, action: read, allow: false},
})
}

10
server/config/config.go
View file

@ -1741,13 +1741,3 @@ func SetTestMDMConfig(t testing.TB, cfg *FleetConfig, cert, key []byte, appleBMT
t.Fatal(err)
}
}
// Undocumented feature flag for Windows MDM, used to determine if the Windows
// MDM feature is visible in the UI and can be enabled. More details here:
// https://github.com/fleetdm/fleet/issues/12257
//
// TODO: remove this flag once the Windows MDM feature is ready for
// release.
func IsMDMFeatureFlagEnabled() bool {
return os.Getenv("FLEET_DEV_MDM_ENABLED") == "1"
}

74
server/datastore/cached_mysql/bench_cached_mysql_test.go
View file

@ -1,74 +0,0 @@
package cached_mysql
import (
"context"
"testing"
"time"
"github.com/fleetdm/fleet/v4/pkg/optjson"
"github.com/fleetdm/fleet/v4/server/fleet"
"github.com/fleetdm/fleet/v4/server/ptr"
"github.com/patrickmn/go-cache"
"github.com/stretchr/testify/require"
)
type unclonableTeamMDMConfig fleet.TeamMDM
// This exported variable is to make sure that the compiler doesn't optimize
// away the benchmarked function call.
var Result interface{}
// On my laptop, results are as follows. Under load, the reflection-based
// approach really adds up and is CPU intensive, resulting in drastic
// performance drops.
//
// goos: linux
// goarch: amd64
// pkg: github.com/fleetdm/fleet/v4/server/datastore/cached_mysql
// cpu: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
// BenchmarkCacheGetFallbackClone-8 53228 22706 ns/op 11976 B/op 217 allocs/op
// BenchmarkCacheGetCustomClone-8 5741186 196.3 ns/op 177 B/op 3 allocs/op
func BenchmarkCacheGetFallbackClone(b *testing.B) {
v := unclonableTeamMDMConfig(cachedValue())
benchmarkCacheGet(b, &v)
}
func BenchmarkCacheGetCustomClone(b *testing.B) {
v := cachedValue()
benchmarkCacheGet(b, &v)
}
func benchmarkCacheGet(b *testing.B, v any) {
ctx := context.Background()
c := &cloneCache{cache.New(time.Minute, time.Minute)}
c.Set(ctx, "k", v, cache.DefaultExpiration)
b.ResetTimer()
var ok bool
for i := 0; i < b.N; i++ {
Result, ok = c.Get(ctx, "k")
if !ok {
b.Fatal("expected ok")
}
}
require.Equal(b, v, Result)
}
func cachedValue() fleet.TeamMDM {
return fleet.TeamMDM{
EnableDiskEncryption: true,
MacOSUpdates: fleet.MacOSUpdates{
MinimumVersion: optjson.SetString("10.10.10"),
Deadline: optjson.SetString("1992-03-01"),
},
MacOSSettings: fleet.MacOSSettings{
CustomSettings: []string{"a", "b"},
DeprecatedEnableDiskEncryption: ptr.Bool(false),
},
MacOSSetup: fleet.MacOSSetup{
BootstrapPackage: optjson.SetString("bootstrap"),
},
}
}

116
server/datastore/cached_mysql/cached_mysql.go
View file

@ -4,15 +4,35 @@ import (
"context"
"encoding/json"
"fmt"
"reflect"
"time"
"github.com/fleetdm/fleet/v4/server/contexts/ctxdb"
"github.com/fleetdm/fleet/v4/server/fleet"
"github.com/jinzhu/copier"
"github.com/patrickmn/go-cache"
)
// NOTE: To add a new cached item, make sure you know how/when to invalidate it
// and how long it can safely be cached. Consider the case where it is read to
// be updated - those cases need to bypass the cache and read directly from the
// DB to always use fresh data (see the ctxdb.BypassCachedMysql method). Follow
// all of these steps:
//
// 1. Add a unique key name and a default expiration duration, which will be
// used in production.
// 2. Define an expiration duration field in the cachedMysql struct,
// initialize it with the default expiration duration in New, and add a
// WithXXXExpiration option to customize it.
// 3. Implement the cloner interface for the type of the cached item. If the
// type is a slice, you will need to define a type for the slice (see
// fleet.ScheduledQueryList for an example, or packsList in this package
// for an alternative approach).
// 4. Add the cached item to fleet/tools/cloner-check/main.go (in the
// cacheableItems slice variable) to ensure it gets properly checked in CI
// when fields are added/modified. Run the tool to update the generated files
// once you're confident that the Clone implementation covers all fields that
// need special care (usually pointers, slices, maps).
// 5. Add the required Datastore methods to get the cached item and to set it,
// and add tests in cached_mysql_test.go to ensure it works as expected.
const (
appConfigKey = "AppConfig:%s"
defaultAppConfigExpiration = 1 * time.Second
@ -32,60 +52,12 @@ const (
defaultQueryResultsCountExpiration = 1 * time.Second
)
// cloner represents any type that can clone itself. Used by types to provide a more efficient clone method.
type cloner interface {
Clone() (interface{}, error)
}
func clone(v interface{}) (interface{}, error) {
if cloner, ok := v.(cloner); ok {
return cloner.Clone()
}
// TODO(mna): consider making implementation of the cloner interface
// mandatory, and panic/fail loudly if not implemented. Reflection-based deep
// cloning has significant performance issues at scale (better yet - make the
// cache accept/return cloner types instead of interface{}).
if v == nil {
return nil, nil
}
// Use reflection to initialize a clone of v of the same type.
vv := reflect.ValueOf(v)
// If the value is a pointer, then calling reflect.New on it will result in a double pointer.
// Instead, dereference the pointer first.
isPtr := false
if vv.Kind() == reflect.Ptr {
isPtr = true
if vv.IsNil() {
return nil, nil
}
vv = vv.Elem()
}
clone := reflect.New(vv.Type())
err := copier.CopyWithOption(clone.Interface(), v, copier.Option{DeepCopy: true, IgnoreEmpty: true})
if err != nil {
return nil, err
}
if isPtr {
return clone.Interface(), nil
}
// The value was not a pointer. Need to dereference it before returning.
return clone.Elem().Interface(), nil
}
// cloneCache wraps the in memory cache with one that clones items before returning them.
type cloneCache struct {
*cache.Cache
}
func (c *cloneCache) Get(ctx context.Context, k string) (interface{}, bool) {
func (c *cloneCache) Get(ctx context.Context, k string) (fleet.Cloner, bool) {
if ctxdb.IsCachedMysqlBypassed(ctx) {
// cache miss if the caller explicitly asked to bypass the cache
return nil, false
@ -95,8 +67,13 @@ func (c *cloneCache) Get(ctx context.Context, k string) (interface{}, bool) {
if !found {
return nil, false
}
xc, ok := x.(fleet.Cloner)
if !ok {
// should never happen, cached item is not a cloner
return nil, false
}
clone, err := clone(x)
clone, err := xc.Clone()
if err != nil {
// Unfortunely, we can't return an error here. Return a cache miss instead of panic'ing.
return nil, false
@ -104,8 +81,8 @@ func (c *cloneCache) Get(ctx context.Context, k string) (interface{}, bool) {
return clone, true
}
func (c *cloneCache) Set(ctx context.Context, k string, x interface{}, d time.Duration) {
clone, err := clone(x)
func (c *cloneCache) Set(ctx context.Context, k string, x fleet.Cloner, d time.Duration) {
clone, err := x.Clone()
if err != nil {
// Unfortunately, we can't return an error here. Skip caching it if clone
// fails, but ensure that we clear any existing cached item for this key,
@ -170,6 +147,18 @@ func WithTeamMDMConfigExpiration(d time.Duration) Option {
}
}
func WithQueryByNameExpiration(d time.Duration) Option {
return func(o *cachedMysql) {
o.queryByNameExp = d
}
}
func WithQueryResultsCountExpiration(d time.Duration) Option {
return func(o *cachedMysql) {
o.queryResultsCountExp = d
}
}
func New(ds fleet.Datastore, opts ...Option) fleet.Datastore {
c := &cachedMysql{
Datastore: ds,
@ -232,7 +221,7 @@ func (ds *cachedMysql) SaveAppConfig(ctx context.Context, info *fleet.AppConfig)
func (ds *cachedMysql) ListPacksForHost(ctx context.Context, hid uint) ([]*fleet.Pack, error) {
key := fmt.Sprintf(packsHostKey, hid)
if x, found := ds.c.Get(ctx, key); found {
cachedPacks, ok := x.([]*fleet.Pack)
cachedPacks, ok := x.(packsList)
if ok {
return cachedPacks, nil
}
@ -243,7 +232,7 @@ func (ds *cachedMysql) ListPacksForHost(ctx context.Context, hid uint) ([]*fleet
return nil, err
}
ds.c.Set(ctx, key, packs, ds.packsExp)
ds.c.Set(ctx, key, packsList(packs), ds.packsExp)
return packs, nil
}
@ -270,8 +259,8 @@ func (ds *cachedMysql) ListScheduledQueriesInPack(ctx context.Context, packID ui
func (ds *cachedMysql) TeamAgentOptions(ctx context.Context, teamID uint) (*json.RawMessage, error) {
key := fmt.Sprintf(teamAgentOptionsKey, teamID)
if x, found := ds.c.Get(ctx, key); found {
if agentOptions, ok := x.(*json.RawMessage); ok {
return agentOptions, nil
if agentOptions, ok := x.(*rawJSONMessage); ok {
return (*json.RawMessage)(agentOptions), nil
}
}
@ -280,7 +269,7 @@ func (ds *cachedMysql) TeamAgentOptions(ctx context.Context, teamID uint) (*json
return nil, err
}
ds.c.Set(ctx, key, agentOptions, ds.teamAgentOptionsExp)
ds.c.Set(ctx, key, (*rawJSONMessage)(agentOptions), ds.teamAgentOptionsExp)
return agentOptions, nil
}
@ -331,7 +320,7 @@ func (ds *cachedMysql) SaveTeam(ctx context.Context, team *fleet.Team) (*fleet.T
featuresKey := fmt.Sprintf(teamFeaturesKey, team.ID)
mdmConfigKey := fmt.Sprintf(teamMDMConfigKey, team.ID)
ds.c.Set(ctx, agentOptionsKey, team.Config.AgentOptions, ds.teamAgentOptionsExp)
ds.c.Set(ctx, agentOptionsKey, (*rawJSONMessage)(team.Config.AgentOptions), ds.teamAgentOptionsExp)
ds.c.Set(ctx, featuresKey, &team.Config.Features, ds.teamFeaturesExp)
ds.c.Set(ctx, mdmConfigKey, &team.Config.MDM, ds.teamMDMConfigExp)
@ -355,6 +344,7 @@ func (ds *cachedMysql) DeleteTeam(ctx context.Context, teamID uint) error {
return nil
}
// TODO: should we handle DeleteQuery/DeleteQueries/SaveQuery to invalidate that cache?
func (ds *cachedMysql) QueryByName(ctx context.Context, teamID *uint, name string) (*fleet.Query, error) {
teamID_ := uint(0) // global team is 0
if teamID != nil {
@ -382,8 +372,8 @@ func (ds *cachedMysql) ResultCountForQuery(ctx context.Context, queryID uint) (i
key := fmt.Sprintf(queryResultsCountKey, queryID)
if x, found := ds.c.Get(ctx, key); found {
if count, ok := x.(int); ok {
return count, nil
if count, ok := x.(integer); ok {
return int(count), nil
}
}
@ -392,7 +382,7 @@ func (ds *cachedMysql) ResultCountForQuery(ctx context.Context, queryID uint) (i
return 0, err
}
ds.c.Set(ctx, key, count, ds.queryResultsCountExp)
ds.c.Set(ctx, key, integer(count), ds.queryResultsCountExp)
return count, nil
}

230
server/datastore/cached_mysql/cached_mysql_test.go
View file

@ -4,7 +4,6 @@ import (
"context"
"encoding/json"
"errors"
"reflect"
"testing"
"time"
@ -17,34 +16,21 @@ import (
"github.com/stretchr/testify/require"
)
func TestClone(t *testing.T) {
var nilRawMessage *json.RawMessage
type nilCloner struct{}
func (n *nilCloner) Clone() (fleet.Cloner, error) {
var nn *nilCloner
return nn, nil
}
func TestClone(t *testing.T) {
tests := []struct {
name string
src interface{}
want interface{}
src fleet.Cloner
want fleet.Cloner
}{
{
name: "string",
src: "foo",
want: "foo",
},
{
name: "struct",
src: fleet.AppConfig{
ServerSettings: fleet.ServerSettings{
EnableAnalytics: true,
},
},
want: fleet.AppConfig{
ServerSettings: fleet.ServerSettings{
EnableAnalytics: true,
},
},
},
{
name: "pointer to struct",
name: "appconfig",
src: &fleet.AppConfig{
ServerSettings: fleet.ServerSettings{
EnableAnalytics: true,
@ -56,28 +42,13 @@ func TestClone(t *testing.T) {
},
},
},
{
name: "slice",
src: []string{"foo", "bar"},
want: []string{"foo", "bar"},
},
{
name: "pointer to slice",
src: &[]string{"foo", "bar"},
want: &[]string{"foo", "bar"},
},
{
name: "nil",
src: nil,
want: nil,
src: (*nilCloner)(nil),
want: (*nilCloner)(nil),
},
{
name: "nil pointer",
src: nilRawMessage,
want: nil,
},
{
name: "pointer to struct with nested slice",
name: "appconfig with nested slice",
src: &fleet.AppConfig{
ServerSettings: fleet.ServerSettings{
DebugHostIDs: []uint{1, 2, 3},
@ -93,33 +64,13 @@ func TestClone(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
clone, err := clone(tc.src)
clone, err := tc.src.Clone()
require.NoError(t, err)
assert.Equal(t, tc.want, clone)
v1, v2 := reflect.ValueOf(tc.src), reflect.ValueOf(clone)
if k := v1.Kind(); k == reflect.Pointer || k == reflect.Slice || k == reflect.Map || k == reflect.Chan || k == reflect.Func || k == reflect.UnsafePointer {
if clone == nil {
assert.True(t, v1.IsNil())
return
}
require.Equal(t, v1.Kind(), v2.Kind())
assert.NotEqual(t, v1.Pointer(), v2.Pointer())
}
// ensure that writing to src does not alter the cloned value (i.e. that
// the nested fields are deeply cloned too).
switch src := tc.src.(type) {
case []string:
if len(src) > 0 {
src[0] = "modified"
assert.NotEqual(t, src, clone)
}
case *[]string:
if len(*src) > 0 {
(*src)[0] = "modified"
assert.NotEqual(t, src, clone)
}
case *fleet.AppConfig:
if len(src.ServerSettings.DebugHostIDs) > 0 {
src.ServerSettings.DebugHostIDs[0] = 999
@ -303,9 +254,12 @@ func TestCachedPacksforHost(t *testing.T) {
return dbPacks, nil
}
// first call gets the result from the DB
packs, err := ds.ListPacksForHost(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, dbPacks, packs)
require.Same(t, dbPacks[0], packs[0])
require.Equal(t, 1, called)
// change "stored" dbPacks.
dbPacks = []*fleet.Pack{
@ -319,16 +273,20 @@ func TestCachedPacksforHost(t *testing.T) {
},
}
// this call gets it from the cache
packs2, err := ds.ListPacksForHost(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, packs, packs2) // returns the old cached value
require.Equal(t, packs, packs2) // returns the old cached value
require.NotSame(t, packs[0], packs2[0]) // have been cloned
require.Equal(t, 1, called)
time.Sleep(200 * time.Millisecond)
// this call gets it from the DB again since the cache expired
packs3, err := ds.ListPacksForHost(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, dbPacks, packs3) // returns the old cached value
require.Equal(t, dbPacks, packs3)
require.Same(t, dbPacks[0], packs3[0])
require.Equal(t, 2, called)
}
@ -354,9 +312,12 @@ func TestCachedListScheduledQueriesInPack(t *testing.T) {
return dbScheduledQueries, nil
}
// this initial call gets the result from the DB
scheduledQueries, err := ds.ListScheduledQueriesInPack(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, dbScheduledQueries, scheduledQueries)
require.Same(t, dbScheduledQueries[0], scheduledQueries[0])
require.Equal(t, 1, called)
// change "stored" dbScheduledQueries.
dbScheduledQueries = fleet.ScheduledQueryList{
@ -366,16 +327,20 @@ func TestCachedListScheduledQueriesInPack(t *testing.T) {
},
}
// this call gets it from the cache
scheduledQueries2, err := ds.ListScheduledQueriesInPack(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, scheduledQueries2, scheduledQueries) // returns the new db entry
require.Equal(t, scheduledQueries2, scheduledQueries)
require.NotSame(t, scheduledQueries[0], scheduledQueries2[0]) // has been cloned
require.Equal(t, 1, called)
time.Sleep(200 * time.Millisecond)
// this call gets it from the DB again, since the cache expired
scheduledQueries3, err := ds.ListScheduledQueriesInPack(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, dbScheduledQueries, scheduledQueries3) // returns the new db entry
require.Equal(t, dbScheduledQueries, scheduledQueries3)
require.Same(t, dbScheduledQueries[0], scheduledQueries3[0])
require.Equal(t, 2, called)
}
@ -433,9 +398,20 @@ func TestCachedTeamAgentOptions(t *testing.T) {
return nil
}
// initial call reads from the DB
options, err := ds.TeamAgentOptions(context.Background(), 1)
require.NoError(t, err)
require.JSONEq(t, string(testOptions), string(*options))
require.Same(t, &testOptions, options)
require.True(t, mockedDS.TeamAgentOptionsFuncInvoked)
mockedDS.TeamAgentOptionsFuncInvoked = false
// subsequent call reads from the cache
options, err = ds.TeamAgentOptions(context.Background(), 1)
require.NoError(t, err)
require.JSONEq(t, string(testOptions), string(*options))
require.NotSame(t, &testOptions, options)
require.False(t, mockedDS.TeamAgentOptionsFuncInvoked)
// saving a team updates agent options in cache
updateOptions := json.RawMessage(`
@ -452,17 +428,26 @@ func TestCachedTeamAgentOptions(t *testing.T) {
_, err = ds.SaveTeam(context.Background(), updateTeam)
require.NoError(t, err)
require.True(t, mockedDS.SaveTeamFuncInvoked)
mockedDS.SaveTeamFuncInvoked = false
// reading reads it from the cache with the updated data
options, err = ds.TeamAgentOptions(context.Background(), testTeam.ID)
require.NoError(t, err)
require.JSONEq(t, string(updateOptions), string(*options))
require.NotSame(t, &updateOptions, options)
require.False(t, mockedDS.TeamAgentOptionsFuncInvoked)
// deleting a team removes the agent options from the cache
err = ds.DeleteTeam(context.Background(), testTeam.ID)
require.NoError(t, err)
require.True(t, mockedDS.DeleteTeamFuncInvoked)
mockedDS.DeleteTeamFuncInvoked = false
// reading hits the DB as the cached item was removed
_, err = ds.TeamAgentOptions(context.Background(), testTeam.ID)
require.Error(t, err)
require.True(t, mockedDS.TeamAgentOptionsFuncInvoked)
}
func TestCachedTeamFeatures(t *testing.T) {
@ -470,8 +455,8 @@ func TestCachedTeamFeatures(t *testing.T) {
mockedDS := new(mock.Store)
ds := New(mockedDS, WithTeamFeaturesExpiration(100*time.Millisecond))
ao := json.RawMessage(`{}`)
ao := json.RawMessage(`{}`)
aq := json.RawMessage(`{"foo": "bar"}`)
testFeatures := fleet.Features{
EnableHostUsers: false,
@ -509,6 +494,7 @@ func TestCachedTeamFeatures(t *testing.T) {
features, err := ds.TeamFeatures(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, testFeatures, *features)
require.Same(t, &testFeatures, features)
require.True(t, mockedDS.TeamFeaturesFuncInvoked)
mockedDS.TeamFeaturesFuncInvoked = false
@ -516,8 +502,15 @@ func TestCachedTeamFeatures(t *testing.T) {
features, err = ds.TeamFeatures(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, testFeatures, *features)
require.NotSame(t, &testFeatures, features)
require.False(t, mockedDS.TeamFeaturesFuncInvoked)
// changing e.g. the DetailQueryOverrides map doesn't affect the stored value
ptrA := features.DetailQueryOverrides["a"]
*ptrA = "AAA"
features.DetailQueryOverrides["c"] = ptr.String("C")
require.NotEqual(t, testFeatures.DetailQueryOverrides, features.DetailQueryOverrides)
// saving a team updates features in cache
aq = json.RawMessage(`{"bar": "baz"}`)
updateFeatures := fleet.Features{
@ -539,16 +532,20 @@ func TestCachedTeamFeatures(t *testing.T) {
_, err = ds.SaveTeam(context.Background(), updateTeam)
require.NoError(t, err)
require.True(t, mockedDS.SaveTeamFuncInvoked)
mockedDS.SaveTeamFuncInvoked = false
// this call gets it from the cache and gets the updated value set by SaveTeam
features, err = ds.TeamFeatures(context.Background(), testTeam.ID)
require.NoError(t, err)
require.Equal(t, updateFeatures, *features)
require.NotSame(t, &updateFeatures, features)
require.False(t, mockedDS.TeamFeaturesFuncInvoked)
// deleting a team removes the features from the cache
err = ds.DeleteTeam(context.Background(), testTeam.ID)
require.NoError(t, err)
// reading hits the DB as the cached item was removed
_, err = ds.TeamFeatures(context.Background(), testTeam.ID)
require.Error(t, err)
require.True(t, mockedDS.TeamFeaturesFuncInvoked)
@ -605,6 +602,7 @@ func TestCachedTeamMDMConfig(t *testing.T) {
mdmConfig, err := ds.TeamMDMConfig(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, testMDMConfig, *mdmConfig)
require.Same(t, &testMDMConfig, mdmConfig)
require.True(t, mockedDS.TeamMDMConfigFuncInvoked)
mockedDS.TeamMDMConfigFuncInvoked = false
@ -612,8 +610,13 @@ func TestCachedTeamMDMConfig(t *testing.T) {
mdmConfig, err = ds.TeamMDMConfig(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, testMDMConfig, *mdmConfig)
require.NotSame(t, &testMDMConfig, mdmConfig)
require.False(t, mockedDS.TeamMDMConfigFuncInvoked)
// changing some deep value doesn't affect the stored value
mdmConfig.MacOSSettings.CustomSettings[0] = "c"
require.NotEqual(t, testMDMConfig, *mdmConfig)
// saving a team updates config in cache
updateMDMConfig := fleet.TeamMDM{
MacOSUpdates: fleet.MacOSUpdates{
@ -638,17 +641,108 @@ func TestCachedTeamMDMConfig(t *testing.T) {
_, err = ds.SaveTeam(context.Background(), updateTeam)
require.NoError(t, err)
require.True(t, mockedDS.SaveTeamFuncInvoked)
mockedDS.SaveTeamFuncInvoked = false
// this call gets it from the cache, with the updated value set by SaveTeam
mdmConfig, err = ds.TeamMDMConfig(context.Background(), testTeam.ID)
require.NoError(t, err)
require.Equal(t, updateMDMConfig, *mdmConfig)
require.NotSame(t, &updateMDMConfig, mdmConfig)
require.False(t, mockedDS.TeamMDMConfigFuncInvoked)
// deleting a team removes the config from the cache
err = ds.DeleteTeam(context.Background(), testTeam.ID)
require.NoError(t, err)
// reading hits the DB as the cached item was removed
_, err = ds.TeamMDMConfig(context.Background(), testTeam.ID)
require.Error(t, err)
require.True(t, mockedDS.TeamMDMConfigFuncInvoked)
}
func TestCachedQueryByName(t *testing.T) {
t.Parallel()
mockedDS := new(mock.Store)
ds := New(mockedDS, WithQueryByNameExpiration(100*time.Millisecond))
testQuery := &fleet.Query{
ID: 1,
TeamID: ptr.Uint(1),
Packs: []fleet.Pack{{ID: 2, Name: "a"}},
}
mockedDS.QueryByNameFunc = func(ctx context.Context, teamID *uint, name string) (*fleet.Query, error) {
return testQuery, nil
}
// first call gets the result from the DB
query1, err := ds.QueryByName(context.Background(), nil, "q")
require.NoError(t, err)
require.Equal(t, testQuery, query1)
require.Same(t, testQuery, query1)
require.True(t, mockedDS.QueryByNameFuncInvoked)
mockedDS.QueryByNameFuncInvoked = false
// change "stored" query.
testQuery = &fleet.Query{
ID: 1,
TeamID: ptr.Uint(2),
Packs: []fleet.Pack{{ID: 3, Name: "b"}},
}
// this call gets it from the cache
query2, err := ds.QueryByName(context.Background(), nil, "q")
require.NoError(t, err)
require.Equal(t, query1, query2) // returns the old cached value
require.NotSame(t, query1, query2) // have been cloned
require.False(t, mockedDS.QueryByNameFuncInvoked)
// a deep change doesn't alter the stored value
query2.Packs[0].Name = "Z"
require.NotEqual(t, query1, query2)
time.Sleep(200 * time.Millisecond)
// this call gets it from the DB again since the cache expired
query3, err := ds.QueryByName(context.Background(), nil, "q")
require.NoError(t, err)
require.Equal(t, testQuery, query3)
require.Same(t, testQuery, query3)
require.True(t, mockedDS.QueryByNameFuncInvoked)
}
func TestCachedResultCountForQuery(t *testing.T) {
t.Parallel()
mockedDS := new(mock.Store)
ds := New(mockedDS, WithQueryResultsCountExpiration(100*time.Millisecond))
testCount := 1
mockedDS.ResultCountForQueryFunc = func(ctx context.Context, queryID uint) (int, error) {
return testCount, nil
}
// first call gets the result from the DB
c1, err := ds.ResultCountForQuery(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, testCount, c1)
require.True(t, mockedDS.ResultCountForQueryFuncInvoked)
mockedDS.ResultCountForQueryFuncInvoked = false
// change "stored" count.
testCount = 2
// this call gets it from the cache
c2, err := ds.ResultCountForQuery(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, c1, c2) // returns the old cached value
require.False(t, mockedDS.ResultCountForQueryFuncInvoked)
time.Sleep(200 * time.Millisecond)
// this call gets it from the DB again since the cache expired
c3, err := ds.ResultCountForQuery(context.Background(), 1)
require.NoError(t, err)
require.Equal(t, testCount, c3)
require.True(t, mockedDS.ResultCountForQueryFuncInvoked)
}

41
server/datastore/cached_mysql/cloners.go Normal file
View file

@ -0,0 +1,41 @@
package cached_mysql
import (
"encoding/json"
"github.com/fleetdm/fleet/v4/server/fleet"
)
type packsList []*fleet.Pack
func (pl packsList) Clone() (fleet.Cloner, error) {
var cloned packsList
if pl == nil {
return cloned, nil
}
cloned = make(packsList, 0, len(pl))
for _, p := range pl {
cloned = append(cloned, p.Copy())
}
return cloned, nil
}
type rawJSONMessage json.RawMessage
func (r *rawJSONMessage) Clone() (fleet.Cloner, error) {
var clone *rawJSONMessage
if r == nil {
return clone, nil
}
msg := make(rawJSONMessage, len(*r))
copy(msg, *r)
return &msg, nil
}
type integer int
func (i integer) Clone() (fleet.Cloner, error) {
return i, nil
}

14
server/datastore/mysql/apple_mdm_test.go
View file

@ -640,7 +640,7 @@ func testIngestMDMAppleHostAlreadyExistsInFleet(t *testing.T, ds *Datastore) {
Platform: "darwin",
})
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(ctx, host.ID, false, false, "https://fleetdm.com", true, fleet.WellKnownMDMFleet)
err = ds.SetOrUpdateMDMData(ctx, host.ID, false, false, "https://fleetdm.com", true, fleet.WellKnownMDMFleet, "")
require.NoError(t, err)
hosts := listHostsCheckCount(t, ds, fleet.TeamFilter{User: test.UserAdmin}, fleet.HostListOptions{}, 1)
require.Equal(t, testSerial, hosts[0].HardwareSerial)
@ -678,7 +678,7 @@ func testIngestMDMNonDarwinHostAlreadyExistsInFleet(t *testing.T, ds *Datastore)
Platform: "linux",
})
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(ctx, host.ID, false, false, "https://fleetdm.com", true, "Fleet MDM")
err = ds.SetOrUpdateMDMData(ctx, host.ID, false, false, "https://fleetdm.com", true, "Fleet MDM", "")
require.NoError(t, err)
hosts := listHostsCheckCount(t, ds, fleet.TeamFilter{User: test.UserAdmin}, fleet.HostListOptions{}, 1)
require.Equal(t, testSerial, hosts[0].HardwareSerial)
@ -3912,7 +3912,7 @@ func TestHostDEPAssignments(t *testing.T) {
require.True(t, *h.DEPAssignedToFleet)
// simulate osquery report of MDM detail query
err = ds.SetOrUpdateMDMData(ctx, testHost.ID, false, true, expectedMDMServerURL, true, fleet.WellKnownMDMFleet)
err = ds.SetOrUpdateMDMData(ctx, testHost.ID, false, true, expectedMDMServerURL, true, fleet.WellKnownMDMFleet, "")
require.NoError(t, err)
// enrollment status changes to "On (automatic)"
@ -3955,7 +3955,7 @@ func TestHostDEPAssignments(t *testing.T) {
require.True(t, *h.DEPAssignedToFleet)
// simulate osquery report of MDM detail query reflecting re-enrollment to MDM
err = ds.SetOrUpdateMDMData(ctx, testHost.ID, false, true, expectedMDMServerURL, true, fleet.WellKnownMDMFleet)
err = ds.SetOrUpdateMDMData(ctx, testHost.ID, false, true, expectedMDMServerURL, true, fleet.WellKnownMDMFleet, "")
require.NoError(t, err)
// host MDM row is re-created when osquery reports MDM detail query
@ -3977,7 +3977,7 @@ func TestHostDEPAssignments(t *testing.T) {
// simulate osquery report of MDM detail query with empty server URL (signals unenrollment
// from MDM)
err = ds.SetOrUpdateMDMData(ctx, testHost.ID, false, false, "", false, "")
err = ds.SetOrUpdateMDMData(ctx, testHost.ID, false, false, "", false, "", "")
require.NoError(t, err)
// host MDM row is reset to defaults when osquery reports MDM detail query with empty server URL
@ -4192,7 +4192,7 @@ func testResetMDMAppleEnrollment(t *testing.T, ds *Datastore) {
ON DUPLICATE KEY UPDATE added_at = CURRENT_TIMESTAMP, deleted_at = NULL
`, host.ID)
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(context.Background(), host.ID, false, true, "foo.mdm.example.com", true, "")
err = ds.SetOrUpdateMDMData(context.Background(), host.ID, false, true, "foo.mdm.example.com", true, "", "")
require.NoError(t, err)
sum, err := ds.GetMDMAppleBootstrapPackageSummary(ctx, uint(0))
@ -4752,7 +4752,7 @@ func TestRestorePendingDEPHost(t *testing.T) {
require.Equal(t, depHostID, h.ID)
// simulate osquery report of MDM detail query
err = ds.SetOrUpdateMDMData(ctx, depHostID, false, true, expectedMDMServerURL, true, fleet.WellKnownMDMFleet)
err = ds.SetOrUpdateMDMData(ctx, depHostID, false, true, expectedMDMServerURL, true, fleet.WellKnownMDMFleet, "")
require.NoError(t, err)
// enrollment status changes to "On (automatic)"

104
server/datastore/mysql/hosts.go
View file

@ -943,9 +943,20 @@ func (ds *Datastore) applyHostFilters(
}
softwareFilter := "TRUE"
if opt.SoftwareIDFilter != nil {
var softwareIDFilter *uint
if opt.SoftwareVersionIDFilter != nil {
softwareIDFilter = opt.SoftwareVersionIDFilter
} else if opt.SoftwareIDFilter != nil {
softwareIDFilter = opt.SoftwareIDFilter
}
if softwareIDFilter != nil {
softwareFilter = "EXISTS (SELECT 1 FROM host_software hs WHERE hs.host_id = h.id AND hs.software_id = ?)"
params = append(params, opt.SoftwareIDFilter)
params = append(params, *softwareIDFilter)
} else if opt.SoftwareTitleIDFilter != nil {
// software (version) ID filter is mutually exclusive with software title ID
// so we're reusing the same filter to avoid adding unnecessary conditions.
softwareFilter = "EXISTS (SELECT 1 FROM host_software hs INNER JOIN software sw ON hs.software_id = sw.id WHERE hs.host_id = h.id AND sw.title_id = ?)"
params = append(params, *opt.SoftwareTitleIDFilter)
}
failingPoliciesJoin := ""
@ -1268,7 +1279,7 @@ func (ds *Datastore) filterHostsByOSSettingsStatus(sql string, opt fleet.HostLis
WHEN (%s) THEN
'bitlocker_pending'
WHEN (%s) THEN
'bitlocker_failed'
'bitlocker_failed'
ELSE
''
END`,
@ -1280,11 +1291,11 @@ func (ds *Datastore) filterHostsByOSSettingsStatus(sql string, opt fleet.HostLis
}
whereWindows += fmt.Sprintf(` AND (
CASE (%s)
CASE (%s)
WHEN 'profiles_failed' THEN
'failed'
WHEN 'profiles_pending' THEN (
CASE (%s)
CASE (%s)
WHEN 'bitlocker_failed' THEN
'failed'
ELSE
@ -1310,7 +1321,7 @@ func (ds *Datastore) filterHostsByOSSettingsStatus(sql string, opt fleet.HostLis
ELSE
'verified'
END)
ELSE
ELSE
REPLACE((%s), 'bitlocker_', '')
END) = ?`, profilesStatus, bitlockerStatus, bitlockerStatus, bitlockerStatus, bitlockerStatus)
@ -2828,11 +2839,14 @@ func (ds *Datastore) ListHostDeviceMapping(ctx context.Context, id uint) ([]*fle
return mappings, nil
}
func (ds *Datastore) ReplaceHostDeviceMapping(ctx context.Context, hid uint, mappings []*fleet.HostDeviceMapping) error {
func (ds *Datastore) ReplaceHostDeviceMapping(ctx context.Context, hid uint, mappings []*fleet.HostDeviceMapping, source string) error {
for _, m := range mappings {
if hid != m.HostID {
return ctxerr.Errorf(ctx, "host device mapping are not all for the provided host id %d, found %d", hid, m.HostID)
}
if m.Source != source {
return ctxerr.Errorf(ctx, "host device mapping are not all for the provided source %s, found %s", source, m.Source)
}
}
// the following SQL statements assume a small number of emails reported
@ -2846,7 +2860,7 @@ func (ds *Datastore) ReplaceHostDeviceMapping(ctx context.Context, hid uint, map
FROM
host_emails
WHERE
host_id = ?`
host_id = ? AND source = ?`
delStmt = `
DELETE FROM
@ -2871,7 +2885,7 @@ func (ds *Datastore) ReplaceHostDeviceMapping(ctx context.Context, hid uint, map
}
var prevMappings []*fleet.HostDeviceMapping
if err := sqlx.SelectContext(ctx, tx, &prevMappings, selStmt, hid); err != nil {
if err := sqlx.SelectContext(ctx, tx, &prevMappings, selStmt, hid, source); err != nil {
return ctxerr.Wrap(ctx, err, "select previous host emails")
}
@ -3276,6 +3290,7 @@ func (ds *Datastore) SetOrUpdateMDMData(
serverURL string,
installedFromDep bool,
name string,
fleetEnrollmentRef string,
) error {
var mdmID *uint
if serverURL != "" {
@ -3288,9 +3303,33 @@ func (ds *Datastore) SetOrUpdateMDMData(
return ds.updateOrInsert(
ctx,
`UPDATE host_mdm SET enrolled = ?, server_url = ?, installed_from_dep = ?, mdm_id = ?, is_server = ? WHERE host_id = ?`,
`INSERT INTO host_mdm (enrolled, server_url, installed_from_dep, mdm_id, is_server, host_id) VALUES (?, ?, ?, ?, ?, ?)`,
enrolled, serverURL, installedFromDep, mdmID, isServer, hostID,
`UPDATE host_mdm SET enrolled = ?, server_url = ?, installed_from_dep = ?, mdm_id = ?, is_server = ?, fleet_enroll_ref = ? WHERE host_id = ?`,
`INSERT INTO host_mdm (enrolled, server_url, installed_from_dep, mdm_id, is_server, fleet_enroll_ref, host_id) VALUES (?, ?, ?, ?, ?, ?, ?)`,
enrolled, serverURL, installedFromDep, mdmID, isServer, fleetEnrollmentRef, hostID,
)
}
const hostEmailsSourceMdmIdpAccounts = "mdm_idp_accounts"
func (ds *Datastore) SetOrUpdateHostEmailsFromMdmIdpAccounts(
ctx context.Context,
hostID uint,
fleetEnrollmentRef string,
) error {
var email *string
if fleetEnrollmentRef != "" {
idp, err := ds.GetMDMIdPAccountByUUID(ctx, fleetEnrollmentRef)
if err != nil {
return err
}
email = &idp.Email
}
return ds.updateOrInsert(
ctx,
`UPDATE host_emails SET email = ? WHERE host_id = ? AND source = ?`,
`INSERT INTO host_emails (email, host_id, source) VALUES (?, ?, ?)`,
email, hostID, hostEmailsSourceMdmIdpAccounts,
)
}
@ -4536,3 +4575,44 @@ func (ds *Datastore) GetMatchingHostSerials(ctx context.Context, serials []strin
return result, nil
}
func (ds *Datastore) GetHostHealth(ctx context.Context, id uint) (*fleet.HostHealth, error) {
sqlStmt := `
SELECT h.os_version, h.updated_at, h.platform, h.team_id, hd.encrypted as disk_encryption_enabled FROM hosts h
LEFT JOIN host_disks hd ON hd.host_id = h.id
WHERE id = ?
`
var hh fleet.HostHealth
if err := sqlx.GetContext(ctx, ds.reader(ctx), &hh, sqlStmt, id); err != nil {
if err == sql.ErrNoRows {
return nil, ctxerr.Wrap(ctx, notFound("Host Health").WithID(id))
}
return nil, ctxerr.Wrap(ctx, err, "loading host health")
}
host := &fleet.Host{ID: id, Platform: hh.Platform}
if err := ds.LoadHostSoftware(ctx, host, true); err != nil {
return nil, err
}
for _, s := range host.Software {
if len(s.Vulnerabilities) > 0 {
hh.VulnerableSoftware = append(hh.VulnerableSoftware, s)
}
}
policies, err := ds.ListPoliciesForHost(ctx, host)
if err != nil {
return nil, err
}
for _, p := range policies {
if p.Response == "fail" {
hh.FailingPolicies = append(hh.FailingPolicies, p)
}
}
return &hh, nil
}

312
server/datastore/mysql/hosts_test.go
View file

@ -157,6 +157,7 @@ func TestHosts(t *testing.T) {
{"ListHostsLiteByIDs", testHostsListHostsLiteByIDs},
{"ListHostsWithPagination", testListHostsWithPagination},
{"LastRestarted", testLastRestarted},
{"HostHealth", testHostHealth},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@ -932,13 +933,13 @@ func testHostsListQuery(t *testing.T, ds *Datastore) {
require.NoError(t, ds.ReplaceHostDeviceMapping(context.Background(), hosts[0].ID, []*fleet.HostDeviceMapping{
{HostID: hosts[0].ID, Email: "a@b.c", Source: "src1"},
{HostID: hosts[0].ID, Email: "b@b.c", Source: "src1"},
}))
}, "src1"))
require.NoError(t, ds.ReplaceHostDeviceMapping(context.Background(), hosts[1].ID, []*fleet.HostDeviceMapping{
{HostID: hosts[1].ID, Email: "c@b.c", Source: "src1"},
}))
}, "src1"))
require.NoError(t, ds.ReplaceHostDeviceMapping(context.Background(), hosts[2].ID, []*fleet.HostDeviceMapping{
{HostID: hosts[2].ID, Email: "dbca@b.cba", Source: "src1"},
}))
}, "src1"))
// add some disks space info for some hosts
require.NoError(t, ds.SetOrUpdateHostDisksSpace(context.Background(), hosts[0].ID, 1.0, 2.0))
@ -1098,9 +1099,9 @@ func testHostsUnenrollFromMDM(t *testing.T, ds *Datastore) {
// Set hosts to be enrolled to an MDM.
const simpleMDM = "https://simplemdm.com"
err = ds.SetOrUpdateMDMData(ctx, h.ID, false, true, simpleMDM, true, "")
err = ds.SetOrUpdateMDMData(ctx, h.ID, false, true, simpleMDM, true, "", "")
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(ctx, h2.ID, false, true, simpleMDM, true, "")
err = ds.SetOrUpdateMDMData(ctx, h2.ID, false, true, simpleMDM, true, "", "")
require.NoError(t, err)
// force is_server to NULL for host 1
@ -1134,7 +1135,7 @@ func testHostsUnenrollFromMDM(t *testing.T, ds *Datastore) {
require.Equal(t, 2, solutions[0].HostsCount)
// Host `h` unenrolls from MDM, so MDM query returns empty server_url.
err = ds.SetOrUpdateMDMData(ctx, h.ID, false, false, "", false, "")
err = ds.SetOrUpdateMDMData(ctx, h.ID, false, false, "", false, "", "")
require.NoError(t, err)
// host_mdm entry should still exist with empty values.
@ -1155,7 +1156,7 @@ func testHostsUnenrollFromMDM(t *testing.T, ds *Datastore) {
require.Equal(t, 1, solutions[0].HostsCount)
// Host `h2` unenrolls from MDM, so MDM query returns empty server_url.
err = ds.SetOrUpdateMDMData(ctx, h2.ID, false, false, "", false, "")
err = ds.SetOrUpdateMDMData(ctx, h2.ID, false, false, "", false, "", "")
require.NoError(t, err)
// host_mdm entry should not exist anymore.
@ -1209,13 +1210,13 @@ func testHostsListMDM(t *testing.T, ds *Datastore) {
require.Nil(t, tmID)
const simpleMDM, kandji, unknown = "https://simplemdm.com", "https://kandji.io", "https://url.com"
err = ds.SetOrUpdateMDMData(ctx, hostIDs[0], false, true, simpleMDM, true, fleet.WellKnownMDMSimpleMDM) // enrollment: automatic
err = ds.SetOrUpdateMDMData(ctx, hostIDs[0], false, true, simpleMDM, true, fleet.WellKnownMDMSimpleMDM, "") // enrollment: automatic
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(ctx, hostIDs[1], false, true, kandji, true, fleet.WellKnownMDMKandji) // enrollment: automatic
err = ds.SetOrUpdateMDMData(ctx, hostIDs[1], false, true, kandji, true, fleet.WellKnownMDMKandji, "") // enrollment: automatic
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(ctx, hostIDs[2], false, true, unknown, false, fleet.UnknownMDMName) // enrollment: manual
err = ds.SetOrUpdateMDMData(ctx, hostIDs[2], false, true, unknown, false, fleet.UnknownMDMName, "") // enrollment: manual
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(ctx, hostIDs[3], false, false, simpleMDM, false, fleet.WellKnownMDMSimpleMDM) // enrollment: unenrolled
err = ds.SetOrUpdateMDMData(ctx, hostIDs[3], false, false, simpleMDM, false, fleet.WellKnownMDMSimpleMDM, "") // enrollment: unenrolled
require.NoError(t, err)
var simpleMDMID uint
@ -1287,9 +1288,9 @@ func testHostsListMDM(t *testing.T, ds *Datastore) {
require.NoError(t, err)
hostIDs = append(hostIDs, h.ID)
}
err = ds.SetOrUpdateMDMData(ctx, hostIDs[10], false, true, "http://intuneexample.com", false, fleet.WellKnownMDMIntune) // enrolled in Intune
err = ds.SetOrUpdateMDMData(ctx, hostIDs[10], false, true, "http://intuneexample.com", false, fleet.WellKnownMDMIntune, "") // enrolled in Intune
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(ctx, hostIDs[11], false, true, "http://example.com", false, fleet.WellKnownMDMFleet) // enrolled in Fleet
err = ds.SetOrUpdateMDMData(ctx, hostIDs[11], false, true, "http://example.com", false, fleet.WellKnownMDMFleet, "") // enrolled in Fleet
require.NoError(t, err)
hosts = listHostsCheckCount(t, ds, filter, fleet.HostListOptions{MDMNameFilter: ptr.String(fleet.WellKnownMDMIntune)}, 1)
@ -1406,7 +1407,7 @@ func testHostMDMSelect(t *testing.T, ds *Datastore) {
require.NoError(t, err)
for _, c := range cases {
require.NoError(t, ds.SetOrUpdateMDMData(ctx, h.ID, c.host.IsServer, c.host.Enrolled, mdmServerURL, c.host.InstalledFromDep, "test"))
require.NoError(t, ds.SetOrUpdateMDMData(ctx, h.ID, c.host.IsServer, c.host.Enrolled, mdmServerURL, c.host.InstalledFromDep, "test", ""))
hosts, err := ds.ListHosts(ctx, fleet.TeamFilter{User: test.UserAdmin}, fleet.HostListOptions{})
require.NoError(t, err)
@ -2766,16 +2767,63 @@ func testHostsListBySoftware(t *testing.T, ds *Datastore) {
}
host1 := hosts[0]
host2 := hosts[1]
host3 := hosts[2]
_, err := ds.UpdateHostSoftware(context.Background(), host1.ID, software)
require.NoError(t, err)
_, err = ds.UpdateHostSoftware(context.Background(), host2.ID, software)
require.NoError(t, err)
// host 3 only has foo v0.0.3
_, err = ds.UpdateHostSoftware(context.Background(), host3.ID, software[1:2])
require.NoError(t, err)
// reconcile software, will sync software titles
err = ds.ReconcileSoftwareTitles(context.Background())
require.NoError(t, err)
var fooV002ID uint
ExecAdhocSQL(t, ds, func(q sqlx.ExtContext) error {
return sqlx.GetContext(context.Background(), q, &fooV002ID,
"SELECT id FROM software WHERE name = ? AND source = ? AND version = ?", "foo", "chrome_extensions", "0.0.2")
})
var fooTitleID uint
ExecAdhocSQL(t, ds, func(q sqlx.ExtContext) error {
return sqlx.GetContext(context.Background(), q, &fooTitleID,
"SELECT id FROM software_titles WHERE name = ? AND source = ?", "foo", "chrome_extensions")
})
require.NoError(t, ds.LoadHostSoftware(context.Background(), host1, false))
require.NoError(t, ds.LoadHostSoftware(context.Background(), host2, false))
hosts = listHostsCheckCount(t, ds, filter, fleet.HostListOptions{SoftwareIDFilter: &host1.Software[0].ID}, 2)
// software_id is foo v0.0.2
hosts = listHostsCheckCount(t, ds, filter, fleet.HostListOptions{SoftwareIDFilter: &fooV002ID}, 2)
require.Len(t, hosts, 2)
got := []uint{hosts[0].ID, hosts[1].ID}
require.ElementsMatch(t, []uint{host1.ID, host2.ID}, got)
// software_version_id is foo v0.0.2 (works exacty the same)
hosts = listHostsCheckCount(t, ds, filter, fleet.HostListOptions{SoftwareVersionIDFilter: &fooV002ID}, 2)
require.Len(t, hosts, 2)
got = []uint{hosts[0].ID, hosts[1].ID}
require.ElementsMatch(t, []uint{host1.ID, host2.ID}, got)
// unknown software_id
hosts = listHostsCheckCount(t, ds, filter, fleet.HostListOptions{SoftwareIDFilter: ptr.Uint(fooV002ID + 100)}, 0)
require.Len(t, hosts, 0)
// unknown software_version_id
hosts = listHostsCheckCount(t, ds, filter, fleet.HostListOptions{SoftwareVersionIDFilter: ptr.Uint(fooV002ID + 100)}, 0)
require.Len(t, hosts, 0)
// software_title_id is foo (any version)
hosts = listHostsCheckCount(t, ds, filter, fleet.HostListOptions{SoftwareTitleIDFilter: &fooTitleID}, 3)
require.Len(t, hosts, 3)
got = []uint{hosts[0].ID, hosts[1].ID, hosts[2].ID}
require.ElementsMatch(t, []uint{host1.ID, host2.ID, host3.ID}, got)
// unknown software_title_id
hosts = listHostsCheckCount(t, ds, filter, fleet.HostListOptions{SoftwareTitleIDFilter: ptr.Uint(fooTitleID + 100)}, 0)
require.Len(t, hosts, 0)
}
func testHostsListBySoftwareChangedAt(t *testing.T, ds *Datastore) {
@ -3550,7 +3598,7 @@ func testHostsSavePackStatsConcurrent(t *testing.T, ds *Datastore) {
PackID: pack1.ID,
AverageMemory: 8000,
Denylisted: false,
Executions: rand.Intn(1000),
Executions: uint64(rand.Intn(1000)),
Interval: 30,
LastExecuted: time.Now().UTC(),
OutputSize: 1337,
@ -3571,7 +3619,7 @@ func testHostsSavePackStatsConcurrent(t *testing.T, ds *Datastore) {
PackID: pack2.ID,
AverageMemory: 8000,
Denylisted: false,
Executions: rand.Intn(1000),
Executions: uint64(rand.Intn(1000)),
Interval: 30,
LastExecuted: time.Now().UTC(),
OutputSize: 1337,
@ -4592,7 +4640,7 @@ func testHostsReplaceHostDeviceMapping(t *testing.T, ds *Datastore) {
})
require.NoError(t, err)
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, nil)
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, nil, "src1")
require.NoError(t, err)
dms, err := ds.ListHostDeviceMapping(ctx, h.ID)
@ -4602,7 +4650,7 @@ func testHostsReplaceHostDeviceMapping(t *testing.T, ds *Datastore) {
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, []*fleet.HostDeviceMapping{
{HostID: h.ID, Email: "a@b.c", Source: "src1"},
{HostID: h.ID + 1, Email: "a@b.c", Source: "src1"},
})
}, "src1")
require.Error(t, err)
require.Contains(t, err.Error(), fmt.Sprintf("found %d", h.ID+1))
@ -4610,7 +4658,18 @@ func testHostsReplaceHostDeviceMapping(t *testing.T, ds *Datastore) {
{HostID: h.ID, Email: "a@b.c", Source: "src1"},
{HostID: h.ID, Email: "b@b.c", Source: "src1"},
{HostID: h.ID, Email: "c@b.c", Source: "src2"},
})
}, "src1")
require.ErrorContains(t, err, "host device mapping are not all for the provided source")
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, []*fleet.HostDeviceMapping{
{HostID: h.ID, Email: "c@b.c", Source: "src2"},
}, "src2")
require.NoError(t, err)
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, []*fleet.HostDeviceMapping{
{HostID: h.ID, Email: "a@b.c", Source: "src1"},
{HostID: h.ID, Email: "b@b.c", Source: "src1"},
}, "src1")
require.NoError(t, err)
dms, err = ds.ListHostDeviceMapping(ctx, h.ID)
@ -4624,7 +4683,19 @@ func testHostsReplaceHostDeviceMapping(t *testing.T, ds *Datastore) {
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, []*fleet.HostDeviceMapping{
{HostID: h.ID, Email: "a@b.c", Source: "src1"},
{HostID: h.ID, Email: "d@b.c", Source: "src2"},
})
}, "src2")
require.ErrorContains(t, err, "host device mapping are not all for the provided source")
// omit b@b.c from src1
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, []*fleet.HostDeviceMapping{
{HostID: h.ID, Email: "a@b.c", Source: "src1"},
}, "src1")
require.NoError(t, err)
// add d@b to src2, omit c@b.c from src2
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, []*fleet.HostDeviceMapping{
{HostID: h.ID, Email: "d@b.c", Source: "src2"},
}, "src2")
require.NoError(t, err)
dms, err = ds.ListHostDeviceMapping(ctx, h.ID)
@ -4635,12 +4706,14 @@ func testHostsReplaceHostDeviceMapping(t *testing.T, ds *Datastore) {
})
// delete only
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, nil)
err = ds.ReplaceHostDeviceMapping(ctx, h.ID, nil, "src1")
require.NoError(t, err)
dms, err = ds.ListHostDeviceMapping(ctx, h.ID)
require.NoError(t, err)
assertHostDeviceMapping(t, dms, nil)
assertHostDeviceMapping(t, dms, []*fleet.HostDeviceMapping{
{Email: "d@b.c", Source: "src2"},
})
}
func assertHostDeviceMapping(t *testing.T, got, want []*fleet.HostDeviceMapping) {
@ -4721,7 +4794,7 @@ func testHostMDMAndMunki(t *testing.T, ds *Datastore) {
_, err = ds.GetHostMDM(context.Background(), 432)
require.True(t, fleet.IsNotFound(err), err)
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 432, false, true, "url", false, ""))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 432, false, true, "url", false, "", ""))
hmdm, err := ds.GetHostMDM(context.Background(), 432)
require.NoError(t, err)
@ -4733,8 +4806,8 @@ func testHostMDMAndMunki(t *testing.T, ds *Datastore) {
urlMDMID := *hmdm.MDMID
assert.Equal(t, fleet.UnknownMDMName, hmdm.Name)
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, true, "https://kandji.io", true, fleet.WellKnownMDMKandji)) // kandji mdm name
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 432, false, false, "url3", true, ""))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, true, "https://kandji.io", true, fleet.WellKnownMDMKandji, "")) // kandji mdm name
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 432, false, false, "url3", true, "", ""))
hmdm, err = ds.GetHostMDM(context.Background(), 432)
require.NoError(t, err)
@ -4768,7 +4841,7 @@ func testHostMDMAndMunki(t *testing.T, ds *Datastore) {
require.ErrorIs(t, err, sql.ErrNoRows)
// switch to simplemdm in an update
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, true, "https://simplemdm.com", false, fleet.WellKnownMDMSimpleMDM)) // now simplemdm name
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, true, "https://simplemdm.com", false, fleet.WellKnownMDMSimpleMDM, "")) // now simplemdm name
hmdm, err = ds.GetHostMDM(context.Background(), 455)
require.NoError(t, err)
@ -4780,7 +4853,7 @@ func testHostMDMAndMunki(t *testing.T, ds *Datastore) {
assert.Equal(t, fleet.WellKnownMDMSimpleMDM, hmdm.Name)
// switch back to "url"
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, false, "url", false, ""))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, false, "url", false, "", ""))
hmdm, err = ds.GetHostMDM(context.Background(), 455)
require.NoError(t, err)
@ -4793,7 +4866,7 @@ func testHostMDMAndMunki(t *testing.T, ds *Datastore) {
// switch to a different Kandji server URL, will have a different MDM ID as
// even though this is another Kandji, the URL is different.
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, true, "https://kandji.io/2", false, fleet.WellKnownMDMKandji))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, true, "https://kandji.io/2", false, fleet.WellKnownMDMKandji, ""))
hmdm, err = ds.GetHostMDM(context.Background(), 455)
require.NoError(t, err)
@ -4979,13 +5052,13 @@ func testAggregatedHostMDMAndMunki(t *testing.T, ds *Datastore) {
},
})
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 432, false, true, "url", false, "")) // manual enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 123, false, true, "url", false, "")) // manual enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 124, false, true, "url", false, "")) // manual enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, true, "https://simplemdm.com", true, fleet.WellKnownMDMSimpleMDM)) // automatic enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 999, false, false, "https://kandji.io", false, fleet.WellKnownMDMKandji)) // unenrolled
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 875, false, false, "https://kandji.io", true, fleet.WellKnownMDMKandji)) // pending enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 1337, false, false, "https://fleetdm.com", true, fleet.WellKnownMDMFleet)) // pending enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 432, false, true, "url", false, "", "")) // manual enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 123, false, true, "url", false, "", "")) // manual enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 124, false, true, "url", false, "", "")) // manual enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 455, false, true, "https://simplemdm.com", true, fleet.WellKnownMDMSimpleMDM, "")) // automatic enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 999, false, false, "https://kandji.io", false, fleet.WellKnownMDMKandji, "")) // unenrolled
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 875, false, false, "https://kandji.io", true, fleet.WellKnownMDMKandji, "")) // pending enrollment
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), 1337, false, false, "https://fleetdm.com", true, fleet.WellKnownMDMFleet, "")) // pending enrollment
require.NoError(t, ds.GenerateAggregatedMunkiAndMDM(context.Background()))
@ -5041,11 +5114,11 @@ func testAggregatedHostMDMAndMunki(t *testing.T, ds *Datastore) {
require.NoError(t, ds.AddHostsToTeam(context.Background(), &team1.ID, []uint{h3.ID}))
require.NoError(t, ds.AddHostsToTeam(context.Background(), &team1.ID, []uint{h4.ID}))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), h1.ID, false, true, "https://simplemdm.com", false, fleet.WellKnownMDMSimpleMDM))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), h2.ID, false, true, "url", false, ""))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), h1.ID, false, true, "https://simplemdm.com", false, fleet.WellKnownMDMSimpleMDM, ""))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), h2.ID, false, true, "url", false, "", ""))
// Add a server, this will be ignored in lists and aggregated data.
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), h4.ID, true, true, "https://simplemdm.com", false, fleet.WellKnownMDMSimpleMDM))
require.NoError(t, ds.SetOrUpdateMDMData(context.Background(), h4.ID, true, true, "https://simplemdm.com", false, fleet.WellKnownMDMSimpleMDM, ""))
require.NoError(t, ds.SetOrUpdateMunkiInfo(context.Background(), h1.ID, "1.2.3", []string{"d"}, nil))
require.NoError(t, ds.SetOrUpdateMunkiInfo(context.Background(), h2.ID, "1.2.3", []string{"d"}, []string{"e"}))
@ -5358,7 +5431,7 @@ func testHostsLoadHostByDeviceAuthToken(t *testing.T, ds *Datastore) {
// create a host enrolled in Simple MDM
hSimple := createHostWithDeviceToken("simple")
err = ds.SetOrUpdateMDMData(ctx, hSimple.ID, false, true, "https://simplemdm.com", true, fleet.WellKnownMDMSimpleMDM)
err = ds.SetOrUpdateMDMData(ctx, hSimple.ID, false, true, "https://simplemdm.com", true, fleet.WellKnownMDMSimpleMDM, "")
require.NoError(t, err)
loadSimple, err := ds.LoadHostByDeviceAuthToken(ctx, "simple", time.Second)
@ -5371,7 +5444,7 @@ func testHostsLoadHostByDeviceAuthToken(t *testing.T, ds *Datastore) {
// create a host that will be pending enrollment in Fleet MDM
hFleet := createHostWithDeviceToken("fleet")
err = ds.SetOrUpdateMDMData(ctx, hFleet.ID, false, false, "https://fleetdm.com", true, fleet.WellKnownMDMFleet)
err = ds.SetOrUpdateMDMData(ctx, hFleet.ID, false, false, "https://fleetdm.com", true, fleet.WellKnownMDMFleet, "")
require.NoError(t, err)
loadFleet, err := ds.LoadHostByDeviceAuthToken(ctx, "fleet", time.Second)
@ -5738,7 +5811,7 @@ func testHostsDeleteHosts(t *testing.T, ds *Datastore) {
// Updates host_emails.
err = ds.ReplaceHostDeviceMapping(context.Background(), host.ID, []*fleet.HostDeviceMapping{
{HostID: host.ID, Email: "a@b.c", Source: "src"},
})
}, "src")
require.NoError(t, err)
// Updates host_additional.
@ -5806,7 +5879,7 @@ func testHostsDeleteHosts(t *testing.T, ds *Datastore) {
require.NoError(t, ds.RecordPolicyQueryExecutions(context.Background(), host, map[uint]*bool{policy.ID: ptr.Bool(true)}, time.Now(), false))
// Update host_mdm.
err = ds.SetOrUpdateMDMData(context.Background(), host.ID, false, true, "foo.mdm.example.com", false, "")
err = ds.SetOrUpdateMDMData(context.Background(), host.ID, false, true, "foo.mdm.example.com", false, "", "")
require.NoError(t, err)
// Update host_munki_info.
err = ds.SetOrUpdateMunkiInfo(context.Background(), host.ID, "42", []string{"a"}, []string{"b"})
@ -6073,9 +6146,16 @@ func testHostsReplaceHostBatteries(t *testing.T, ds *Datastore) {
}
func testHostsReplaceHostBatteriesDeadlock(t *testing.T, ds *Datastore) {
// To increase chance of deadlock increase these numbers.
// We are keeping them low to not cause CI issues ("too many connections" errors
// due to concurrent tests).
const (
hostCount = 10
replaceCount = 10
)
ctx := context.Background()
var hosts []*fleet.Host
for i := 1; i <= 100; i++ {
for i := 1; i <= hostCount; i++ {
h, err := ds.NewHost(ctx, &fleet.Host{
ID: uint(i),
OsqueryHostID: ptr.String(fmt.Sprintf("id-%d", i)),
@ -6095,10 +6175,10 @@ func testHostsReplaceHostBatteriesDeadlock(t *testing.T, ds *Datastore) {
for _, h := range hosts {
hostID := h.ID
g.Go(func() error {
for i := 0; i < 100; i++ {
for i := 0; i < replaceCount; i++ {
if err := ds.ReplaceHostBatteries(ctx, hostID, []*fleet.HostBattery{
{HostID: hostID, SerialNumber: fmt.Sprintf("%d-0000", hostID), CycleCount: 1, Health: "Good"},
{HostID: hostID, SerialNumber: fmt.Sprintf("%d-0000", hostID), CycleCount: 2, Health: "Fair"},
{HostID: hostID, SerialNumber: fmt.Sprintf("%d-0001", hostID), CycleCount: 2, Health: "Fair"},
}); err != nil {
return err
}
@ -6630,7 +6710,7 @@ func testHostsGetHostMDMCheckinInfo(t *testing.T, ds *Datastore) {
TeamID: &tm.ID,
})
require.NoError(t, err)
err = ds.SetOrUpdateMDMData(ctx, host.ID, false, true, "https://fleetdm.com", true, fleet.WellKnownMDMFleet)
err = ds.SetOrUpdateMDMData(ctx, host.ID, false, true, "https://fleetdm.com", true, fleet.WellKnownMDMFleet, "")
require.NoError(t, err)
info, err := ds.GetHostMDMCheckinInfo(ctx, host.UUID)
@ -6714,7 +6794,7 @@ func testHostsLoadHostByOrbitNodeKey(t *testing.T, ds *Datastore) {
// create a host enrolled in Simple MDM
hSimple := createOrbitHost("simple")
err = ds.SetOrUpdateMDMData(ctx, hSimple.ID, false, true, "https://simplemdm.com", true, fleet.WellKnownMDMSimpleMDM)
err = ds.SetOrUpdateMDMData(ctx, hSimple.ID, false, true, "https://simplemdm.com", true, fleet.WellKnownMDMSimpleMDM, "")
require.NoError(t, err)
loadSimple, err := ds.LoadHostByOrbitNodeKey(ctx, *hSimple.OrbitNodeKey)
@ -6727,7 +6807,7 @@ func testHostsLoadHostByOrbitNodeKey(t *testing.T, ds *Datastore) {
// create a host that will be pending enrollment in Fleet MDM
hFleet := createOrbitHost("fleet")
err = ds.SetOrUpdateMDMData(ctx, hFleet.ID, false, false, "https://fleetdm.com", true, fleet.WellKnownMDMFleet)
err = ds.SetOrUpdateMDMData(ctx, hFleet.ID, false, false, "https://fleetdm.com", true, fleet.WellKnownMDMFleet, "")
require.NoError(t, err)
loadFleet, err := ds.LoadHostByOrbitNodeKey(ctx, *hFleet.OrbitNodeKey)
@ -7690,3 +7770,137 @@ func testLastRestarted(t *testing.T, ds *Datastore) {
require.Equal(t, time.Duration(uptimeVal), host.Uptime)
require.Equal(t, hostsToVals[host.ID], host.LastRestartedAt)
}
func testHostHealth(t *testing.T, ds *Datastore) {
_, err := ds.GetHostHealth(context.Background(), 1)
require.Error(t, err)
var nfe fleet.NotFoundError
require.True(t, errors.As(err, &nfe))
// We'll check TeamIDs because at this level they should still be populated
team, err := ds.NewTeam(context.Background(), &fleet.Team{
Name: "team1",
})
require.NoError(t, err)
now := time.Now()
_, err = ds.NewHost(context.Background(), &fleet.Host{
ID: 1,
OsqueryHostID: ptr.String("foobar"),
NodeKey: ptr.String("nodekey"),
Hostname: "foobar.local",
UUID: "uuid",
Platform: "darwin",
DistributedInterval: 60,
LoggerTLSPeriod: 50,
ConfigTLSRefresh: 40,
DetailUpdatedAt: now,
LabelUpdatedAt: now,
LastEnrolledAt: now,
PolicyUpdatedAt: now,
RefetchRequested: true,
TeamID: ptr.Uint(team.ID),
SeenTime: now,
CPUType: "cpuType",
})
require.NoError(t, err)
h, err := ds.Host(context.Background(), 1)
require.NoError(t, err)
// set up policies
u := test.NewUser(t, ds, "Jack", "jack@example.com", true)
q := test.NewQuery(t, ds, nil, "passing_query", "select 1", 0, true)
passingPolicy, err := ds.NewGlobalPolicy(context.Background(), &u.ID, fleet.PolicyPayload{QueryID: &q.ID})
require.NoError(t, err)
q = test.NewQuery(t, ds, nil, "failing_query", "select 1", 0, true)
failingPolicy, err := ds.NewGlobalPolicy(context.Background(), &u.ID, fleet.PolicyPayload{QueryID: &q.ID})
require.NoError(t, err)
require.NoError(t, ds.RecordPolicyQueryExecutions(context.Background(), h, map[uint]*bool{passingPolicy.ID: ptr.Bool(true)}, time.Now(), false))
require.NoError(t, ds.RecordPolicyQueryExecutions(context.Background(), h, map[uint]*bool{failingPolicy.ID: ptr.Bool(false)}, time.Now(), false))
// set up vulnerable software
software := []fleet.Software{
{Name: "foo", Version: "0.0.1", Source: "chrome_extensions"},
{Name: "bar", Version: "0.0.3", Source: "apps"},
{Name: "baz", Version: "0.0.4", Source: "apps"},
}
_, err = ds.UpdateHostSoftware(context.Background(), h.ID, software)
require.NoError(t, err)
require.NoError(t, ds.LoadHostSoftware(context.Background(), h, false))
soft1 := h.Software[0]
if soft1.Name != "bar" {
soft1 = h.Software[1]
}
cpes := []fleet.SoftwareCPE{{SoftwareID: soft1.ID, CPE: "somecpe"}}
_, err = ds.UpsertSoftwareCPEs(context.Background(), cpes)
require.NoError(t, err)
// Reload software so that 'GeneratedCPEID is set.
require.NoError(t, ds.LoadHostSoftware(context.Background(), h, false))
soft1 = h.Software[0]
if soft1.Name != "bar" {
soft1 = h.Software[1]
}
inserted, err := ds.InsertSoftwareVulnerability(
context.Background(), fleet.SoftwareVulnerability{
SoftwareID: soft1.ID,
CVE: "cve-123-123-132",
}, fleet.NVDSource,
)
require.NoError(t, err)
require.True(t, inserted)
hh, err := ds.GetHostHealth(context.Background(), h.ID)
require.NoError(t, err)
require.Equal(t, h.Platform, hh.Platform)
require.Equal(t, h.DiskEncryptionEnabled, hh.DiskEncryptionEnabled)
require.Equal(t, h.OSVersion, hh.OsVersion)
require.Equal(t, ptr.Uint(team.ID), hh.TeamID)
require.Equal(t, h.UpdatedAt, hh.UpdatedAt)
require.Len(t, hh.FailingPolicies, 1)
require.Equal(t, failingPolicy.ID, hh.FailingPolicies[0].ID)
require.Len(t, hh.VulnerableSoftware, 1)
require.Equal(t, soft1.ID, hh.VulnerableSoftware[0].ID)
// Validate a host with no software or policies or team
_, err = ds.NewHost(context.Background(), &fleet.Host{
ID: 2,
OsqueryHostID: ptr.String("empty"),
NodeKey: ptr.String("empty_nodekey"),
Hostname: "empty.local",
UUID: "uuid123",
Platform: "darwin",
DistributedInterval: 60,
LoggerTLSPeriod: 50,
ConfigTLSRefresh: 40,
DetailUpdatedAt: now,
LabelUpdatedAt: now,
LastEnrolledAt: now,
PolicyUpdatedAt: now,
RefetchRequested: true,
SeenTime: now,
CPUType: "cpuType",
})
require.NoError(t, err)
h, err = ds.Host(context.Background(), 2)
require.NoError(t, err)
hh, err = ds.GetHostHealth(context.Background(), h.ID)
require.NoError(t, err)
require.Equal(t, h.Platform, hh.Platform)
require.Equal(t, h.DiskEncryptionEnabled, hh.DiskEncryptionEnabled)
require.Equal(t, h.OSVersion, hh.OsVersion)
require.Empty(t, hh.FailingPolicies)
require.Empty(t, hh.VulnerableSoftware)
require.Equal(t, h.TeamID, hh.TeamID)
}

8
server/datastore/mysql/labels_test.go
View file

@ -292,11 +292,11 @@ func testLabelsListHostsInLabel(t *testing.T, db *Datastore) {
ctx := context.Background()
const simpleMDM, kandji = "https://simplemdm.com", "https://kandji.io"
err = db.SetOrUpdateMDMData(ctx, h1.ID, false, true, simpleMDM, true, fleet.WellKnownMDMSimpleMDM) // enrollment: automatic
err = db.SetOrUpdateMDMData(ctx, h1.ID, false, true, simpleMDM, true, fleet.WellKnownMDMSimpleMDM, "") // enrollment: automatic
require.NoError(t, err)
err = db.SetOrUpdateMDMData(ctx, h2.ID, false, true, kandji, true, fleet.WellKnownMDMKandji) // enrollment: automatic
err = db.SetOrUpdateMDMData(ctx, h2.ID, false, true, kandji, true, fleet.WellKnownMDMKandji, "") // enrollment: automatic
require.NoError(t, err)
err = db.SetOrUpdateMDMData(ctx, h3.ID, false, false, simpleMDM, false, fleet.WellKnownMDMSimpleMDM) // enrollment: unenrolled
err = db.SetOrUpdateMDMData(ctx, h3.ID, false, false, simpleMDM, false, fleet.WellKnownMDMSimpleMDM, "") // enrollment: unenrolled
require.NoError(t, err)
var simpleMDMID uint
@ -1394,7 +1394,7 @@ func testLabelsListHostsInLabelOSSettings(t *testing.T, db *Datastore) {
// add two hosts to MDM to enforce disk encryption, fleet doesn't enforce settings on centos so h3 is not included
for _, h := range []*fleet.Host{h1, h2} {
require.NoError(t, db.SetOrUpdateMDMData(context.Background(), h.ID, false, true, "https://example.com", false, fleet.WellKnownMDMFleet))
require.NoError(t, db.SetOrUpdateMDMData(context.Background(), h.ID, false, true, "https://example.com", false, fleet.WellKnownMDMFleet, ""))
}
// add disk encryption key for h1
require.NoError(t, db.SetOrUpdateHostDiskEncryptionKey(context.Background(), h1.ID, "test-key", "", ptr.Bool(true)))

16
server/datastore/mysql/microsoft_mdm_test.go
View file

@ -287,7 +287,7 @@ func testMDMWindowsDiskEncryption(t *testing.T, ds *Datastore) {
require.NotNil(t, h)
hosts = append(hosts, h)
require.NoError(t, ds.SetOrUpdateMDMData(ctx, h.ID, false, true, "https://example.com", false, fleet.WellKnownMDMFleet))
require.NoError(t, ds.SetOrUpdateMDMData(ctx, h.ID, false, true, "https://example.com", false, fleet.WellKnownMDMFleet, ""))
}
t.Run("Disk encryption disabled", func(t *testing.T) {
@ -509,7 +509,7 @@ func testMDMWindowsDiskEncryption(t *testing.T, ds *Datastore) {
require.NoError(t, ds.SetOrUpdateMDMData(ctx,
hosts[3].ID,
true, // set is_server to true for hosts[3]
true, "https://example.com", false, fleet.WellKnownMDMFleet))
true, "https://example.com", false, fleet.WellKnownMDMFleet, ""))
// Check Windows servers not counted
checkExpected(t, nil, hostIDsByDEStatus{
@ -655,7 +655,7 @@ func testMDMWindowsProfilesSummary(t *testing.T, ds *Datastore) {
require.NotNil(t, h)
hosts = append(hosts, h)
require.NoError(t, ds.SetOrUpdateMDMData(ctx, h.ID, false, true, "https://example.com", false, fleet.WellKnownMDMFleet))
require.NoError(t, ds.SetOrUpdateMDMData(ctx, h.ID, false, true, "https://example.com", false, fleet.WellKnownMDMFleet, ""))
}
t.Run("profiles summary accounts for bitlocker status", func(t *testing.T) {
@ -927,7 +927,7 @@ func testMDMWindowsProfilesSummary(t *testing.T, ds *Datastore) {
require.NotNil(t, h)
otherHosts = append(otherHosts, h)
require.NoError(t, ds.SetOrUpdateMDMData(ctx, h.ID, false, true, "https://example.com", false, fleet.WellKnownMDMFleet))
require.NoError(t, ds.SetOrUpdateMDMData(ctx, h.ID, false, true, "https://example.com", false, fleet.WellKnownMDMFleet, ""))
}
checkExpected(t, nil, expected)
@ -1018,7 +1018,7 @@ func testMDMWindowsProfilesSummary(t *testing.T, ds *Datastore) {
checkExpected(t, nil, expected)
// report otherHosts[0] as a server
require.NoError(t, ds.SetOrUpdateMDMData(ctx, otherHosts[0].ID, true, true, "https://example.com", false, fleet.WellKnownMDMFleet))
require.NoError(t, ds.SetOrUpdateMDMData(ctx, otherHosts[0].ID, true, true, "https://example.com", false, fleet.WellKnownMDMFleet, ""))
// otherHosts[0] is no longer counted
expected = hostIDsByProfileStatus{
fleet.MDMDeliveryPending: []uint{hosts[0].ID, hosts[3].ID, otherHosts[1].ID, otherHosts[2].ID, otherHosts[3].ID, otherHosts[4].ID},
@ -1027,7 +1027,7 @@ func testMDMWindowsProfilesSummary(t *testing.T, ds *Datastore) {
checkExpected(t, nil, expected)
// report hosts[0] as a server
require.NoError(t, ds.SetOrUpdateMDMData(ctx, hosts[0].ID, true, true, "https://example.com", false, fleet.WellKnownMDMFleet))
require.NoError(t, ds.SetOrUpdateMDMData(ctx, hosts[0].ID, true, true, "https://example.com", false, fleet.WellKnownMDMFleet, ""))
// hosts[0] is no longer counted
expected = hostIDsByProfileStatus{
fleet.MDMDeliveryPending: []uint{hosts[3].ID, otherHosts[1].ID, otherHosts[2].ID, otherHosts[3].ID, otherHosts[4].ID},
@ -1036,7 +1036,7 @@ func testMDMWindowsProfilesSummary(t *testing.T, ds *Datastore) {
checkExpected(t, nil, expected)
// report hosts[3] as not enrolled
require.NoError(t, ds.SetOrUpdateMDMData(ctx, hosts[3].ID, false, false, "https://example.com", false, fleet.WellKnownMDMFleet))
require.NoError(t, ds.SetOrUpdateMDMData(ctx, hosts[3].ID, false, false, "https://example.com", false, fleet.WellKnownMDMFleet, ""))
// hosts[3] is no longer counted
expected = hostIDsByProfileStatus{
fleet.MDMDeliveryPending: []uint{otherHosts[1].ID, otherHosts[2].ID, otherHosts[3].ID, otherHosts[4].ID},
@ -1045,7 +1045,7 @@ func testMDMWindowsProfilesSummary(t *testing.T, ds *Datastore) {
checkExpected(t, nil, expected)
// report hosts[4] as enrolled to a different MDM
require.NoError(t, ds.SetOrUpdateMDMData(ctx, hosts[4].ID, false, true, "https://some-other-mdm.example.com", false, "some-other-mdm"))
require.NoError(t, ds.SetOrUpdateMDMData(ctx, hosts[4].ID, false, true, "https://some-other-mdm.example.com", false, "some-other-mdm", ""))
// hosts[4] is no longer counted
expected = hostIDsByProfileStatus{
fleet.MDMDeliveryPending: []uint{otherHosts[1].ID, otherHosts[2].ID, otherHosts[3].ID, otherHosts[4].ID},

23
server/datastore/mysql/migrations/tables/20231206142340_AlterHostMdmAddFleetEnrollRef.go Normal file
View file

@ -0,0 +1,23 @@
package tables
import (
"database/sql"
"fmt"
)
func init() {
MigrationClient.AddMigration(Up_20231206142340, Down_20231206142340)
}
func Up_20231206142340(tx *sql.Tx) error {
stmt := `ALTER TABLE host_mdm ADD COLUMN fleet_enroll_ref VARCHAR(36) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '';`
if _, err := tx.Exec(stmt); err != nil {
return fmt.Errorf("add fleet_enroll_ref to host_mdm: %w", err)
}
return nil
}
func Down_20231206142340(tx *sql.Tx) error {
return nil
}

65
server/datastore/mysql/migrations/tables/20231206142340_AlterHostMdmAddFleetEnrollRef_test.go Normal file
View file

@ -0,0 +1,65 @@
package tables
import (
"testing"
"github.com/google/uuid"
"github.com/stretchr/testify/require"
)
func TestUp_20231206142340(t *testing.T) {
db := applyUpToPrev(t)
insertStmt := `
INSERT INTO host_mdm (
host_id,
enrolled,
server_url
) VALUES (?, ?, ?)`
execNoErr(t, db, insertStmt, 1, 1, "https://example.com")
applyNext(t, db)
// verify that the new column is present
var hmdm struct {
ID uint `db:"id"`
HostID uint `db:"host_id"`
Enrolled bool `db:"enrolled"`
ServerURL string `db:"server_url"`
InstalledFromDEP bool `db:"installed_from_dep"`
MDMID *uint `db:"mdm_id"`
IsServer *bool `db:"is_server"`
FleetEnrollRef string `db:"fleet_enroll_ref"`
}
err := db.Get(&hmdm, "SELECT * FROM host_mdm WHERE host_id = ?", 1)
require.NoError(t, err)
require.Equal(t, uint(1), hmdm.HostID)
require.Equal(t, true, hmdm.Enrolled)
require.Equal(t, "https://example.com", hmdm.ServerURL)
require.Equal(t, false, hmdm.InstalledFromDEP)
require.Nil(t, hmdm.MDMID)
require.Nil(t, hmdm.IsServer)
require.Equal(t, "", hmdm.FleetEnrollRef)
insertStmt = `
INSERT INTO host_mdm (
host_id,
enrolled,
server_url,
fleet_enroll_ref
) VALUES (?, ?, ?, ?)`
ref := uuid.NewString()
execNoErr(t, db, insertStmt, 2, 1, "https://example.com", ref)
err = db.Get(&hmdm, "SELECT * FROM host_mdm WHERE host_id = ?", 2)
require.NoError(t, err)
require.Equal(t, uint(2), hmdm.HostID)
require.Equal(t, true, hmdm.Enrolled)
require.Equal(t, "https://example.com", hmdm.ServerURL)
require.Equal(t, false, hmdm.InstalledFromDEP)
require.Nil(t, hmdm.MDMID)
require.Nil(t, hmdm.IsServer)
require.Equal(t, ref, hmdm.FleetEnrollRef)
}

35
server/datastore/mysql/migrations/tables/20231207102320_AlterSoftwareTitlesAddBrowser.go Normal file
View file

@ -0,0 +1,35 @@
package tables
import (
"database/sql"
"fmt"
)
func init() {
MigrationClient.AddMigration(Up_20231207102320, Down_20231207102320)
}
func Up_20231207102320(tx *sql.Tx) error {
_, err := tx.Exec((`DELETE FROM software_titles;`)) // delete all software titles, it will be repopulated on the next cron
if err != nil {
return fmt.Errorf("failed to delete software titles: %w", err)
}
_, err = tx.Exec(`
ALTER TABLE software_titles
ADD COLUMN browser varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '';`)
if err != nil {
return fmt.Errorf("failed to add browser column to software titles table: %w", err)
}
_, err = tx.Exec(`ALTER TABLE software_titles DROP KEY idx_software_titles_name_source;`)
if err != nil {
return fmt.Errorf("failed to drop name-source key from software titles table: %w", err)
}
return nil
}
func Down_20231207102320(tx *sql.Tx) error {
return nil
}

74
server/datastore/mysql/migrations/tables/20231207102320_AlterSoftwareTitlesAddBrowser_test.go Normal file
View file

@ -0,0 +1,74 @@
package tables
import (
"context"
"testing"
"github.com/jmoiron/sqlx"
"github.com/stretchr/testify/require"
)
func TestUp_20231207102320(t *testing.T) {
db := applyUpToPrev(t)
insertStmt := "INSERT INTO software_titles (name, source) VALUES (?, ?)"
_, err := db.Exec(insertStmt, "test-name", "test-source")
require.NoError(t, err)
selectStmt := "SELECT id, name, source FROM software_titles"
var rows []struct {
ID uint `db:"id"`
Name string `db:"name"`
Source string `db:"source"`
}
err = sqlx.SelectContext(context.Background(), db, &rows, selectStmt)
require.NoError(t, err)
require.Len(t, rows, 1)
applyNext(t, db)
selectStmt = "SELECT id, name, source, browser FROM software_titles"
type newRow struct {
ID uint `db:"id"`
Name string `db:"name"`
Source string `db:"source"`
Browser string `db:"browser"`
}
var newRows []newRow
// migration should delete all rows
err = sqlx.SelectContext(context.Background(), db, &newRows, selectStmt)
require.NoError(t, err)
require.Len(t, newRows, 0)
// re-insert the old row
_, err = db.Exec(insertStmt, "test-name", "test-source")
require.NoError(t, err)
err = sqlx.SelectContext(context.Background(), db, &newRows, selectStmt)
require.NoError(t, err)
require.Len(t, newRows, 1)
require.Equal(t, "test-name", newRows[0].Name)
require.Equal(t, "test-source", newRows[0].Source)
require.Equal(t, "", newRows[0].Browser) // default browser is empty string
insertStmt = "INSERT INTO software_titles (name, source, browser) VALUES (?, ?, ?)"
_, err = db.Exec(insertStmt, "test-name", "test-source", "test-browser")
require.NoError(t, err)
newRows = []newRow{}
err = sqlx.SelectContext(context.Background(), db, &newRows, selectStmt)
require.NoError(t, err)
require.Len(t, newRows, 2)
var found bool
for _, row := range newRows {
if row.Browser == "test-browser" {
require.False(t, found)
found = true
} else {
// browser should be empty for existing rows
require.Equal(t, "", row.Browser)
}
}
}

27
server/datastore/mysql/migrations/tables/20231207102321_AddIndexSoftwareNameSourceBrowser.go Normal file
View file

@ -0,0 +1,27 @@
package tables
import (
"database/sql"
"fmt"
)
func init() {
MigrationClient.AddMigration(Up_20231207102321, Down_20231207102321)
}
func Up_20231207102321(tx *sql.Tx) error {
_, err := tx.Exec(`ALTER TABLE software_titles ADD UNIQUE INDEX idx_sw_titles (name, source, browser);`)
if err != nil {
return fmt.Errorf("failed to add unique index to software titles table: %w", err)
}
_, err = tx.Exec(`ALTER TABLE software ADD INDEX idx_sw_name_source_browser (name, source, browser);`)
if err != nil {
return fmt.Errorf("failed to add name-source-browser index to software table: %w", err)
}
return nil
}
func Down_20231207102321(tx *sql.Tx) error {
return nil
}

39
server/datastore/mysql/migrations/tables/20231207102321_AddIndexSoftwareNameSourceBrowser_test.go Normal file
View file

@ -0,0 +1,39 @@
package tables
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestUp_20231207102321(t *testing.T) {
db := applyUpToPrev(t)
insertStmt := "INSERT INTO software_titles (name, source, browser) VALUES (?, ?, ?)"
_, err := db.Exec(insertStmt, "test-name", "test-source", "")
require.NoError(t, err)
_, err = db.Exec(insertStmt, "test-name2", "test-source", "")
require.NoError(t, err)
applyNext(t, db)
// unique constraint applies to name+source+browser
_, err = db.Exec(insertStmt, "test-name", "test-source", "")
require.ErrorContains(t, err, "Duplicate entry")
_, err = db.Exec(insertStmt, "test-name", "test-source", "test-browser")
require.NoError(t, err)
_, err = db.Exec(insertStmt, "test-name2", "test-source", "test-browser")
require.NoError(t, err)
_, err = db.Exec(insertStmt, "test-name2", "test-source2", "test-browser")
require.NoError(t, err)
_, err = db.Exec(insertStmt, "test-name2", "test-source2", "test-browser2")
require.NoError(t, err)
_, err = db.Exec(insertStmt, "test-name2", "test-source2", "test-browser2")
require.ErrorContains(t, err, "Duplicate entry")
}

40
server/datastore/mysql/migrations/tables/20231207133731_FixStatsTypes.go Normal file
View file

@ -0,0 +1,40 @@
package tables
import (
"database/sql"
"fmt"
)
func init() {
MigrationClient.AddMigration(Up_20231207133731, Down_20231207133731)
}
func Up_20231207133731(tx *sql.Tx) error {
// Updating some bad data from osquery (which will be ignored in a later PR).
// Seems safer to update rather than delete.
stmt := `
UPDATE scheduled_query_stats SET last_executed = '1970-01-01 00:00:01' WHERE YEAR(last_executed) = '0000';
`
if _, err := tx.Exec(stmt); err != nil {
return fmt.Errorf("fixing last_executed in scheduled_query_stats: %w", err)
}
stmt = `
ALTER TABLE scheduled_query_stats
MODIFY COLUMN average_memory BIGINT UNSIGNED NOT NULL,
MODIFY COLUMN executions BIGINT UNSIGNED NOT NULL,
MODIFY COLUMN output_size BIGINT UNSIGNED NOT NULL,
MODIFY COLUMN system_time BIGINT UNSIGNED NOT NULL,
MODIFY COLUMN user_time BIGINT UNSIGNED NOT NULL,
MODIFY COLUMN wall_time BIGINT UNSIGNED NOT NULL;
`
if _, err := tx.Exec(stmt); err != nil {
return fmt.Errorf("changing data types for scheduled_query_stats: %w", err)
}
return nil
}
func Down_20231207133731(tx *sql.Tx) error {
return nil
}

68
server/datastore/mysql/migrations/tables/20231207133731_FixStatsTypes_test.go Normal file
View file

@ -0,0 +1,68 @@
package tables
import (
"github.com/stretchr/testify/require"
"math"
"testing"
)
func TestUp_20231207133731(t *testing.T) {
db := applyUpToPrev(t)
setupStmt := `
INSERT INTO scheduled_query_stats (host_id, scheduled_query_id, average_memory, denylisted, executions, schedule_interval, output_size, system_time, user_time, wall_time, last_executed) VALUES
(?,?,?,?,?,?,?,?,?,?,?);
`
_, err := db.Exec(setupStmt, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, "2023-12-07 13:17:17")
require.NoError(t, err)
// Apply current migration.
applyNext(t, db)
stmt := `
SELECT host_id, average_memory FROM scheduled_query_stats WHERE host_id = 1;
`
rows, err := db.Query(stmt)
require.NoError(t, rows.Err())
require.NoError(t, err)
defer rows.Close()
count := 0
for rows.Next() {
count += 1
var hostId int
var avgMem uint64
err := rows.Scan(&hostId, &avgMem)
require.NoError(t, err)
require.Equal(t, 1, hostId)
require.Equal(t, uint64(3), avgMem)
}
require.Equal(t, 1, count)
_, err = db.Exec(setupStmt, 2, 2, uint64(math.MaxUint64), 4, uint64(math.MaxUint64-1), 6, uint64(math.MaxUint64-2), uint64(math.MaxUint64-3), uint64(math.MaxUint64-4), uint64(math.MaxUint64-5), "2023-12-07 13:17:17")
require.NoError(t, err)
stmt = `
SELECT host_id, average_memory, executions, output_size, system_time, user_time, wall_time FROM scheduled_query_stats WHERE host_id = 2;
`
rows, err = db.Query(stmt)
require.NoError(t, rows.Err())
require.NoError(t, err)
defer rows.Close()
count = 0
for rows.Next() {
count += 1
var hostId int
var avgMem, executions, outputSize, systemTime, userTime, wallTime uint64
err := rows.Scan(&hostId, &avgMem, &executions, &outputSize, &systemTime, &userTime, &wallTime)
require.NoError(t, err)
require.Equal(t, 2, hostId)
require.Equal(t, uint64(math.MaxUint64), avgMem)
require.Equal(t, uint64(math.MaxUint64-1), executions)
require.Equal(t, uint64(math.MaxUint64-2), outputSize)
require.Equal(t, uint64(math.MaxUint64-3), systemTime)
require.Equal(t, uint64(math.MaxUint64-4), userTime)
require.Equal(t, uint64(math.MaxUint64-5), wallTime)
}
require.Equal(t, 1, count)
}

12
server/datastore/mysql/packs_test.go
View file

@ -444,15 +444,15 @@ func randomPackStatsForHost(packID uint, packName string, packType string, sched
QueryName: sq.QueryName,
Description: sq.Description,
PackID: packID,
AverageMemory: rand.Intn(100),
AverageMemory: uint64(rand.Intn(100)),
Denylisted: false,
Executions: rand.Intn(100),
Executions: uint64(rand.Intn(100)),
Interval: rand.Intn(100),
LastExecuted: time.Now(),
OutputSize: rand.Intn(1000),
SystemTime: rand.Intn(1000),
UserTime: rand.Intn(1000),
WallTime: rand.Intn(1000),
OutputSize: uint64(rand.Intn(1000)),
SystemTime: uint64(rand.Intn(1000)),
UserTime: uint64(rand.Intn(1000)),
WallTime: uint64(rand.Intn(1000)),
})
}
return []fleet.PackStats{

10
server/datastore/mysql/query_results.go
View file

@ -2,6 +2,7 @@ package mysql
import (
"context"
"fmt"
"strings"
"github.com/fleetdm/fleet/v4/server/contexts/ctxerr"
@ -87,14 +88,15 @@ func (ds *Datastore) OverwriteQueryResultRows(ctx context.Context, rows []*fleet
// TODO(lucas): Any chance we can store hostname in the query_results table?
// (to avoid having to left join hosts).
func (ds *Datastore) QueryResultRows(ctx context.Context, queryID uint) ([]*fleet.ScheduledQueryResultRow, error) {
selectStmt := `
func (ds *Datastore) QueryResultRows(ctx context.Context, queryID uint, filter fleet.TeamFilter) ([]*fleet.ScheduledQueryResultRow, error) {
selectStmt := fmt.Sprintf(`
SELECT qr.query_id, qr.host_id, qr.last_fetched, qr.data,
h.hostname, h.computer_name, h.hardware_model, h.hardware_serial
FROM query_results qr
LEFT JOIN hosts h ON (qr.host_id=h.id)
WHERE query_id = ? AND data IS NOT NULL
`
WHERE query_id = ? AND data IS NOT NULL AND %s
`, ds.whereFilterHostsByTeams(filter, "h"))
results := []*fleet.ScheduledQueryResultRow{}
err := sqlx.SelectContext(ctx, ds.reader(ctx), &results, selectStmt, queryID)
if err != nil {

107
server/datastore/mysql/query_results_test.go
View file

@ -28,6 +28,7 @@ func TestQueryResults(t *testing.T) {
{"Overwrite", testOverwriteQueryResultRows},
{"MaxRows", testQueryResultRowsDoNotExceedMaxRows},
{"QueryResultRows", testQueryResultRows},
{"QueryResultRowsFilter", testQueryResultRowsTeamFilter},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@ -117,6 +118,108 @@ func testGetQueryResultRows(t *testing.T, ds *Datastore) {
require.Len(t, results, 0)
}
func testQueryResultRowsTeamFilter(t *testing.T, ds *Datastore) {
team, err := ds.NewTeam(context.Background(), &fleet.Team{
Name: "teamFoo",
})
require.NoError(t, err)
observerTeam, err := ds.NewTeam(context.Background(), &fleet.Team{
Name: "observerTeam",
})
require.NoError(t, err)
teamUser, err := ds.NewUser(context.Background(), &fleet.User{
Password: []byte("foo"),
Salt: "bar",
Name: "teamUser",
Email: "teamUser@example.com",
GlobalRole: nil,
Teams: []fleet.UserTeam{
{
Team: *team,
Role: fleet.RoleAdmin,
},
{
Team: *observerTeam,
Role: fleet.RoleObserver,
},
},
})
require.NoError(t, err)
query := test.NewQuery(t, ds, nil, "New Query", "SELECT 1", teamUser.ID, true)
globalHost := test.NewHost(t, ds, "globalHost", "192.168.1.100", "1111", "UI8XB1223", time.Now())
teamHost := test.NewHost(t, ds, "teamHost", "192.168.1.100", "2222", "UI8XB1223", time.Now())
err = ds.AddHostsToTeam(context.Background(), &team.ID, []uint{teamHost.ID})
require.NoError(t, err)
observerTeamHost := test.NewHost(t, ds, "teamHost", "192.168.1.100", "3333", "UI8XB1223", time.Now())
err = ds.AddHostsToTeam(context.Background(), &observerTeam.ID, []uint{observerTeamHost.ID})
require.NoError(t, err)
mockTime := time.Now().UTC().Truncate(time.Second)
globalRow := []*fleet.ScheduledQueryResultRow{
{
QueryID: query.ID,
HostID: globalHost.ID,
LastFetched: mockTime,
Data: json.RawMessage(`{
"model": "Global USB Keyboard",
"vendor": "Global Inc."
}`),
},
}
err = ds.OverwriteQueryResultRows(context.Background(), globalRow)
require.NoError(t, err)
teamRow := []*fleet.ScheduledQueryResultRow{
{
QueryID: query.ID,
HostID: teamHost.ID,
LastFetched: mockTime,
Data: json.RawMessage(`{
"model": "Team USB Keyboard",
"vendor": "Team Inc."
}`),
},
}
err = ds.OverwriteQueryResultRows(context.Background(), teamRow)
require.NoError(t, err)
observerTeamRow := []*fleet.ScheduledQueryResultRow{
{
QueryID: query.ID,
HostID: observerTeamHost.ID,
LastFetched: mockTime,
Data: json.RawMessage(`{
"model": "Team USB Keyboard",
"vendor": "Team Inc."
}`),
},
}
err = ds.OverwriteQueryResultRows(context.Background(), observerTeamRow)
require.NoError(t, err)
filter := fleet.TeamFilter{
User: teamUser,
IncludeObserver: true,
}
results, err := ds.QueryResultRows(context.Background(), query.ID, filter)
require.NoError(t, err)
require.Len(t, results, 2)
require.Equal(t, teamRow[0].HostID, results[0].HostID)
require.Equal(t, teamRow[0].QueryID, results[0].QueryID)
require.Equal(t, teamRow[0].LastFetched, results[0].LastFetched)
require.JSONEq(t, string(teamRow[0].Data), string(results[0].Data))
require.Equal(t, observerTeamRow[0].HostID, results[1].HostID)
require.Equal(t, observerTeamRow[0].QueryID, results[1].QueryID)
require.Equal(t, observerTeamRow[0].LastFetched, results[1].LastFetched)
require.JSONEq(t, string(observerTeamRow[0].Data), string(results[1].Data))
}
func testCountResultsForQuery(t *testing.T, ds *Datastore) {
user := test.NewUser(t, ds, "Test User", "test@example.com", true)
query1 := test.NewQuery(t, ds, nil, "New Query", "SELECT 1", user.ID, true)
@ -456,8 +559,10 @@ func testQueryResultRows(t *testing.T, ds *Datastore) {
err := ds.OverwriteQueryResultRows(context.Background(), overwriteRows)
require.NoError(t, err)
filter := fleet.TeamFilter{User: user, IncludeObserver: true}
// Test calling QueryResultRows with a query that has an entry with a host that doesn't exist anymore.
results, err := ds.QueryResultRows(context.Background(), query.ID)
results, err := ds.QueryResultRows(context.Background(), query.ID, filter)
require.NoError(t, err)
require.Len(t, results, 1)
}

2
server/datastore/mysql/scheduled_queries_test.go
View file

@ -508,7 +508,7 @@ func testScheduledQueriesAsyncBatchSaveStats(t *testing.T, ds *Datastore) {
for hid, stats := range m {
for _, st := range stats {
ExecAdhocSQL(t, ds, func(tx sqlx.ExtContext) error {
var got int
var got uint64
err := sqlx.GetContext(ctx, tx, &got, `SELECT executions FROM scheduled_query_stats WHERE host_id = ? AND scheduled_query_id = ?`, hid, st.ScheduledQueryID)
if err != nil {
return err

23
server/datastore/mysql/schema.sql
View file

File diff suppressed because one or more lines are too long

66
server/datastore/mysql/software.go
View file

@ -5,6 +5,7 @@ import (
"database/sql"
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
@ -32,6 +33,39 @@ func (ds *Datastore) UpdateHostSoftware(ctx context.Context, hostID uint, softwa
result = r
return err
})
if err != nil {
return result, err
}
// We perform the following cleanup on a separate transaction to avoid deadlocks.
//
// Cleanup the software table when no more hosts have the deleted host_software
// table entries. Otherwise the software will be listed by ds.ListSoftware but
// ds.SoftwareByID, ds.CountHosts and ds.ListHosts will return a *notFoundError
// error for such software.
if len(result.Deleted) > 0 {
deletesHostSoftwareIDs := make([]uint, 0, len(result.Deleted))
for _, software := range result.Deleted {
deletesHostSoftwareIDs = append(deletesHostSoftwareIDs, software.ID)
}
slices.Sort(deletesHostSoftwareIDs)
if err := ds.withRetryTxx(ctx, func(tx sqlx.ExtContext) error {
stmt := `DELETE FROM software WHERE id IN (?) AND NOT EXISTS (
SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
)`
stmt, args, err := sqlx.In(stmt, deletesHostSoftwareIDs)
if err != nil {
return ctxerr.Wrap(ctx, err, "build delete software query")
}
if _, err := tx.ExecContext(ctx, stmt, args...); err != nil {
return ctxerr.Wrap(ctx, err, "delete software")
}
return nil
}); err != nil {
return result, err
}
}
return result, err
}
@ -276,6 +310,7 @@ SELECT
s.name,
s.version,
s.source,
s.browser,
s.bundle_identifier,
s.release,
s.vendor,
@ -374,23 +409,6 @@ func deleteUninstalledHostSoftwareDB(
return nil, ctxerr.Wrap(ctx, err, "delete host software")
}
// Cleanup the software table when no more hosts have the deleted host_software
// table entries.
// Otherwise the software will be listed by ds.ListSoftware but ds.SoftwareByID,
// ds.CountHosts and ds.ListHosts will return a *notFoundError error for such
// software.
stmt = `DELETE FROM software WHERE id IN (?) AND
NOT EXISTS (
SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
)`
stmt, args, err = sqlx.In(stmt, deletesHostSoftwareIDs)
if err != nil {
return nil, ctxerr.Wrap(ctx, err, "build delete software query")
}
if _, err := tx.ExecContext(ctx, stmt, args...); err != nil {
return nil, ctxerr.Wrap(ctx, err, "delete software")
}
return deletedSoftware, nil
}
@ -1066,6 +1084,7 @@ func (ds *Datastore) SoftwareByID(ctx context.Context, id uint, includeCVEScores
"s.name",
"s.version",
"s.source",
"s.browser",
"s.bundle_identifier",
"s.release",
"s.vendor",
@ -1292,14 +1311,15 @@ func (ds *Datastore) ReconcileSoftwareTitles(ctx context.Context) error {
// ensure all software titles are in the software_titles table
upsertTitlesStmt := `
INSERT INTO software_titles (name, source)
INSERT INTO software_titles (name, source, browser)
SELECT DISTINCT
name,
source
source,
browser
FROM
software s
WHERE
NOT EXISTS (SELECT 1 FROM software_titles st WHERE (s.name, s.source) = (st.name, st.source))
NOT EXISTS (SELECT 1 FROM software_titles st WHERE (s.name, s.source, s.browser) = (st.name, st.source, st.browser))
ON DUPLICATE KEY UPDATE software_titles.id = software_titles.id`
// TODO: consider the impact of on duplicate key update vs. risk of insert ignore
// or performing a select first to see if the title exists and only inserting
@ -1320,15 +1340,15 @@ UPDATE
SET
s.title_id = st.id
WHERE
(s.name, s.source) = (st.name, st.source)
(s.name, s.source, s.browser) = (st.name, st.source, st.browser)
AND (s.title_id IS NULL OR s.title_id != st.id)`
res, err = ds.writer(ctx).ExecContext(ctx, updateSoftwareStmt)
if err != nil {
return ctxerr.Wrap(ctx, err, "update software titles")
return ctxerr.Wrap(ctx, err, "update software title_id")
}
n, _ = res.RowsAffected()
level.Debug(ds.logger).Log("msg", "update software titles", "rows_affected", n)
level.Debug(ds.logger).Log("msg", "update software title_id", "rows_affected", n)
// clean up orphaned software titles
cleanupStmt := `

117
server/datastore/mysql/software_test.go
View file

@ -17,6 +17,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/exp/slices"
"golang.org/x/sync/errgroup"
)
func TestSoftware(t *testing.T) {
@ -39,6 +40,7 @@ func TestSoftware(t *testing.T) {
{"HostsByCVE", testHostsByCVE},
{"HostVulnSummariesBySoftwareIDs", testHostVulnSummariesBySoftwareIDs},
{"UpdateHostSoftware", testUpdateHostSoftware},
{"UpdateHostSoftwareDeadlock", testUpdateHostSoftwareDeadlock},
{"UpdateHostSoftwareUpdatesSoftware", testUpdateHostSoftwareUpdatesSoftware},
{"ListSoftwareByHostIDShort", testListSoftwareByHostIDShort},
{"ListSoftwareVulnerabilitiesByHostIDsSource", testListSoftwareVulnerabilitiesByHostIDsSource},
@ -1953,7 +1955,6 @@ func testSoftwareByIDNoDuplicatedVulns(t *testing.T, ds *Datastore) {
require.NoError(t, ds.LoadHostSoftware(ctx, hostB, false))
// Add one vulnerability to each software
var vulns []fleet.SoftwareVulnerability
for i, s := range hostA.Software {
inserted, err := ds.InsertSoftwareVulnerability(ctx, fleet.SoftwareVulnerability{
SoftwareID: s.ID,
@ -1961,7 +1962,6 @@ func testSoftwareByIDNoDuplicatedVulns(t *testing.T, ds *Datastore) {
}, fleet.UbuntuOVALSource)
require.NoError(t, err)
require.True(t, inserted)
vulns = append(vulns)
}
for _, s := range hostA.Software {
@ -2588,7 +2588,7 @@ func TestReconcileSoftwareTitles(t *testing.T) {
host3 := test.NewHost(t, ds, "host3", "", "host3key", "host3uuid", time.Now())
expectedSoftware := []fleet.Software{
{Name: "foo", Version: "0.0.1", Source: "chrome_extensions"},
{Name: "foo", Version: "0.0.1", Source: "chrome_extensions", Browser: "chrome"},
{Name: "foo", Version: "v0.0.2", Source: "chrome_extensions"},
{Name: "foo", Version: "0.0.3", Source: "chrome_extensions"},
{Name: "bar", Version: "0.0.3", Source: "deb_packages"},
@ -2608,7 +2608,7 @@ func TestReconcileSoftwareTitles(t *testing.T) {
getSoftware := func() ([]fleet.Software, error) {
var sw []fleet.Software
err := ds.writer(ctx).SelectContext(ctx, &sw, `SELECT * FROM software ORDER BY name, version`)
err := ds.writer(ctx).SelectContext(ctx, &sw, `SELECT * FROM software ORDER BY name, source, browser, version`)
if err != nil {
return nil, err
}
@ -2617,32 +2617,32 @@ func TestReconcileSoftwareTitles(t *testing.T) {
getTitles := func() ([]fleet.SoftwareTitle, error) {
var swt []fleet.SoftwareTitle
err := ds.writer(ctx).SelectContext(ctx, &swt, `SELECT * FROM software_titles ORDER BY name, source`)
err := ds.writer(ctx).SelectContext(ctx, &swt, `SELECT * FROM software_titles ORDER BY name, source, browser`)
if err != nil {
return nil, err
}
return swt, nil
}
expectedTitlesByNS := map[string]fleet.SoftwareTitle{}
expectedTitlesByNSB := map[string]fleet.SoftwareTitle{}
assertSoftware := func(t *testing.T, wantSoftware []fleet.Software, wantNilTitleID []fleet.Software) {
gotSoftware, err := getSoftware()
require.NoError(t, err)
require.Len(t, gotSoftware, len(wantSoftware))
byNSV := map[string]fleet.Software{}
byNSBV := map[string]fleet.Software{}
for _, s := range wantSoftware {
byNSV[s.Name+s.Source+s.Version] = s
byNSBV[s.Name+s.Source+s.Browser+s.Version] = s
}
for _, r := range gotSoftware {
_, ok := byNSV[r.Name+r.Source+r.Version]
_, ok := byNSBV[r.Name+r.Source+r.Browser+r.Version]
require.True(t, ok)
if r.TitleID == nil {
var found bool
for _, s := range wantNilTitleID {
if s.Name == r.Name && s.Source == r.Source && s.Version == r.Version {
if s.Name == r.Name && s.Source == r.Source && s.Browser == r.Browser && s.Version == r.Version {
found = true
break
}
@ -2650,12 +2650,13 @@ func TestReconcileSoftwareTitles(t *testing.T) {
require.True(t, found)
} else {
require.NotNil(t, r.TitleID)
swt, ok := expectedTitlesByNS[r.Name+r.Source]
swt, ok := expectedTitlesByNSB[r.Name+r.Source+r.Browser]
require.True(t, ok)
require.NotNil(t, r.TitleID)
require.Equal(t, swt.ID, *r.TitleID)
require.Equal(t, swt.Name, r.Name)
require.Equal(t, swt.Source, r.Source)
require.Equal(t, swt.Browser, r.Browser)
}
}
}
@ -2665,11 +2666,12 @@ func TestReconcileSoftwareTitles(t *testing.T) {
if len(expectMissing) > 0 {
require.NotContains(t, expectMissing, r.Name)
}
e, ok := expectedTitlesByNS[r.Name+r.Source]
e, ok := expectedTitlesByNSB[r.Name+r.Source+r.Browser]
require.True(t, ok)
require.Equal(t, e.ID, r.ID)
require.Equal(t, e.Name, r.Name)
require.Equal(t, e.Source, r.Source)
require.Equal(t, e.Browser, r.Browser)
}
}
@ -2680,19 +2682,27 @@ func TestReconcileSoftwareTitles(t *testing.T) {
require.NoError(t, ds.ReconcileSoftwareTitles(ctx))
swt, err := getTitles()
require.NoError(t, err)
require.Len(t, swt, 3)
require.Len(t, swt, 4)
require.Equal(t, swt[0].Name, "bar")
require.Equal(t, swt[0].Source, "deb_packages")
expectedTitlesByNS[swt[0].Name+swt[0].Source] = swt[0]
require.Equal(t, swt[0].Browser, "")
expectedTitlesByNSB[swt[0].Name+swt[0].Source+swt[0].Browser] = swt[0]
require.Equal(t, swt[1].Name, "baz")
require.Equal(t, swt[1].Source, "deb_packages")
expectedTitlesByNS[swt[1].Name+swt[1].Source] = swt[1]
require.Equal(t, swt[1].Browser, "")
expectedTitlesByNSB[swt[1].Name+swt[1].Source+swt[1].Browser] = swt[1]
require.Equal(t, swt[2].Name, "foo")
require.Equal(t, swt[2].Source, "chrome_extensions")
expectedTitlesByNS[swt[2].Name+swt[2].Source] = swt[2]
require.Equal(t, swt[2].Browser, "")
expectedTitlesByNSB[swt[2].Name+swt[2].Source+swt[2].Browser] = swt[2]
require.Equal(t, swt[3].Name, "foo")
require.Equal(t, swt[3].Source, "chrome_extensions")
require.Equal(t, swt[3].Browser, "chrome")
expectedTitlesByNSB[swt[3].Name+swt[3].Source+swt[3].Browser] = swt[3]
// title_id is now populated for all software entries
assertSoftware(t, expectedSoftware, nil)
@ -2706,7 +2716,7 @@ func TestReconcileSoftwareTitles(t *testing.T) {
require.NoError(t, ds.ReconcileSoftwareTitles(context.Background()))
gotTitles, err := getTitles()
require.NoError(t, err)
require.Len(t, gotTitles, 2)
require.Len(t, gotTitles, 3)
assertTitles(t, gotTitles, []string{"bar"})
// add bar to host 3
@ -2720,20 +2730,20 @@ func TestReconcileSoftwareTitles(t *testing.T) {
// bar isn't added back to software titles until we reconcile software titles
gotTitles, err = getTitles()
require.NoError(t, err)
require.Len(t, gotTitles, 2)
require.Len(t, gotTitles, 3)
assertTitles(t, gotTitles, []string{"bar"})
// reconcile software titles
require.NoError(t, ds.ReconcileSoftwareTitles(ctx))
gotTitles, err = getTitles()
require.NoError(t, err)
require.Len(t, gotTitles, 3)
require.Len(t, gotTitles, 4)
// bar was added back to software titles with a new ID
require.Equal(t, gotTitles[0].Name, "bar")
require.Equal(t, gotTitles[0].Source, "deb_packages")
require.NotEqual(t, expectedTitlesByNS[gotTitles[0].Name+gotTitles[0].Source], gotTitles[0].ID)
expectedTitlesByNS[gotTitles[0].Name+gotTitles[0].Source] = gotTitles[0]
require.Equal(t, "bar", gotTitles[0].Name)
require.Equal(t, "deb_packages", gotTitles[0].Source)
require.NotEqual(t, expectedTitlesByNSB[gotTitles[0].Name+gotTitles[0].Source], gotTitles[0].ID)
expectedTitlesByNSB[gotTitles[0].Name+gotTitles[0].Source] = gotTitles[0]
assertTitles(t, gotTitles, nil)
// title_id is now populated for bar
@ -2751,7 +2761,7 @@ func TestReconcileSoftwareTitles(t *testing.T) {
require.NoError(t, ds.ReconcileSoftwareTitles(ctx))
gotTitles, err = getTitles()
require.NoError(t, err)
require.Len(t, gotTitles, 3)
require.Len(t, gotTitles, 4)
assertTitles(t, gotTitles, nil)
// title_id is now populated for new version of foo
@ -2769,12 +2779,63 @@ func TestReconcileSoftwareTitles(t *testing.T) {
require.NoError(t, ds.ReconcileSoftwareTitles(ctx))
gotTitles, err = getTitles()
require.NoError(t, err)
require.Len(t, gotTitles, 4)
require.Equal(t, gotTitles[3].Name, "foo")
require.Equal(t, gotTitles[3].Source, "rpm_packages")
expectedTitlesByNS[gotTitles[3].Name+gotTitles[3].Source] = gotTitles[3]
require.Len(t, gotTitles, 5)
require.Equal(t, "foo", gotTitles[4].Name)
require.Equal(t, "rpm_packages", gotTitles[4].Source)
require.Equal(t, "", gotTitles[4].Browser)
expectedTitlesByNSB[gotTitles[4].Name+gotTitles[4].Source+gotTitles[4].Browser] = gotTitles[4]
assertTitles(t, gotTitles, nil)
// title_id is now populated for new source of foo
assertSoftware(t, expectedSoftware, nil)
}
func testUpdateHostSoftwareDeadlock(t *testing.T, ds *Datastore) {
// To increase chance of deadlock increase these numbers.
// We are keeping them low to not cause CI issues ("too many connections" errors
// due to concurrent tests).
const (
hostCount = 10
updateCount = 10
)
ctx := context.Background()
var hosts []*fleet.Host
for i := 1; i <= hostCount; i++ {
h, err := ds.NewHost(ctx, &fleet.Host{
ID: uint(i),
OsqueryHostID: ptr.String(fmt.Sprintf("id-%d", i)),
NodeKey: ptr.String(fmt.Sprintf("key-%d", i)),
Platform: "linux",
Hostname: fmt.Sprintf("host-%d", i),
DetailUpdatedAt: time.Now(),
LabelUpdatedAt: time.Now(),
PolicyUpdatedAt: time.Now(),
SeenTime: time.Now(),
})
require.NoError(t, err)
hosts = append(hosts, h)
}
var g errgroup.Group
for _, h := range hosts {
hostID := h.ID
g.Go(func() error {
for i := 0; i < updateCount; i++ {
software := []fleet.Software{
{Name: "foo", Version: "0.0.1", Source: "test", GenerateCPE: "cpe_foo"},
{Name: "bar", Version: "0.0.2", Source: "test", GenerateCPE: "cpe_bar"},
{Name: "baz", Version: "0.0.3", Source: "test", GenerateCPE: "cpe_baz"},
}
removeIdx := rand.Intn(len(software))
software = append(software[:removeIdx], software[removeIdx+1:]...)
if _, err := ds.UpdateHostSoftware(ctx, hostID, software); err != nil {
return err
}
time.Sleep(10 * time.Millisecond)
}
return nil
})
}
err := g.Wait()
require.NoError(t, err)
}

199
server/datastore/mysql/software_titles.go Normal file
View file

@ -0,0 +1,199 @@
package mysql
import (
"context"
"database/sql"
"fmt"
"github.com/fleetdm/fleet/v4/server/contexts/ctxerr"
"github.com/fleetdm/fleet/v4/server/fleet"
"github.com/jmoiron/sqlx"
)
func (ds *Datastore) SoftwareTitleByID(ctx context.Context, id uint) (*fleet.SoftwareTitle, error) {
const selectSoftwareTitleStmt = `
SELECT
st.id,
st.name,
st.source,
COUNT(DISTINCT hs.host_id) AS hosts_count,
COUNT(DISTINCT s.id) AS versions_count
FROM software_titles st
JOIN software s ON s.title_id = st.id
JOIN host_software hs ON hs.software_id = s.id
WHERE st.id = ?
GROUP BY st.id
`
var title fleet.SoftwareTitle
if err := sqlx.GetContext(ctx, ds.reader(ctx), &title, selectSoftwareTitleStmt, id); err != nil {
if err == sql.ErrNoRows {
return nil, notFound("SoftwareTitle").WithID(id)
}
return nil, ctxerr.Wrap(ctx, err, "get software title")
}
selectSoftwareVersionsStmt, args, err := selectSoftwareVersionsSQL([]uint{id}, 0, true)
if err != nil {
return nil, ctxerr.Wrap(ctx, err, "building versions statement")
}
if err := sqlx.SelectContext(ctx, ds.reader(ctx), &title.Versions, selectSoftwareVersionsStmt, args...); err != nil {
return nil, ctxerr.Wrap(ctx, err, "get software title version")
}
return &title, nil
}
func (ds *Datastore) ListSoftwareTitles(
ctx context.Context,
opt fleet.SoftwareTitleListOptions,
) ([]fleet.SoftwareTitle, int, *fleet.PaginationMetadata, error) {
dbReader := ds.reader(ctx)
getTitlesStmt, args := selectSoftwareTitlesSQL(opt)
// build the count statement before adding the pagination constraints to `getTitlesStmt`
getTitlesCountStmt := fmt.Sprintf(`SELECT COUNT(DISTINCT s.id) FROM (%s) AS s`, getTitlesStmt)
// grab titles that match the list options
var titles []fleet.SoftwareTitle
getTitlesStmt, args = appendListOptionsWithCursorToSQL(getTitlesStmt, args, &opt.ListOptions)
if err := sqlx.SelectContext(ctx, dbReader, &titles, getTitlesStmt, args...); err != nil {
return nil, 0, nil, ctxerr.Wrap(ctx, err, "select software titles")
}
// perform a second query to grab the counts
var counts int
if err := sqlx.GetContext(ctx, dbReader, &counts, getTitlesCountStmt, args...); err != nil {
return nil, 0, nil, ctxerr.Wrap(ctx, err, "get software titles count")
}
// if we don't have any matching titles, there's no point trying to
// find matching versions. Early return
if len(titles) == 0 {
return titles, counts, &fleet.PaginationMetadata{}, nil
}
// grab all the IDs to find matching versions below
titleIDs := make([]uint, len(titles))
// build an index to quickly access a title by it's ID
titleIndex := make(map[uint]int, len(titles))
for i, title := range titles {
titleIDs[i] = title.ID
titleIndex[title.ID] = i
}
// we grab matching versions separately and build the desired object in
// the application logic. This is because we need to support MySQL 5.7
// and there's no good way to do an aggregation that builds a structure
// (like a JSON) object for nested arrays.
var teamID uint
if opt.TeamID != nil {
teamID = *opt.TeamID
}
getVersionsStmt, args, err := selectSoftwareVersionsSQL(titleIDs, teamID, false)
if err != nil {
return nil, 0, nil, ctxerr.Wrap(ctx, err, "build get versions stmt")
}
var versions []fleet.SoftwareVersion
if err := sqlx.SelectContext(ctx, dbReader, &versions, getVersionsStmt, args...); err != nil {
return nil, 0, nil, ctxerr.Wrap(ctx, err, "get software versions")
}
// append matching versions to titles
for _, version := range versions {
if i, ok := titleIndex[version.TitleID]; ok {
titles[i].Versions = append(titles[i].Versions, version)
}
}
var metaData *fleet.PaginationMetadata
if opt.ListOptions.IncludeMetadata {
metaData = &fleet.PaginationMetadata{HasPreviousResults: opt.ListOptions.Page > 0}
if len(titles) > int(opt.ListOptions.PerPage) {
metaData.HasNextResults = true
titles = titles[:len(titles)-1]
}
}
return titles, counts, metaData, nil
}
func selectSoftwareTitlesSQL(opt fleet.SoftwareTitleListOptions) (string, []any) {
stmt := `
SELECT
st.id,
st.name,
st.source,
COUNT(DISTINCT hs.host_id) AS hosts_count,
COUNT(DISTINCT s.id) AS versions_count
FROM software_titles st
JOIN software s ON s.title_id = st.id
JOIN host_software hs ON hs.software_id = s.id
-- placeholder for changing the JOIN type to filter vulnerable software
%s JOIN software_cve scve ON s.id = scve.software_id
-- placeholder for potential JOIN on hosts
%s
-- placeholder for WHERE clause
WHERE %s
GROUP BY st.id`
cveJoinType := "LEFT"
if opt.VulnerableOnly {
cveJoinType = "INNER"
}
var args []any
hostsJoin := ""
whereClause := "TRUE"
if opt.TeamID != nil {
hostsJoin = "JOIN hosts h ON h.id = hs.host_id"
whereClause = "h.team_id = ?"
args = append(args, opt.TeamID)
}
if match := opt.ListOptions.MatchQuery; match != "" {
whereClause += " AND (st.name LIKE ? OR scve.cve LIKE ?)"
match = likePattern(match)
args = append(args, match, match)
}
stmt = fmt.Sprintf(stmt, cveJoinType, hostsJoin, whereClause)
return stmt, args
}
func selectSoftwareVersionsSQL(titleIDs []uint, teamID uint, withCounts bool) (string, []any, error) {
selectVersionsStmt := `
SELECT
st.id as title_id,
s.id, s.version,
%s -- placeholder for optional host_counts
CONCAT('[', GROUP_CONCAT(JSON_QUOTE(scve.cve) SEPARATOR ','), ']') as vulnerabilities
FROM software_titles st
JOIN software s ON s.title_id = st.id
LEFT JOIN host_software hs ON hs.software_id = s.id
LEFT JOIN software_cve scve ON s.id = scve.software_id
%s -- placeholder for optional JOIN ON host_counts
WHERE st.id IN (?)
GROUP BY s.id`
var args []any
extraSelect := ""
extraJoin := ""
if withCounts {
args = append(args, teamID)
extraSelect = "MAX(shc.hosts_count) AS hosts_count,"
extraJoin = `
JOIN software_host_counts shc
ON shc.software_id = s.id
AND shc.hosts_count > 0
AND shc.team_id = ?
`
}
args = append(args, titleIDs)
selectVersionsStmt = fmt.Sprintf(selectVersionsStmt, extraSelect, extraJoin)
selectVersionsStmt, args, err := sqlx.In(selectVersionsStmt, args...)
if err != nil {
return "", nil, fmt.Errorf("bulding sqlx.In query: %w", err)
}
return selectVersionsStmt, args, nil
}

5
server/fleet/agent_options.go
View file

@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"strings"
)
@ -59,6 +60,10 @@ func ValidateJSONAgentOptions(ctx context.Context, ds Datastore, rawJSON json.Ra
for platform, platformOpts := range opts.Overrides.Platforms {
if len(platformOpts) > 0 {
if string(platformOpts) == "null" {
return errors.New("platforms cannot be null. To remove platform overrides omit overrides from agent options.")
}
if err := validateJSONAgentOptionsSet(platformOpts); err != nil {
return fmt.Errorf("%s platform config: %w", platform, err)
}

6
server/fleet/agent_options_test.go
View file

@ -27,6 +27,12 @@ func TestValidateAgentOptions(t *testing.T) {
}
}}`, `unknown field "foo"`},
{"overrides.platform is null", `{"overrides": {
"platforms": {
"darwin": null
}
}}`, `platforms cannot be null. To remove platform overrides omit overrides from agent options.`},
{"extra top-level bytes", `{}true`, `extra bytes`},
{"extra config bytes", `{"config":{}true}`, `invalid character 't' after object`},
{"extra overrides bytes", `{"overrides":{}true}`, `invalid character 't' after object`},

Some files were not shown because too many files have changed in this diff Show more