fleet/server/service/async
Lucas Manuel Rodriguez 9142c5de79
Prevent thundering herd when applying large number of policies on large number of hosts (#13552)
#13527

(Adding @mna to double check the changes in the async implementation of
policy result storage)

This PR also adds the osquery-perf changes needed to define the count of
macOS and Windows hosts.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~

Test with 80k hosts: 70k simulated macOS, 10k simulated Windows.
Apply Windows policies first, then apply macOS policies:
```
fleetctl apply -f ee/cis/win-10/cis-policy-queries.yml

# Leave running for some time

fleetctl apply -f ee/cis/macos-13/cis-policy-queries.yml
```

After applying CIS policies previous to these changes:
![Screenshot 2023-08-23 at 11 36
18](https://github.com/fleetdm/fleet/assets/2073526/72c1dc7d-e601-4248-be35-93c85b749f5d)

After applying these changes and applying the same policies:
![Screenshot 2023-08-28 at 15 42
57](https://github.com/fleetdm/fleet/assets/2073526/6b6d76b8-6acb-4893-a913-bf603a68f1a4)
2023-08-31 10:58:50 -03:00
..
async.go Support async saving of scheduled query statistics (#7012) 2022-08-10 10:01:05 -04:00
async_bench_test.go Implement async processing of hosts for label queries (#2288) 2021-11-01 14:13:16 -04:00
async_host_seen.go Support per-task configuration for async host processing configuration (#5700) 2022-05-16 09:44:50 -04:00
async_host_seen_test.go Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
async_label.go Support per-task configuration for async host processing configuration (#5700) 2022-05-16 09:44:50 -04:00
async_label_test.go Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
async_policy.go Prevent thundering herd when applying large number of policies on large number of hosts (#13552) 2023-08-31 10:58:50 -03:00
async_policy_test.go Prevent thundering herd when applying large number of policies on large number of hosts (#13552) 2023-08-31 10:58:50 -03:00
async_scheduled_query_stats.go Combine Schedules and Queries: API changes (#12778) 2023-07-24 20:17:20 -04:00
async_scheduled_query_stats_test.go Combine Schedules and Queries: API changes (#12778) 2023-07-24 20:17:20 -04:00
async_test.go Prevent thundering herd when applying large number of policies on large number of hosts (#13552) 2023-08-31 10:58:50 -03:00
collect.go Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
collect_test.go Set authz checked when rate limiting device endpoints (#6702) 2022-07-18 14:22:49 -03:00
doc.go Support async saving of hosts' last seen time (#5640) 2022-05-10 11:29:17 -04:00