fleet/changes
Lucas Manuel Rodriguez 9142c5de79
Prevent thundering herd when applying large number of policies on large number of hosts (#13552)
#13527

(Adding @mna to double check the changes in the async implementation of
policy result storage)

This PR also adds the osquery-perf changes needed to define the count of
macOS and Windows hosts.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~

Test with 80k hosts: 70k simulated macOS, 10k simulated Windows.
Apply Windows policies first, then apply macOS policies:
```
fleetctl apply -f ee/cis/win-10/cis-policy-queries.yml

# Leave running for some time

fleetctl apply -f ee/cis/macos-13/cis-policy-queries.yml
```

After applying CIS policies previous to these changes:
![Screenshot 2023-08-23 at 11 36
18](https://github.com/fleetdm/fleet/assets/2073526/72c1dc7d-e601-4248-be35-93c85b749f5d)

After applying these changes and applying the same policies:
![Screenshot 2023-08-28 at 15 42
57](https://github.com/fleetdm/fleet/assets/2073526/6b6d76b8-6acb-4893-a913-bf603a68f1a4)
2023-08-31 10:58:50 -03:00
..
.keep Issue 1009 calculate diff software (#1305) 2021-07-08 13:57:43 -03:00
12243-policy-resolution-urls Fleet UI: Convert URLs in Policy resolution text to be clickable links (#13023) 2023-08-18 09:15:23 -04:00
12476-ui-export-shows-all-columns Fleet UI: Live query UI and export results tables include all columns returned (#13428) 2023-08-29 08:51:37 -04:00
12550-add-validation-for-policy-specs Add validation for policy specs (#13294) 2023-08-21 10:22:07 -06:00
13064-highlighter-bug Fleet UI: Improvements to highlighting while tabbing and not clicking (#13508) 2023-08-25 12:26:03 -04:00
13071-disable-fleetd-enroll-errors environment variable to disable orbit enroll logs (#13519) 2023-08-25 15:25:07 -06:00
13082-fix-dashboard-double-scrollbar Fleet UI: [released tiny bug] Fix dashboard double scroll bar (#13309) 2023-08-18 09:15:44 -04:00
13158-nag-and-copy adjust MDM migration copy and timers (#13366) 2023-08-18 18:58:40 -03:00
13158-update-macadmins-tables update macadmins/osquery-extension to v0.0.15 (#13371) 2023-08-18 12:38:49 -03:00
13205-browser-page-titles-added Fleet UI: Add more specific page titles to browser tab (#13335) 2023-08-30 16:11:49 -07:00
13205-table-links-can-open-in-new-tab Fleet UI: [feature] All table links can open in a new tab (#13349) 2023-08-22 09:20:40 -04:00
13238-client-debug-error allow clients to report errors back to the server (#13478) 2023-08-24 13:04:27 -03:00
13345-fix-queries-nav-bar-bug Fleet UI: Link improvements (Back links and dashboard tiles right clickable, fix queries link on nav bar) (#13361) 2023-08-23 08:56:32 -04:00
13363-match-dep-profiles automatically set DEP profile for teams created by Puppet (#13496) 2023-08-28 11:36:00 -03:00
13367-custom-query-frequencies Fleet UI: Fix custom query frequency on manage automations modal (#13511) 2023-08-25 13:00:14 -04:00
13431-sso-idp-store store email used for authentication during MDM SSO (#13480) 2023-08-23 18:23:26 -03:00
13433-host-query-optimization 13433 host query optimization (#13451) 2023-08-23 10:34:55 -06:00
13434-paginate-policies paginate the policies API (#13459) 2023-08-30 16:30:17 -06:00
13495-host-details-rbac Fleet UI: Store host's team on details page for correct RBAC dropdown (#13530) 2023-08-28 13:11:19 -04:00
13498-fix-sso-layout fix MDM SSO layout (#13578) 2023-08-30 14:16:08 -03:00
13521-label-alignment Fleet UI: [tiny released styling bug] Fix alignment on long label names (#13568) 2023-08-29 14:47:26 -04:00
13527-applying-policies-at-scale Prevent thundering herd when applying large number of policies on large number of hosts (#13552) 2023-08-31 10:58:50 -03:00
add-jumpcloud-to-mdm-list Add JumpCloud to the list of well-known MDM solutions (#13554) 2023-08-29 10:41:18 -03:00
fix-broken-win-10-cis-queries Add tooling for load testing Windows CIS policies and fix typos in policy queries (#13384) 2023-08-18 17:32:22 -03:00
fix-duplicate-cis-policy-queries-yml Fix Windows CIS policies issues found by new fleetctl apply checks (#13460) 2023-08-23 07:42:41 -03:00
issue-2943-update-nanomdm Update nanomdm dependency (#12721) 2023-08-21 11:07:57 -03:00
issue-12847-change-win-mdm-auth use OrbitNodeKey for windows mdm enrollment authentication instead of HostUUID (#13503) 2023-08-29 14:50:13 +01:00
issue-13305-api-run-script Add orbit notification and API endpoints to send/receive scripts to hosts (part 2 of ticket) (#13447) 2023-08-23 16:47:47 -04:00
issue-13307-run-script-on-agent Implement script execution on the fleetd agent (disabled by default) (#13569) 2023-08-30 14:02:44 -04:00
issue-13308-implement-script-run-activity-and-details-ui Feat UI view script activity and script details (#13388) 2023-08-29 11:47:37 +01:00