mirror of
https://github.com/fleetdm/fleet
synced 2026-05-24 09:28:54 +00:00
1cd37ef966
For #25555 # Checklist for submitter - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) This PR updates the `NewLabel` service to use the `UpdateLabelMembershipByHostIDs` method previously added by @jacobshandling rather than using `ApplyLabels`. The latter method has performance issues when adding large numbers of hosts at once to a manual label (see #25555) because it does an expensive lookup of host names before transforming those into Fleet host IDs. The new code skips the middleman and transforms host identifiers directly to Fleet host IDs, and does so using a batching strategy to ensure the queries don't get too large. This PR does update `UpdateLabelMembershipByHostIDs` slightly to return an updated Label object and host IDs array, as this is the expected return value for `NewLabel`. I update the method's tests accordingly. I don't think any new tests for `NewLabel` are needed as it should have the same functionality and return values. ## Manual Testing On the main branch, I launched my local MySQL with the thread stack size set to the minimal allowed, and used the API to try and create a new label with 5,000 hosts attached, and received a 422 response from the server. Server logs showed: ``` level=error ts=2025-01-28T15:08:20.465401Z component=http user=scott@fleetdm.com method=POST uri=/api/latest/fleet/labels took=16.610292ms err="get hostnames by identifiers: Error 1436 (HY000): Thread stack overrun: 111136 bytes used of a 131072 byte stack, and 20000 bytes needed. Use 'mysqld --thread_stack=#' to specify a bigger stack." ``` On this branch, I kept the same MySQL settings and tried my API request again and it was successful: <img width="776" alt="image" src="https://github.com/user-attachments/assets/c4f0f52b-4d09-457b-8096-4dd3a747b1f4" /> ## QA The script I used to create a new manual label with lots of hosts is at: https://gist.github.com/sgress454/84f12064c437da456c456e25c26d9069 To run it, first grab a bearer token from any API request by opening the network tab, clicking a Fleet API request, and in the headers tab scrolling down to Authorization: <img width="892" alt="image" src="https://github.com/user-attachments/assets/5680f3bf-8db8-469a-9f03-000b86622c04" /> (only take the part _after_ "Bearer") Then download the script from that gist and in its folder run: ``` NODE_TLS_REJECT_UNAUTHORIZED=0 node ./add_hosts_to_label.js <the bearer token> "<a label name>" ``` e.g. ``` NODE_TLS_REJECT_UNAUTHORIZED=0 node ./add_hosts_to_label.js U3HpbdtadmJXGKYSB0U/PbwfOpHbBt7FpkWmGKKYolOO1moLNZA6XxP+QO5LVukvAotZ7d+JbNUEEhYHZtxoqg== "some test label" ``` This will invoke the API on https://localhost:8080 and try to add 5000 hosts a new label "some test label". If you need to change the # of hosts or the url of the server, there are additional arguments: ``` NODE_TLS_REJECT_UNAUTHORIZED=0 node ./add_hosts_to_label.js <the bearer token> "<a label name>" <number of hosts> <url> ``` e.g. ``` NODE_TLS_REJECT_UNAUTHORIZED=0 node ./add_hosts_to_label.js U3HpbdtadmJXGKYSB0U/PbwfOpHbBt7FpkWmGKKYolOO1moLNZA6XxP+QO5LVukvAotZ7d+JbNUEEhYHZtxoqg== "some test label" 10000 https://foo.bar ``` |
||
|---|---|---|
| .. | ||
| .keep | ||
| 16865-increase-statistics-frequency | ||
| 21827-edit-vpp-teams | ||
| 22353-abm-hosts-upcoming-activities | ||
| 22364-vuln-cron | ||
| 22464-list-hosts-populate-users-labels | ||
| 22544-move-linux-lock-wipe | ||
| 22919-semver-util | ||
| 23096-fma-errors | ||
| 23115-vpp-policy | ||
| 23116-fma-dl-url | ||
| 23241-lock-api-response | ||
| 23302-fma-click-bug | ||
| 23312-update-policies-empty-state | ||
| 23512-clarify-expected-behavior-of-host-counts | ||
| 23528-install-software-policy-filter | ||
| 23770-fleetctl-linux-arm | ||
| 23811-empty-cell-for-no-uuid | ||
| 23823-cloudfront-cdn | ||
| 23924-handle-long-team-names | ||
| 23971-persist-hosts-column-settings-across-sessions | ||
| 24035-team-agent-options-ui-resets | ||
| 24038-agent-options-key-error | ||
| 24148-re-install-fma | ||
| 24335-dropdown-styling-bug | ||
| 24366-success-email-message | ||
| 24418-bad-links | ||
| 24421-fix-my-device-paths | ||
| 24486-error-for-invalid-invites | ||
| 24602-editable-scripts | ||
| 24618-make-email-logo-dark-mode-compatible | ||
| 24629-ui-os-updates-table | ||
| 24653-live-query-from-edit-affects-performance-stats | ||
| 24660-team-admins-cant-set-sso-mfa | ||
| 24720-msi-large-interned-strings | ||
| 24732-gzip | ||
| 24754-require-pw-for-pw-auth | ||
| 24790-admx-policies | ||
| 24795-host-count | ||
| 24804-deleted-profiles | ||
| 24816-fix-double-mdm-commands | ||
| 24873-pkg-name | ||
| 24876-dashboard-cards | ||
| 24948-display-api-errors-in-user-form | ||
| 24958-gitops-webhooks-disable | ||
| 24959-ui-my-device-software-filter | ||
| 24962-ui-dashboard-mdm-solutions-table | ||
| 25004-fleetctl-packge-cli-instructions | ||
| 25009-smtp-page-validation | ||
| 25015-user-page-responsive | ||
| 25072-25073-software-name-overflow | ||
| 25075-false-positive | ||
| 25114-include-team-queries-in-host-details-query-modal | ||
| 25130-iterm-false-neg | ||
| 25144-uninstall-after-mdm-action | ||
| 25160-optimize-software-during-enrollment | ||
| 25194-vpp-app-clear | ||
| 25201-unknown-installer-version | ||
| 25241-smtp-helo-domain | ||
| 25244-batch-set-declarations | ||
| 25251-url-fleet-app-response | ||
| 25257-dropdown-improvements | ||
| 25261-identical-hostnames-label-membership | ||
| 25264-sso-form-validation | ||
| 25265-boostrap-package-not-found | ||
| 25307-fleetctl-package-link | ||
| 25318-update-sso-settings-error-states | ||
| 25346-fix-manage-automations-link-on-dash | ||
| 25366-manage-automation-dropdown-styling | ||
| 25427-allow-excluding-fma-from-software-titles | ||
| 25555-batch-hostnames-on-new-label | ||
| 25567-renew-vpp | ||
| 25581-session-id | ||
| 25590-node | ||
| 25597-false-positives | ||
| 25609-archive-encryption-keys | ||
| 25615-windows-mdm-profiles | ||
| 25640-fix-idp-source | ||
| 25748-remove-fleetctl-from-fleetdm-fleet-docker-image | ||
| 25759-illegal-argument-errors | ||
| issue-21691-windows-disk-encryption-dont-resend | ||
| issue-23912-ui-for-activities | ||
| issue-24824-tooltip-verified-verifying | ||
| issue-24901-fixes-error-cutoff | ||
| issue-24992-padding-fixes-around-lists | ||
| issue-25507-upgrade-github-cache-action | ||
| issue-25735-fix-500-vulnerable-host-software | ||
| osquery-constant-prerelease | ||