mirror of
https://github.com/fleetdm/fleet
synced 2026-04-21 21:47:20 +00:00
aff440236e
**Related issue:** Ref #34797 Ref #42675 ## Problem When a software installer spec has no `hash_sha256`, Fleet re-downloads the package, re-extracts metadata, and re-upserts the DB on every GitOps run, even if the upstream file hasn't changed. For deployments with 50+ URL-only packages across multiple teams, this wastes bandwidth and processing time on every run. ## Solution By default, use etags to avoid unnecessary downloads: 1. First run: Fleet downloads the package normally and stores the server's ETag header 2. Subsequent runs: Fleet sends a conditional GET with `If-None-Match`. If the server returns 304 Not Modified, Fleet skips the download, metadata extraction, S3 upload, and DB upsert entirely Opt-out with `always_download:true`, meaning packages continue to be downloaded and re-processed on every run, same as today. No UI changes needed. ```yaml url: https://nvidia.gpcloudservice.com/global-protect/getmsi.esp?version=64&platform=windows always_download: true install_script: path: install.ps1 ``` ### Why conditional GET instead of HEAD Fleet team [analysis of 276 maintained apps](https://github.com/fleetdm/fleet/pull/42216#issuecomment-4105430061) showed 7 apps where HEAD requests fail (405, 403, timeout) but GET works for all. Conditional GET eliminates that failure class: if the server doesn't support conditional requests, it returns 200 with the full body, same as today. ### Why opt-in 5 of 276 apps (1.8%) have stale ETags (content changes but ETag stays the same), caused by CDN caching artifacts (CloudFront, Cloudflare, nginx inode-based ETags). The `cache` key lets users opt in per package for URLs where they've verified ETag behavior is correct. Validation rejects `always_download: true` when hash_sha256` is set ## Changes - New YAML field: `cache` (bool, package-level) - New migration: `http_etag` VARCHAR(512) column (explicit `utf8mb4_unicode_ci` collation) + composite index `(global_or_team_id, url(255))` on `software_installers` - New datastore method: `GetInstallerByTeamAndURL` - `downloadURLFn` accepts optional `If-None-Match` header, returns 304 as `(resp, nil, nil)` with `http.NoBody` - ETag validated per RFC 7232 (ASCII printable only, no control chars, max 512 bytes) at both write and read time - Cache skipped for `.ipa` packages (multi-platform extraInstallers) - TempFileReader and HTTP response leak prevention on download retry - Docs updated in `yaml-files.md` ## What doesn't change - Packages with `hash_sha256`: existing hash-based skip, untouched - FMA packages: FMA version cache, untouched - Packages with `always_download: true`: identical to current behavior - Fleet UI: no changes ## Test plan Automated testing: - [x] 16 unit tests for `validETag` - [x] 8 unit tests for conditional GET behavior (304, 200, 403, 500, weak ETag, S3 multipart, no ETag) - [x] MySQL integration test for `GetInstallerByTeamAndURL` - [x] All 23 existing `TestSoftwareInstallers` datastore tests pass - [x] All existing service tests pass Manual testing: - [x] E2E: 86 packages across 6 CDN patterns, second apply shows 51 conditional hits (304) - [x] @sgress454 used a local fileserver tool to test w/ a new instance and dummy packages <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * ETag-based conditional downloads to skip unchanged remote installer files. * New always_download flag to force full re-downloads. * **Tests** * Added integration and unit tests covering conditional GETs, ETag validation, retries, edge cases, and payload behavior. * **Chores** * Persist HTTP ETag and related metadata; DB migration and index to speed installer lookups. * Added installer lookup by team+URL to support conditional download flow. * **Bug Fix** * Rejects using always_download together with an explicit SHA256 in uploads. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Scott Gress <scott@fleetdm.com> Co-authored-by: Scott Gress <scott@pigandcow.com> Co-authored-by: Ian Littman <iansltx@gmail.com> |
||
|---|---|---|
| .. | ||
| .keep | ||
| 14827-prevent-TOCTOU-last-admin | ||
| 29657-custom-settings-configuration-profiles | ||
| 31289-acme-for-mdm-protocol | ||
| 32126-macos-fleetd-reinstall | ||
| 32662-include-correct-cpe | ||
| 32773-preview-windows-mdm | ||
| 33106-fix-generate-gitops-vpp | ||
| 33418-windows-mdm-profile-deletion | ||
| 34288-setup-experience-cancel-activity | ||
| 34433-speedup-macos-profile-delivery | ||
| 34667-scim-user-host-emails-association | ||
| 34950-nano-tables-cleanup | ||
| 35067-windows-pro-missing-vulnerabilities | ||
| 35467-detail-query-config-preload | ||
| 35484-improve-policy_membership-contention | ||
| 36312-trim-spaces-from-fleets-names | ||
| 36643-fix-back-button-policies-reports | ||
| 36751-add-fmas-to-policy-automation | ||
| 36799-macos-disk-space-purgeable | ||
| 37323-jetbrains-cve | ||
| 37546-android-certificate-install-activity | ||
| 37556-resend-android-certs | ||
| 38002-throttle-ca-certificate-profiles | ||
| 38036-gitops-ca-delete-order | ||
| 38041-entra-windows-conditional-access | ||
| 38647-remove-duplicate-disk-encryption-activity | ||
| 38785-windows-setup-experience-cancel | ||
| 38793-python-scripts | ||
| 38929-reports-tab | ||
| 38988-fma-pin-major-version | ||
| 39066-vpp-timeout-install-details | ||
| 39082-setup-logo-light-background | ||
| 39190-display-sw-version-filter | ||
| 39308-team-ca-read-access | ||
| 39316-winoffice-vulnerability-detection | ||
| 39570-clear-passcode | ||
| 39842-generate-gitops-bug | ||
| 39899-deterministic-cpe-matching | ||
| 39968-sso-validity-increase-default | ||
| 40015-activate-deprecation-warnings | ||
| 40050-server-core-msrc-differentiation | ||
| 40057-osv-vulns | ||
| 40117-fix-sql-table-alias-platform-detection | ||
| 40137-update-default-fleet | ||
| 40177-config-profile-name-status | ||
| 40581-os-versions-vuln-details | ||
| 40702-simplif-os-modal | ||
| 40715-allow-whitespace-end-users-form | ||
| 40751-google-drive-brew-version | ||
| 40785-fix-gitops-vpp-token-assignment | ||
| 40841-gitops-sw-upload-error | ||
| 40910-correct-request-certificate-pem | ||
| 40972-policy-description | ||
| 41167-skip-bootstrap-clear-on-cert-renewal | ||
| 41324-support-labels-include-all-for-installers | ||
| 41381-eua-ms-installer | ||
| 41409-use-fleetctl-new-templates-as-starter-lib | ||
| 41484-fix-windows-mdm-profile-upload-panic | ||
| 41500-validate-scripts | ||
| 41534-host-details-reports-api-end-point | ||
| 41540-host-details-reports-db-optimizations | ||
| 41542-android-cert-resend-backend | ||
| 41586-admin-by-request-false-positive | ||
| 41601-use-multiplatform-names-in-front-end | ||
| 41603-fix-query-responses | ||
| 41631-not-installed | ||
| 41636-typo-in-msrc-json | ||
| 41644-improve-cpe-matching | ||
| 41670-auto-rotate-recovery-lock | ||
| 41672-allow-omitting-manual-hosts-label | ||
| 41710-overwrite-software-title | ||
| 41741-order | ||
| 41742-fix-my-device-500-fleet-free | ||
| 41753-policy-details-page | ||
| 41778-fix-enqueue-setup-experience-items-for-arch-linux | ||
| 41815-override-patch-policy-query | ||
| 41888-otel-service-name | ||
| 42017-host-details-reports-tab | ||
| 42047-android-web-app-banner | ||
| 42103-accept-89-on-profile-removal | ||
| 42185-add-flatcar-coreos-linux-platforms | ||
| 42327-apple-profile-retries | ||
| 42383-android-display-name | ||
| 42399-support-vpp-policy-automations-in-generate-gitops | ||
| 42402-fix-query-results-deduplication | ||
| 42405-android-onc-after-cert | ||
| 42443-fix-show-disk-encryption-key-modal | ||
| 42572-fix-duplicate-text | ||
| 42600-android-cert-templates-cleared-on-reenroll | ||
| 42675-conditional-download-cache | ||
| 42751-r2-fma | ||
| 42799-option-to-unlock-not-available-afler-lock | ||
| 42808-rwmutex-jitter-shouldupdate | ||
| 42814-sso-learn-more-link | ||
| 42836-deduplicate-flipping-policies-queries | ||
| 42881-api-endpoints-initial-models | ||
| 42883-api-endpoint-for-listing-rest-api-endpoints | ||
| 42979-fix-for-multi-team-user | ||
| 42991-patch-policy-gitops-bug | ||
| 43034-optimize-policy-queries-for-host | ||
| 43114-pagination-bug | ||
| 43125-reports-nav-underline | ||
| 43162-software-table-bookmarkable-page | ||
| 43341-gitops-mode-wonkiness | ||
| 43342-improved-button-link-styling | ||
| 43389-patch-policy-gitops-bugs | ||
| add-api-time-on-gitops-errors | ||
| allow-clearing-windows-update-settings | ||
| bump-mysql-8.0.42 | ||
| docs-read-replica-config-clarity | ||
| input-field-to-ts | ||
| issue-40076-clear-enrolled-from-migration | ||
| issue-40809-bitlocker-loop | ||
| refactor-named-functions-nil-checks | ||
| ts6 | ||
| turn-off-prom-in-dev-mode | ||
| up-default-software-batch | ||
| update-go-1.26.1 | ||