Commit graph

3520 commits

Author SHA1 Message Date
Luke Heath
0056d36d81
Adding changes for Fleet v4.76.0 (#34486) (#35380) 2025-11-07 19:19:12 -06:00
jacobshandling
acb563337e
Ingest, store, consider in unique_identifier, and serve upgrade_codes for Windows software (#34786)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33907 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
~- [ ] Confirmed that updating the timestamps is acceptable, and will
not cause unwanted side effects.~ N/A
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Windows software inventory now includes upgrade code data for better
software identification and tracking.

* **Chores**
* Database schema updated to support upgrade code storage for software
titles and inventory records.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-07 15:33:31 -08:00
Carlo
715d963f82
My device page (self-service) for iOS/iPadOS (#35238)
Implements #32247. This is the complete feature branch, consolidating:

- https://github.com/fleetdm/fleet/pull/35018
- https://github.com/fleetdm/fleet/pull/34758
- https://github.com/fleetdm/fleet/pull/35009
- https://github.com/fleetdm/fleet/pull/35181
- https://github.com/fleetdm/fleet/pull/35342

---------

Co-authored-by: Jonathan Katz <44128041+jkatz01@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: jkatz01 <yehonatankatz@gmail.com>
2025-11-07 17:30:51 -05:00
Victor Lyuboslavsky
5cfc28ae5a
Okta IdP factor (#35143)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34544 

Demo video: https://www.youtube.com/watch?v=VzOkISWmEKw
[Original research
doc](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/research/orchestration/okta-conditional-access.md)
[Victor's POC
branch](https://github.com/fleetdm/fleet/tree/victor/33165-okta-conditional-access-poc)

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Conditional Access IdP integration added (IdP metadata & SSO) with
device-health aware session checks.
  * Endpoint to download the IdP signing certificate (PEM) added.
* Automatic revocation of old conditional access certificates with a
configurable grace period.

* **Tests**
* Extensive tests for certificate rotation, lifecycle, SSO flows, URL
construction, and IdP metadata.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-07 16:19:25 -06:00
RachelElysia
3efeeb1ad0
Fleet UI: Ability to edit software display names (#34872) 2025-11-07 09:59:30 -05:00
Ian Littman
6b638189d4
Don't make extra queries for unused data when grabbing team config for conditional access (#35337)
For #35333.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
2025-11-06 22:39:10 -06:00
RachelElysia
f12bedece7
Fleet UI: Fix table selection styling (#35295) 2025-11-06 15:31:14 -05:00
Matt Hatcher
369f9070c3
Add InstallAnywhere self extracting archive to metadata extraction (#34874)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34827

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [ ] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
2025-11-06 14:25:07 -05:00
Juan Fernandez
666c223456
Make styling more consistent in edit policies view for Firefox. (#35247)
**Related issue:** Resolves
[#34999](https://github.com/fleetdm/fleet/issues/34999)

- Align edit icon to start of line.
- Fix font size for policy name and contents elements.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [X] QA'd all new/changed functionality manually
2025-11-06 13:00:49 -06:00
Ian Littman
724860bd6e
Move end user authentication to a tab under Integrations > SSO (#35084)
Resolves #34525.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually
2025-11-05 14:34:59 -06:00
Gabriel Hernandez
293a3be219
Fix DB errors when running the apple_mdm_iphone_ipad_refetcher cron job (#35047)
**Related issue:** Resolves #33436

This attempts to make the query to get the list of iOS and iPad devices
that need a refetch more resilient by ensuring the hosts are in the
nano_enrolled table and are currently enrolled.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [ ] Added/updated automated tests
2025-11-05 13:39:41 +00:00
jacobshandling
50e7947b67
Update Add, Edit, and Delete Certificate Authority modals to support Custom EST (#35085)
**Related issue:** Resolves #34276 

<img width="1241" height="924" alt="Screenshot 2025-10-31 at 5 21 57 PM"
src="https://github.com/user-attachments/assets/44d94842-c4d0-4770-9072-6a87da2ae6cb"
/>


![ezgif-6f70f761e3ad5b](https://github.com/user-attachments/assets/606a4696-7fc2-409f-a047-6436f1916899)


- [x] Changes file added for user-visible changes in `changes/`
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-11-04 15:06:07 -08:00
Jordan Montgomery
ac69cb7bcc
reduce android device reconciler frequency (#35121)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #35117 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.


## Testing

- [x] QA'd all new/changed functionality manually
2025-11-04 14:58:31 -05:00
Juan Fernandez
82fe6c8560
Better error messages when using built-in labels (#34739)
Resolves #32776 

Show better error messages when using built-in labels.
2025-11-04 12:52:54 -04:00
Sarah Gillespie
475614f19d
Update manual MDM enroll flow for macOS to use OTA with SSO (#34650) 2025-11-04 08:24:50 -06:00
Tim Lee
81831fdd35
SCIM user reconcile migration (#35017) 2025-11-03 16:00:20 -07:00
Jonathan Katz
59a73b1e47
32084 Cisco Secure Client installer fix (#35077)
**Related issue:** Resolves #32084 
This PR modifies `isValidAppFilePath` to allow subdirectors in
`Applications/`, like in this case `Applications/Cisco/Cisco Secure
Client.app`.
This also changes the metadata extraction from packageinfo to trim
`.app` from the name in all cases.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

### Test plan:
---
I ran this on my local environment and it seemed fine
- Have environment with the bug recreated, it has two software titles
for "Cisco Secure Client", and the one with the bundle id
`com.cisco.pkg.anyconnect.vpn` is used by the installer.
- URL to pkg:
https://fndtnfleetmsp.blob.core.windows.net/fndtnpkgs/cisco-secure-client-macos-5.1.3.62-core-vpn-webdeploy-k9.pkg
- Cisco Secure Client doesn't show as installed in UI even after
installing.
- Run the new migration.
- Cisco Secure Client shows as installed now in ui, software title with
bundle id `com.cisco.pkg.anyconnect.vpn` is gone from the database, and
the software installer references the correct title
(`com.cisco.secureclient.gui`).
- Check that deleting and reuploading the installer doesn't recreate the
bad software title.

### QA Note:
---
There are some problems with the install script, but that is probably a
different scope than this ticket.
`Reinstall` wont work, it says Cisco Secure Client is already installed.
Uninstalling through Fleet then Installing again works fine though.
2025-11-03 15:22:39 -05:00
Victor Lyuboslavsky
ba5f02f9ca
os_versions endpoint performance improvements (#34897)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34500 and Resolves #33758

Video demo: https://www.youtube.com/watch?v=4HZlKG0G1B0

- Added a new aggregation table
`operating_system_version_vulnerabilities` for faster queries. The table
is currently used only for Linux vulnerabilities, but could be used for
other OS vulnerabilities.
- Added `max_vulnerabilities` parameter per [API
doc](https://github.com/fleetdm/fleet/pull/33533)
- Also added `max_vulnerabilities` parameter to `os_versions/{id}`
endpoint, but not making it public since that endpoint is still slow and
needs other API changes. bug #34974
- Removed `"kernels": []` from `os_versions` endpoint result

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added ability to limit the number of vulnerabilities displayed for
operating system versions via an optional parameter.
* Introduced vulnerability count tracking for operating system versions,
now visible in API responses and UI displays.
* Enhanced operating system vulnerability visualization with improved
count-based rendering.

* **Tests**
* Added comprehensive test coverage for vulnerability limiting behavior
across multiple operating system versions and architectures.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-03 13:07:44 -06:00
Victor Lyuboslavsky
072ee68eda
Updating to Go 1.25.3 (#35082) 2025-11-03 09:47:07 -06:00
Zach Wasserman
0cdde239b9
Add activity feed entries for host deletion and expiration (#34720)
**Related issue:** Resolves #33513 

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-31 09:37:31 -07:00
Ian Littman
7f5652daff
Remove previews, add preview links, make copy tweaks to setup experience configuration UI (#34980)
Fixes #34530 and #34452. idP config is in a subsequent commit and will
handle the dangling path added here.

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually
2025-10-30 17:32:06 -05:00
Matt Hatcher
c4a6c9110b
remove premium check for OS settings (#34808)
**Related issue:** Resolves #34801

a quick fix to remove the premium check for os settings display on the
host details page. This feature does not require premium so we do not
need this check.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] QA'd all new/changed functionality manually
2025-10-30 10:34:32 +00:00
Martin Angers
b776398e67
Cherrypick of bugfix for 4.76.0: make software title status counts consistent (#34944)
Cherry-pick PR into `main` from
https://github.com/fleetdm/fleet/pull/34932 that targeted 4.76.0
2025-10-29 14:56:56 -04:00
Ian Littman
b4aae8976c
Reflect current reality in dark background logo tooltip copy (#34906)
Fixes #34621.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually
2025-10-29 09:44:41 -05:00
jacobshandling
3aad722924
(releases on merge to main) Fix vuln false positives for "Logi Bolt.app" (#33920)
**_QA on-branch before merge_**
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #31082 

Before:
<img width="1258" height="989" alt="Screenshot 2025-10-06 at 9 31 46 PM"
src="https://github.com/user-attachments/assets/b027b3b6-6201-468d-9141-76b80daa35c8"
/>

After
<img width="1258" height="989" alt="Screenshot 2025-10-06 at 9 26 03 PM"
src="https://github.com/user-attachments/assets/147bdd41-5ebc-4d23-bd85-c1cb963a429d"
/>


- [x] Changes file added for user-visible changes in `changes/`
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-27 16:55:30 -07:00
Ian Littman
7502880869
Allow query editing in GitOps Mode for policies, add ▶ icon to live query "Run" button (#34838)
Fixes #34086.

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually
2025-10-27 18:52:17 -05:00
jacobshandling
cb2d42de78
(releases on merge to main) Fix vuln false positives for vscode golang extension (#33839)
**_QA on-branch before merge_**

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
## Resolves #33235 

<img width="2556" height="1419" alt="Screenshot 2025-10-03 at 5 55
40 PM"
src="https://github.com/user-attachments/assets/49078de7-699a-4a64-86ab-f435065f91ed"
/>



- [x] Changes file added for user-visible changes in `changes/`
2025-10-27 14:48:29 -07:00
jacobshandling
ab0065ab39
(releases on merge to main) Detect JetBrains IDE plugin vulnerabilities (#34331)
**Related issue:** Resolves #32266


[Demo](https://drive.google.com/file/d/1ZDYJkWkxZ519le8v9qGmcrL8YkP-uivW/view?usp=sharing)

- [x] Changes file added for user-visible changes in `changes/`,
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-24 12:06:35 -07:00
Tim Lee
c5d7c9f626
31970 NPM vuln support (#33100) 2025-10-24 12:54:57 -06:00
Jordan Montgomery
79b886455a
Update host expiry logic to account for Apple MDM checkin times (#34698)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32499

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved host expiry logic to correctly identify and preserve Apple
MDM-enrolled hosts that don't check in through Orbit, preventing
unintended host deletions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-24 13:12:03 -04:00
Juan Fernandez
758bc75f85
Added missing Batch Scripts tab counts (#34638)
Resolves #33393

Added status counts to batch script detail page tabs.
2025-10-24 11:54:47 -04:00
Scott Gress
1f54c42f05
Update tooltip about setup experience software ordering (#34688)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33790

# Details

This is just the tooltip update; the code to update the order of
software installs is was released as part of #34173 (see
https://github.com/fleetdm/fleet/pull/34173/files#diff-c5babdad542a72acf2ec2ecb7cb43967fc53850b6998ac629e253336b87e008b)

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [X] QA'd all new/changed functionality manually

<img width="560" height="321" alt="image"
src="https://github.com/user-attachments/assets/1bde02ca-e180-49e1-92a7-e197305dd8ee"
/>
2025-10-24 09:30:59 -05:00
jacobshandling
0d6f4a69c9
Fix batch details page copy gap (#34699)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34697 

<img width="819" height="283" alt="Screenshot 2025-10-23 at 9 58 57 AM"
src="https://github.com/user-attachments/assets/d30bd017-da75-4752-b562-fd4c1fb51db0"
/>


- [x] Changes file added for user-visible changes in `changes/`
- [x] QA'd all new/changed functionality manually
2025-10-23 10:20:09 -07:00
Magnus Jensen
d6a23a79ee
Modify Windows replacement code to allow Custom SCEP variables (#34633)
and refactor to share with apple mdm

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34246 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
2025-10-22 15:46:48 -03:00
Jordan Montgomery
7593d102fb
Experimental fleet server config for custom updates & disk encryption settings (#34598)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33316

Merges in changes made in this community PR:
https://github.com/fleetdm/fleet/pull/33665

Adds support for Windows and tests, also blocks the feature on fleet
free

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

---------

Co-authored-by: Wesley Whetstone <wesw@stripe.com>
Co-authored-by: Wesley Whetstone <jckwhet@gmail.com>
2025-10-22 13:51:10 -04:00
Sarah Gillespie
97cf97ca3e
Adjust UI section headers and layout of Settings > Integrations (#34585) 2025-10-21 15:28:47 -05:00
Tim Lee
d4004a4f8e
IDP user update API (#34332) 2025-10-21 12:02:25 -06:00
jacobshandling
f700d1029e
Hide software host count, versions table when no hosts have the software installed (#34484)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32635
Hide host count and version table when no hosts have the software
installed

<img width="1518" height="568" alt="Screenshot 2025-10-17 at 4 36 13 PM"
src="https://github.com/user-attachments/assets/4f77d039-d9d0-427e-a3f4-a8774a3f6ff7"
/>



- [x] Changes file added for user-visible changes in `changes/`
- [x] QA'd all new/changed functionality manually
2025-10-20 14:46:35 -07:00
Ian Littman
75a3af5b43
Don't try to pull config profiles when MDM isn't enabled for any platform when generating GitOps YAML (#34549)
Fixes #33677.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
2025-10-20 15:47:44 -05:00
Luke Heath
2c8ae8cc78
Adding changes for Fleet v4.75.0 (#33583) (#34483) 2025-10-17 21:51:17 -05:00
jacobshandling
c9a14d5038
Update vuln processing guide, add change file for JetBrains IDE extension vulns (#34478)
Addresses #32266 

Update docs, add change file
2025-10-17 17:39:21 -05:00
Sarah Gillespie
e11ddc9866
Support $FLEET_VAR_SCEP_RENEWAL_ID in OU field (#34403) 2025-10-17 12:47:04 -05:00
Scott Gress
81f589d661
Update add script UI (#34349)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32632

# Details

This PR updates the Script Library page in the following ways:
* When no scripts are uploaded for a team, it shows the "Add script" UI
with a button that opens a new "Add Script" modal
* When scripts are uploaded, the "Add script" button is instead added to
the header of the scripts list, and clicking it opens that modal

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests
working on this

- [X] QA'd all new/changed functionality manually
- [X] Test empty state: go to controls/scripts/library for a team with
no scripts. Clicking "upload" button in empty state should open the add
script modal.
- [X] In the modal, select a .ps1 script. Should not see additional
text.
- [X] Close modal without uploading. Re-open. File field should be
cleared & upload button visible again.
- [X] Select a .sh script. Should see additional text about macOS and
Linux.
  - [X] Add script. Make sure script saves and modal closes.
- [X] Once script has been added, make sure empty state is gone and "Add
script" button is at the top of the list.
- [X] Go to /controls/os-settings/custom-settings for a team with no
profiles uploaded. Make sure empty state text styles match the empty
state for script uploads.
- [X] Open modal to add profile. Make sure upload text styles match the
script upload modal.
- [X] Enable GitOps mode. Go to controls/scripts/library for a team with
scripts added. Make sure new "Add script" button is disabled w/ standard
tooltip in GitOps mode.

Scripts empty state:

<img width="697" height="352" alt="image"
src="https://github.com/user-attachments/assets/32f0f246-bddb-4bb7-bc39-48d9978de9fa"
/>

Scripts uploader:

<img width="745" height="590" alt="image"
src="https://github.com/user-attachments/assets/f82414e2-9318-4543-b5ca-41e759662587"
/>

Scripts uploader with .sh

<img width="750" height="539" alt="image"
src="https://github.com/user-attachments/assets/0b989067-921a-4d18-93ed-09aac90fc9cb"
/>

Scripts table:

<img width="686" height="256" alt="image"
src="https://github.com/user-attachments/assets/848f1b56-6e9e-48d4-9a03-6fdf5427301e"
/>

Profiles empty state:

<img width="700" height="377" alt="image"
src="https://github.com/user-attachments/assets/8f92bcd9-2215-41f6-a540-4774f7e9542b"
/>

Profiles uploader:

<img width="707" height="682" alt="image"
src="https://github.com/user-attachments/assets/eef216af-3447-48e7-882a-e42e888e1c17"
/>
2025-10-17 10:49:59 -05:00
jacobshandling
5f626e2a8c
Add gigs_all_disk_space vital collection, storage, service, and UI rendering for Linux hosts (#34077)
## Addresses #31671 

- [x] Changes file added for user-visible changes in `changes/`
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added total disk space metrics for all partitions on Linux hosts. The
disk space indicator now displays comprehensive storage information
including root partition and all other partitions, improving visibility
into host storage capacity.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
2025-10-17 08:24:23 -07:00
Victor Lyuboslavsky
9295a82e83
Improved MySQL query performance software versions and vulnerabilities endpoints (#34262)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32178

Software optimization: skipping an unnecessary software_cve join when
vulnerability details are not needed. Vulnerabilities are still
returned, so functionality remains unchanged.

Vulnerabilities optimization: Query vulnerability_host_counts directly
and LEFT JOIN for metadata. This eliminates the expensive UNION of all
CVE rows that was causing performance issues.

Previous approach: UNION all CVEs (many rows) → JOIN
vulnerability_host_counts → filter
New approach: Start with filtered vulnerability_host_counts → LEFT JOIN
for metadata
This reduces the working set before any expensive operations

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests (see below for the test)
- [x] QA'd all new/changed functionality manually
- Planning to test in loadtest after also improving the software
endpoint

Performance test for replicating the problem and testing the fix:
```go
package mysql

import (
	"context"
	"fmt"
	"testing"
	"time"

	"github.com/fleetdm/fleet/v4/server/fleet"
	"github.com/fleetdm/fleet/v4/server/ptr"
	"github.com/fleetdm/fleet/v4/server/test"
	"github.com/stretchr/testify/require"
)

// TestListVulnerabilitiesPerformance is a performance test that replicates
// the production performance problem with ListVulnerabilities.
//
// This test creates a realistic dataset with thousands of CVEs and measures
// query performance under various conditions. Run with:
//
//	go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
//
// To see detailed timing output, set the environment variable:
//
//	VERBOSE=1 go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
func TestListVulnerabilitiesPerformance(t *testing.T) {
	if testing.Short() {
		t.Skip("skipping performance test in short mode")
	}

	ds := CreateMySQLDS(t)
	defer TruncateTables(t, ds)

	ctx := context.Background()

	// Create a realistic dataset
	t.Log("Setting up test data...")
	setupPerformanceTestData(t, ds)
	t.Log("Test data setup complete")

	// Test cases covering common query patterns
	testCases := []struct {
		name string
		opts fleet.VulnListOptions
	}{
		{
			name: "Global list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE:   true,
				TeamID: ptr.Uint(1),
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - with exploit filter",
			opts: fleet.VulnListOptions{
				IsEE:         true,
				TeamID:       ptr.Uint(1),
				KnownExploit: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - with CVE search",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					MatchQuery:     "2023",
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - second page",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           1,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Free version - global list",
			opts: fleet.VulnListOptions{
				IsEE: false,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
	}

	// Run performance tests
	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			// Warm up the query cache
			_, _, err := ds.ListVulnerabilities(ctx, tc.opts)
			require.NoError(t, err)

			// Measure query performance
			const iterations = 5
			var totalDuration time.Duration

			for i := 0; i < iterations; i++ {
				start := time.Now()
				vulns, meta, err := ds.ListVulnerabilities(ctx, tc.opts)
				duration := time.Since(start)
				totalDuration += duration

				require.NoError(t, err)
				require.NotNil(t, meta)
				require.NotEmpty(t, vulns, "expected vulnerabilities to be returned")

				if i == 0 {
					t.Logf("  First run: %v (returned %d results)", duration, len(vulns))
				}
			}

			avgDuration := totalDuration / iterations
			t.Logf("  Average of %d runs: %v", iterations, avgDuration)

			// Performance assertions
			// These thresholds represent the current performance problem
			// After optimization, these should be reduced significantly
			if avgDuration > 2*time.Second {
				t.Logf("  ⚠️  WARNING: Query took %v (>2s) - performance issue detected", avgDuration)
			} else if avgDuration > 500*time.Millisecond {
				t.Logf("  ⚠️  SLOW: Query took %v (>500ms)", avgDuration)
			} else {
				t.Logf("  ✓ GOOD: Query took %v (<500ms)", avgDuration)
			}
		})
	}

	// Test count query performance
	t.Run("Count vulnerabilities performance", func(t *testing.T) {
		opts := fleet.VulnListOptions{
			IsEE: true,
		}

		// Warm up
		_, err := ds.CountVulnerabilities(ctx, opts)
		require.NoError(t, err)

		// Measure
		const iterations = 5
		var totalDuration time.Duration

		for i := 0; i < iterations; i++ {
			start := time.Now()
			count, err := ds.CountVulnerabilities(ctx, opts)
			duration := time.Since(start)
			totalDuration += duration

			require.NoError(t, err)
			require.Greater(t, count, uint(0))

			if i == 0 {
				t.Logf("  First run: %v (count=%d)", duration, count)
			}
		}

		avgDuration := totalDuration / iterations
		t.Logf("  Average of %d runs: %v", iterations, avgDuration)

		if avgDuration > 2*time.Second {
			t.Logf("  ⚠️  WARNING: Count query took %v (>2s)", avgDuration)
		} else if avgDuration > 500*time.Millisecond {
			t.Logf("  ⚠️  SLOW: Count query took %v (>500ms)", avgDuration)
		} else {
			t.Logf("  ✓ GOOD: Count query took %v (<500ms)", avgDuration)
		}
	})
}

// BenchmarkListVulnerabilities provides benchmark results for ListVulnerabilities.
// Run with:
//
//	go test -bench=BenchmarkListVulnerabilities -benchmem -run=^$ ./server/datastore/mysql
func BenchmarkListVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()

	// Setup test data
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	// Benchmark the most common query pattern
	opts := fleet.VulnListOptions{
		IsEE: true,
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkListVulnerabilitiesWithTeam benchmarks team-specific queries
func BenchmarkListVulnerabilitiesWithTeam(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE:   true,
		TeamID: ptr.Uint(1),
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkCountVulnerabilities benchmarks the count query
func BenchmarkCountVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE: true,
	}

	for i := 0; i < b.N; i++ {
		_, err := ds.CountVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// setupPerformanceTestData creates a realistic dataset that mimics production
// This creates:
// - ~80,000+ unique CVEs (matching production scale)
// - ~73,000 software_cve entries
// - ~35,000 operating_system_vulnerabilities entries
// - Multiple teams
// - Various host counts per vulnerability
//
// Note: This will take several minutes to run but will replicate production performance issues
func setupPerformanceTestData(t testing.TB, ds *Datastore) {
	ctx := context.Background()

	// Create 100 hosts across different teams and OS types (doubled from 50)
	// More hosts = more realistic host count distributions
	hosts := make([]*fleet.Host, 100)
	for i := 0; i < 100; i++ {
		hosts[i] = test.NewHost(t, ds, fmt.Sprintf("host%d", i),
			fmt.Sprintf("192.168.1.%d", i%255+1), // Handle more than 255 hosts
			fmt.Sprintf("key%d", i),
			fmt.Sprintf("uuid%d", i),
			time.Now())
	}

	// Create 3 teams
	team1, err := ds.NewTeam(ctx, &fleet.Team{Name: "Engineering"})
	require.NoError(t, err)

	team2, err := ds.NewTeam(ctx, &fleet.Team{Name: "Sales"})
	require.NoError(t, err)

	team3, err := ds.NewTeam(ctx, &fleet.Team{Name: "Support"})
	require.NoError(t, err)

	// Distribute hosts across teams
	// 40 hosts in team1, 30 in team2, 20 in team3, 10 with no team
	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team1.ID, getHostIDs(hosts[0:40])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team2.ID, getHostIDs(hosts[40:70])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team3.ID, getHostIDs(hosts[70:90])))
	require.NoError(t, err)

	// Set up OS versions (Windows, macOS, Ubuntu)
	windowsOS := fleet.OperatingSystem{
		Name:     "Microsoft Windows 11 Enterprise",
		Version:  "10.0.22621.2715",
		Arch:     "x86_64",
		Platform: "windows",
	}

	macOS := fleet.OperatingSystem{
		Name:     "macOS",
		Version:  "14.1.2",
		Arch:     "arm64",
		Platform: "darwin",
	}

	ubuntuOS := fleet.OperatingSystem{
		Name:     "Ubuntu",
		Version:  "22.04",
		Arch:     "x86_64",
		Platform: "ubuntu",
	}

	// Assign OS to hosts: 50 Windows, 30 macOS, 20 Ubuntu
	for i := 0; i < 50; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, windowsOS)
		require.NoError(t, err)
	}
	for i := 50; i < 80; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, macOS)
		require.NoError(t, err)
	}
	for i := 80; i < 100; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, ubuntuOS)
		require.NoError(t, err)
	}

	err = ds.UpdateOSVersions(ctx)
	require.NoError(t, err)

	// Create realistic CVE distribution matching production scale
	// In production, we see:
	// - ~73,000 software CVEs
	// - ~35,000 OS CVEs
	// - Many CVEs overlap between software and OS
	// - ~80,000 unique CVEs after deduplication

	// We now create production-scale data:
	// - 50,000 software CVEs (70% of production)
	// - 30,000 OS CVEs (85% of production)
	// - Some overlap to create ~80,000 total CVEs after UNION
	// This should replicate the 500-1000ms+ query times seen in production

	t.Log("Creating software vulnerabilities... (this will take a few minutes)")
	createSoftwareVulnerabilities(t, ds, hosts[:60], 50000)

	t.Log("Creating OS vulnerabilities...")
	createOSVulnerabilities(t, ds, 30000)

	t.Log("Creating CVE metadata...")
	createCVEMetadata(t, ds, 80000)

	t.Log("Updating vulnerability host counts...")
	err = ds.UpdateVulnerabilityHostCounts(ctx, 10)
	require.NoError(t, err)

	t.Log("Setup complete - ready for performance testing")
}

// createSoftwareVulnerabilities creates software entries and their CVEs
func createSoftwareVulnerabilities(t testing.TB, ds *Datastore, hosts []*fleet.Host, numCVEs int) {
	ctx := context.Background()

	// Create more software packages to better distribute CVEs
	softwarePackages := []fleet.Software{
		{Name: "Chrome", Version: "120.0.1", Source: "programs"},
		{Name: "Firefox", Version: "121.0", Source: "programs"},
		{Name: "Node.js", Version: "18.19.0", Source: "programs"},
		{Name: "Python", Version: "3.11.7", Source: "programs"},
		{Name: "Docker", Version: "24.0.7", Source: "programs"},
		{Name: "nginx", Version: "1.24.0", Source: "deb_packages"},
		{Name: "postgresql", Version: "15.5", Source: "deb_packages"},
		{Name: "redis", Version: "7.2.3", Source: "deb_packages"},
		{Name: "mysql", Version: "8.0.35", Source: "deb_packages"},
		{Name: "git", Version: "2.43.0", Source: "deb_packages"},
		{Name: "openssl", Version: "3.0.12", Source: "deb_packages"},
		{Name: "curl", Version: "8.5.0", Source: "deb_packages"},
		{Name: "vim", Version: "9.0", Source: "deb_packages"},
		{Name: "apache2", Version: "2.4.58", Source: "deb_packages"},
		{Name: "php", Version: "8.2.14", Source: "deb_packages"},
	}

	// Install software on hosts
	for i, host := range hosts {
		// Each host gets 5-8 software packages
		numPackages := 5 + (i % 4)
		if numPackages > len(softwarePackages) {
			numPackages = len(softwarePackages)
		}
		hostSoftware := softwarePackages[:numPackages]
		_, err := ds.UpdateHostSoftware(ctx, host.ID, hostSoftware)
		require.NoError(t, err)
	}

	err := ds.SyncHostsSoftware(ctx, time.Now())
	require.NoError(t, err)

	// Create CVEs for software (distributed across 15 software IDs)
	// Each software gets many CVEs to simulate real-world vulnerability distribution
	cvesPerSoftware := numCVEs / 15
	t.Logf("  Creating %d CVEs per software package (15 packages)", cvesPerSoftware)

	for softwareID := uint(1); softwareID <= 15; softwareID++ {
		// Insert CVEs in batches for better performance
		batchSize := 1000
		for batchStart := 0; batchStart < cvesPerSoftware; batchStart += batchSize {
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerSoftware {
				batchEnd = cvesPerSoftware
			}

			for i := batchStart; i < batchEnd; i++ {
				cveNum := int(softwareID-1)*cvesPerSoftware + i
				// Use wider CVE number range to avoid duplicates
				cve := fmt.Sprintf("CVE-2023-%05d", cveNum)

				_, err := ds.InsertSoftwareVulnerability(ctx, fleet.SoftwareVulnerability{
					SoftwareID: softwareID,
					CVE:        cve,
				}, fleet.NVDSource)
				require.NoError(t, err)
			}
		}

		if softwareID%5 == 0 {
			t.Logf("  Progress: %d/%d software packages completed", softwareID, 15)
		}
	}
}

// createOSVulnerabilities creates OS vulnerabilities
func createOSVulnerabilities(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()

	// Create CVEs for each OS type
	// OS ID 1 = Windows, 2 = macOS, 3 = Ubuntu
	osIDs := []uint{1, 2, 3}
	cvesPerOS := numCVEs / len(osIDs)

	t.Logf("  Creating %d CVEs per OS type (3 OS types)", cvesPerOS)

	for _, osID := range osIDs {
		// Insert in batches to avoid memory issues with large slices
		batchSize := 5000
		totalBatches := (cvesPerOS + batchSize - 1) / batchSize

		for batchNum := 0; batchNum < totalBatches; batchNum++ {
			batchStart := batchNum * batchSize
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerOS {
				batchEnd = cvesPerOS
			}

			vulns := make([]fleet.OSVulnerability, batchEnd-batchStart)
			for i := 0; i < len(vulns); i++ {
				actualIndex := batchStart + i
				// Use different year to avoid overlap with software CVEs
				// (but still create some overlap)
				cveNum := int(osID-1)*cvesPerOS + actualIndex
				cve := fmt.Sprintf("CVE-2022-%05d", cveNum)

				// 10% overlap with software CVEs for realism
				if actualIndex%10 == 0 {
					cve = fmt.Sprintf("CVE-2023-%05d", cveNum)
				}

				vulns[i] = fleet.OSVulnerability{
					OSID: osID,
					CVE:  cve,
				}
			}

			_, err := ds.InsertOSVulnerabilities(ctx, vulns, fleet.MSRCSource)
			require.NoError(t, err)

			if (batchNum+1)%2 == 0 || batchNum == totalBatches-1 {
				t.Logf("  Progress: OS %d - batch %d/%d completed", osID, batchNum+1, totalBatches)
			}
		}
	}
}

// createCVEMetadata creates CVE metadata entries
func createCVEMetadata(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()
	mockTime := time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC)

	// Create metadata in batches of 500 for better performance
	batchSize := 500
	totalBatches := (numCVEs + batchSize - 1) / batchSize
	t.Logf("  Creating CVE metadata in %d batches", totalBatches)

	for start := 0; start < numCVEs; start += batchSize {
		end := start + batchSize
		if end > numCVEs {
			end = numCVEs
		}

		batch := make([]fleet.CVEMeta, end-start)
		for i := start; i < end; i++ {
			// Alternate between 2022 and 2023 CVEs to match our created vulnerabilities
			year := 2023
			if i >= 50000 {
				year = 2022
			}

			// Use 5-digit format to match our CVE creation
			cve := fmt.Sprintf("CVE-%d-%05d", year, i%100000)

			// 30% have CISA known exploit
			cisaExploit := (i % 10) < 3

			batch[i-start] = fleet.CVEMeta{
				CVE:              cve,
				CVSSScore:        ptr.Float64(5.0 + float64(i%50)/10.0),
				EPSSProbability:  ptr.Float64(float64(i%100) / 100.0),
				CISAKnownExploit: ptr.Bool(cisaExploit),
				Published:        ptr.Time(mockTime.Add(time.Duration(i) * time.Hour)),
				Description:      fmt.Sprintf("Test vulnerability %s", cve),
			}
		}

		err := ds.InsertCVEMeta(ctx, batch)
		require.NoError(t, err)

		// Report progress every 10 batches
		batchNum := (start / batchSize) + 1
		if batchNum%10 == 0 || batchNum == totalBatches {
			t.Logf("  Progress: %d/%d batches completed (%d CVEs)", batchNum, totalBatches, end)
		}
	}
}

// getHostIDs extracts host IDs from a slice of hosts
func getHostIDs(hosts []*fleet.Host) []uint {
	ids := make([]uint, len(hosts))
	for i, h := range hosts {
		ids[i] = h.ID
	}
	return ids
}

// CreateMySQLDSForBenchmark creates a datastore for benchmarking
func CreateMySQLDSForBenchmark(b *testing.B) *Datastore {
	return CreateMySQLDS(b)
}
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Performance Improvements**
* Faster loading of the vulnerabilities list via optimized database
queries for the vulnerabilities API endpoint.

* **Bug Fixes**
* More accurate “created at” timestamps for vulnerabilities, improving
sorting and consistency.
* More consistent source attribution for vulnerabilities when multiple
sources are available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-17 09:57:47 -05:00
Scott Gress
61970118e9
Stop setup experience on software install failure (#34173)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33173
**Related issue:** Resolves #33111 

# Details

This is the remaining work to implement the "Stop the setup experience
when required software fails to install" feature. This didn't turn out
to be quite as straightforward as expected so I ended up doing a bit of
design-by-code and expect some feedback on the approach. I tried to make
it as low-touch as possible. The general design is:

1. In the `maybeUpdateSetupExperienceStatus` function which is called in
various places when a setup experience step is marked as completed, call
a new `maybeCancelPendingSetupExperienceSteps` function if the setup
step was marked as failed. Similarly call
`maybeCancelPendingSetupExperienceSteps` if a VPP app install fails to
enqueue.
2. In `maybeCancelPendingSetupExperienceSteps`, check whether the
specified host is MacOS and whether the "RequireAllSoftwareMacOS" flag
is set in the team (or global) config. If so, mark the remaining setup
experience items as canceled and cancel any upcoming activities related
to those steps.
3. On the front-end, if the `require_all_software_macos` is set and a
software step is marked as failed, show a new failure page indicating
that setup has failed and showing details of the failed software.
4. On the agent side, when checking setup experience status, send a
`reset_after_failure` flag _only the first time_. If this flag is set,
then the code in the `/orbit/setup_experience/status` handler will clear
and re-queue any failed setup experience steps (but leave successful
steps to avoid re-installing already-installed software). This
facilitates re-starting the setup experience when the host is rebooted.

I also updated the way that software (packages and VPP) is queued up for
the setup experience to be ordered alphabetically, to make it easier to
test _and_ because this is a desired outcome for a future story. Since
the order is not deterministic now, this update shouldn't cause any
problems (aside from a couple of test updates), but I'm ok taking it out
if desired.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [X] Added/updated automated tests
* Added a new integration test for software packages, testing that a
failed software package causes the rest of the setup experience to be
marked as failed when `require_all_software_macos` is set, and testing
that the "reset after failure" code works.
* Added a new integration test for VPP packages, testing that a failed
VPP enqueue causes the same halting of the setup experience.
I _don't_ have test for a failure _during_ a VPP install. It should go
through the same code path as the software package failure, so it's not
a huge gap.

- [ ] QA'd all new/changed functionality manually
Working on it 

## fleetd/orbit/Fleet Desktop

- [X] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [X] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [X] Verified that fleetd runs on macOS, Linux and Windows


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- New Features
- Configurable option to halt macOS device setup if any software install
fails.
- Device setup page now shows a clear “Device setup failed” state with
expandable error details when all software is required on macOS.
- Improvements
- Setup status now includes per-step error messages for better
troubleshooting.
- Pending setup steps are automatically canceled after a failure when
applicable, with support to reset and retry the setup flow as
configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
2025-10-17 08:38:53 -05:00
Ian Littman
4e0c34eccd
Fail GitOps run when software package YAML is supplied but fields specific to that file are still specified in the team YAML (#34142)
Fixes #34066. Included a changes file because this catches fields other
than software icon.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
2025-10-16 13:06:36 -05:00
Juan Fernandez
32f2ddaa63
Show warning when viewing settings without auth (#34309)
**Related issue:** Resolves #32229

Show flash warning when viewing settings from a team where auth is not
granted.
2025-10-15 20:01:22 -04:00
Victor Lyuboslavsky
b9520f6b14
Changed the default name of Host Identity CA to 'Fleet Host Identity CA' to avoid conflict with Fleet's Apple MDM CA. (#34219)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34217

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually
2025-10-15 09:52:45 -05:00
Ian Littman
acff7b7343
Add software name cleanup on ingestion for various items where bundle executable isn't well-named (#34232)
Fixes #34159. Split from CPE translation fixes so this can be merged
into `main` pre-QA.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

- [ ] QA'd all new/changed functionality manually
2025-10-15 09:50:52 -05:00