Elgato_dark/fleet
1
0
Fork 0
mirror of https://github.com/fleetdm/fleet synced 2026-04-21 21:47:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 117
19439 commits 3115 branches 563 tags 2.1 GiB
Commit graph

238 commits

Author SHA1 Message Date
Martin Angers
4e97263881
GitOps: support categories for in-house apps gitops (#35531) 2025-11-12 09:06:16 -05:00
Jonathan Katz
9b36abc608
Support categories for in-house apps (#35542) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #35464 
Also allows #35531 to be merged
Adds self-service categories for #32247

Changes:
- Move category insertion up in update installer so in house apps can
use it
- Add migration for in_house_app_software_categories table
- Fix some overly wrapped errors
- Add new tests for categories
- Fix unused param in another test

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

## Database migrations

- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
 2025-11-11 15:13:24 -05:00
jacobshandling
acb563337e
Ingest, store, consider in unique_identifier, and serve upgrade_codes for Windows software (#34786) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33907 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
~- [ ] Confirmed that updating the timestamps is acceptable, and will
not cause unwanted side effects.~ N/A
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Windows software inventory now includes upgrade code data for better
software identification and tracking.

* **Chores**
* Database schema updated to support upgrade code storage for software
titles and inventory records.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-11-07 15:33:31 -08:00
Carlo
715d963f82
My device page (self-service) for iOS/iPadOS (#35238) 
Implements #32247. This is the complete feature branch, consolidating:

- https://github.com/fleetdm/fleet/pull/35018
- https://github.com/fleetdm/fleet/pull/34758
- https://github.com/fleetdm/fleet/pull/35009
- https://github.com/fleetdm/fleet/pull/35181
- https://github.com/fleetdm/fleet/pull/35342

---------

Co-authored-by: Jonathan Katz <44128041+jkatz01@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: jkatz01 <yehonatankatz@gmail.com>
 2025-11-07 17:30:51 -05:00
Jahziel Villasana-Espinoza
621012356f
software display names: API support (#35182) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33778

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
 2025-11-05 12:03:30 -05:00
Carlo
8b03dcb73d
CP: Fix multiple activities for package-only installs (#35185) 
Cherry-picks #35137, and implements changes based on feedback in that PR for future use.
 2025-11-05 08:47:07 -05:00
Jahziel Villasana-Espinoza
1cf16f0539
software display names: DB changes (#35066) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33776 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
 2025-11-04 10:04:42 -05:00
Jonathan Katz
f4fd1c106e
Fix in house app package name and title name (#34868) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34823 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually
## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
 2025-10-28 13:19:13 -04:00
Martin Angers
cab7cc15be
Initial support for in-house apps on iOS/iPadOS (#34802) 2025-10-28 08:33:58 -04:00
Victor Lyuboslavsky
27b8d1aa4b
Fixed issue searching software versions (#34770) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34713

Fix for unreleased but. Needs to be cherry picked into 4.76.0

# Checklist for submitter

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
 2025-10-25 16:02:02 -05:00
Victor Lyuboslavsky
9295a82e83
Improved MySQL query performance software versions and vulnerabilities endpoints (#34262) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32178

Software optimization: skipping an unnecessary software_cve join when
vulnerability details are not needed. Vulnerabilities are still
returned, so functionality remains unchanged.

Vulnerabilities optimization: Query vulnerability_host_counts directly
and LEFT JOIN for metadata. This eliminates the expensive UNION of all
CVE rows that was causing performance issues.

Previous approach: UNION all CVEs (many rows) → JOIN
vulnerability_host_counts → filter
New approach: Start with filtered vulnerability_host_counts → LEFT JOIN
for metadata
This reduces the working set before any expensive operations

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests (see below for the test)
- [x] QA'd all new/changed functionality manually
- Planning to test in loadtest after also improving the software
endpoint

Performance test for replicating the problem and testing the fix:
```go
package mysql

import (
	"context"
	"fmt"
	"testing"
	"time"

	"github.com/fleetdm/fleet/v4/server/fleet"
	"github.com/fleetdm/fleet/v4/server/ptr"
	"github.com/fleetdm/fleet/v4/server/test"
	"github.com/stretchr/testify/require"
)

// TestListVulnerabilitiesPerformance is a performance test that replicates
// the production performance problem with ListVulnerabilities.
//
// This test creates a realistic dataset with thousands of CVEs and measures
// query performance under various conditions. Run with:
//
//	go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
//
// To see detailed timing output, set the environment variable:
//
//	VERBOSE=1 go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
func TestListVulnerabilitiesPerformance(t *testing.T) {
	if testing.Short() {
		t.Skip("skipping performance test in short mode")
	}

	ds := CreateMySQLDS(t)
	defer TruncateTables(t, ds)

	ctx := context.Background()

	// Create a realistic dataset
	t.Log("Setting up test data...")
	setupPerformanceTestData(t, ds)
	t.Log("Test data setup complete")

	// Test cases covering common query patterns
	testCases := []struct {
		name string
		opts fleet.VulnListOptions
	}{
		{
			name: "Global list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE:   true,
				TeamID: ptr.Uint(1),
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - with exploit filter",
			opts: fleet.VulnListOptions{
				IsEE:         true,
				TeamID:       ptr.Uint(1),
				KnownExploit: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - with CVE search",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					MatchQuery:     "2023",
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - second page",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           1,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Free version - global list",
			opts: fleet.VulnListOptions{
				IsEE: false,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
	}

	// Run performance tests
	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			// Warm up the query cache
			_, _, err := ds.ListVulnerabilities(ctx, tc.opts)
			require.NoError(t, err)

			// Measure query performance
			const iterations = 5
			var totalDuration time.Duration

			for i := 0; i < iterations; i++ {
				start := time.Now()
				vulns, meta, err := ds.ListVulnerabilities(ctx, tc.opts)
				duration := time.Since(start)
				totalDuration += duration

				require.NoError(t, err)
				require.NotNil(t, meta)
				require.NotEmpty(t, vulns, "expected vulnerabilities to be returned")

				if i == 0 {
					t.Logf("  First run: %v (returned %d results)", duration, len(vulns))
				}
			}

			avgDuration := totalDuration / iterations
			t.Logf("  Average of %d runs: %v", iterations, avgDuration)

			// Performance assertions
			// These thresholds represent the current performance problem
			// After optimization, these should be reduced significantly
			if avgDuration > 2*time.Second {
				t.Logf("  ⚠️  WARNING: Query took %v (>2s) - performance issue detected", avgDuration)
			} else if avgDuration > 500*time.Millisecond {
				t.Logf("  ⚠️  SLOW: Query took %v (>500ms)", avgDuration)
			} else {
				t.Logf("  ✓ GOOD: Query took %v (<500ms)", avgDuration)
			}
		})
	}

	// Test count query performance
	t.Run("Count vulnerabilities performance", func(t *testing.T) {
		opts := fleet.VulnListOptions{
			IsEE: true,
		}

		// Warm up
		_, err := ds.CountVulnerabilities(ctx, opts)
		require.NoError(t, err)

		// Measure
		const iterations = 5
		var totalDuration time.Duration

		for i := 0; i < iterations; i++ {
			start := time.Now()
			count, err := ds.CountVulnerabilities(ctx, opts)
			duration := time.Since(start)
			totalDuration += duration

			require.NoError(t, err)
			require.Greater(t, count, uint(0))

			if i == 0 {
				t.Logf("  First run: %v (count=%d)", duration, count)
			}
		}

		avgDuration := totalDuration / iterations
		t.Logf("  Average of %d runs: %v", iterations, avgDuration)

		if avgDuration > 2*time.Second {
			t.Logf("  ⚠️  WARNING: Count query took %v (>2s)", avgDuration)
		} else if avgDuration > 500*time.Millisecond {
			t.Logf("  ⚠️  SLOW: Count query took %v (>500ms)", avgDuration)
		} else {
			t.Logf("  ✓ GOOD: Count query took %v (<500ms)", avgDuration)
		}
	})
}

// BenchmarkListVulnerabilities provides benchmark results for ListVulnerabilities.
// Run with:
//
//	go test -bench=BenchmarkListVulnerabilities -benchmem -run=^$ ./server/datastore/mysql
func BenchmarkListVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()

	// Setup test data
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	// Benchmark the most common query pattern
	opts := fleet.VulnListOptions{
		IsEE: true,
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkListVulnerabilitiesWithTeam benchmarks team-specific queries
func BenchmarkListVulnerabilitiesWithTeam(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE:   true,
		TeamID: ptr.Uint(1),
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkCountVulnerabilities benchmarks the count query
func BenchmarkCountVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE: true,
	}

	for i := 0; i < b.N; i++ {
		_, err := ds.CountVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// setupPerformanceTestData creates a realistic dataset that mimics production
// This creates:
// - ~80,000+ unique CVEs (matching production scale)
// - ~73,000 software_cve entries
// - ~35,000 operating_system_vulnerabilities entries
// - Multiple teams
// - Various host counts per vulnerability
//
// Note: This will take several minutes to run but will replicate production performance issues
func setupPerformanceTestData(t testing.TB, ds *Datastore) {
	ctx := context.Background()

	// Create 100 hosts across different teams and OS types (doubled from 50)
	// More hosts = more realistic host count distributions
	hosts := make([]*fleet.Host, 100)
	for i := 0; i < 100; i++ {
		hosts[i] = test.NewHost(t, ds, fmt.Sprintf("host%d", i),
			fmt.Sprintf("192.168.1.%d", i%255+1), // Handle more than 255 hosts
			fmt.Sprintf("key%d", i),
			fmt.Sprintf("uuid%d", i),
			time.Now())
	}

	// Create 3 teams
	team1, err := ds.NewTeam(ctx, &fleet.Team{Name: "Engineering"})
	require.NoError(t, err)

	team2, err := ds.NewTeam(ctx, &fleet.Team{Name: "Sales"})
	require.NoError(t, err)

	team3, err := ds.NewTeam(ctx, &fleet.Team{Name: "Support"})
	require.NoError(t, err)

	// Distribute hosts across teams
	// 40 hosts in team1, 30 in team2, 20 in team3, 10 with no team
	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team1.ID, getHostIDs(hosts[0:40])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team2.ID, getHostIDs(hosts[40:70])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team3.ID, getHostIDs(hosts[70:90])))
	require.NoError(t, err)

	// Set up OS versions (Windows, macOS, Ubuntu)
	windowsOS := fleet.OperatingSystem{
		Name:     "Microsoft Windows 11 Enterprise",
		Version:  "10.0.22621.2715",
		Arch:     "x86_64",
		Platform: "windows",
	}

	macOS := fleet.OperatingSystem{
		Name:     "macOS",
		Version:  "14.1.2",
		Arch:     "arm64",
		Platform: "darwin",
	}

	ubuntuOS := fleet.OperatingSystem{
		Name:     "Ubuntu",
		Version:  "22.04",
		Arch:     "x86_64",
		Platform: "ubuntu",
	}

	// Assign OS to hosts: 50 Windows, 30 macOS, 20 Ubuntu
	for i := 0; i < 50; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, windowsOS)
		require.NoError(t, err)
	}
	for i := 50; i < 80; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, macOS)
		require.NoError(t, err)
	}
	for i := 80; i < 100; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, ubuntuOS)
		require.NoError(t, err)
	}

	err = ds.UpdateOSVersions(ctx)
	require.NoError(t, err)

	// Create realistic CVE distribution matching production scale
	// In production, we see:
	// - ~73,000 software CVEs
	// - ~35,000 OS CVEs
	// - Many CVEs overlap between software and OS
	// - ~80,000 unique CVEs after deduplication

	// We now create production-scale data:
	// - 50,000 software CVEs (70% of production)
	// - 30,000 OS CVEs (85% of production)
	// - Some overlap to create ~80,000 total CVEs after UNION
	// This should replicate the 500-1000ms+ query times seen in production

	t.Log("Creating software vulnerabilities... (this will take a few minutes)")
	createSoftwareVulnerabilities(t, ds, hosts[:60], 50000)

	t.Log("Creating OS vulnerabilities...")
	createOSVulnerabilities(t, ds, 30000)

	t.Log("Creating CVE metadata...")
	createCVEMetadata(t, ds, 80000)

	t.Log("Updating vulnerability host counts...")
	err = ds.UpdateVulnerabilityHostCounts(ctx, 10)
	require.NoError(t, err)

	t.Log("Setup complete - ready for performance testing")
}

// createSoftwareVulnerabilities creates software entries and their CVEs
func createSoftwareVulnerabilities(t testing.TB, ds *Datastore, hosts []*fleet.Host, numCVEs int) {
	ctx := context.Background()

	// Create more software packages to better distribute CVEs
	softwarePackages := []fleet.Software{
		{Name: "Chrome", Version: "120.0.1", Source: "programs"},
		{Name: "Firefox", Version: "121.0", Source: "programs"},
		{Name: "Node.js", Version: "18.19.0", Source: "programs"},
		{Name: "Python", Version: "3.11.7", Source: "programs"},
		{Name: "Docker", Version: "24.0.7", Source: "programs"},
		{Name: "nginx", Version: "1.24.0", Source: "deb_packages"},
		{Name: "postgresql", Version: "15.5", Source: "deb_packages"},
		{Name: "redis", Version: "7.2.3", Source: "deb_packages"},
		{Name: "mysql", Version: "8.0.35", Source: "deb_packages"},
		{Name: "git", Version: "2.43.0", Source: "deb_packages"},
		{Name: "openssl", Version: "3.0.12", Source: "deb_packages"},
		{Name: "curl", Version: "8.5.0", Source: "deb_packages"},
		{Name: "vim", Version: "9.0", Source: "deb_packages"},
		{Name: "apache2", Version: "2.4.58", Source: "deb_packages"},
		{Name: "php", Version: "8.2.14", Source: "deb_packages"},
	}

	// Install software on hosts
	for i, host := range hosts {
		// Each host gets 5-8 software packages
		numPackages := 5 + (i % 4)
		if numPackages > len(softwarePackages) {
			numPackages = len(softwarePackages)
		}
		hostSoftware := softwarePackages[:numPackages]
		_, err := ds.UpdateHostSoftware(ctx, host.ID, hostSoftware)
		require.NoError(t, err)
	}

	err := ds.SyncHostsSoftware(ctx, time.Now())
	require.NoError(t, err)

	// Create CVEs for software (distributed across 15 software IDs)
	// Each software gets many CVEs to simulate real-world vulnerability distribution
	cvesPerSoftware := numCVEs / 15
	t.Logf("  Creating %d CVEs per software package (15 packages)", cvesPerSoftware)

	for softwareID := uint(1); softwareID <= 15; softwareID++ {
		// Insert CVEs in batches for better performance
		batchSize := 1000
		for batchStart := 0; batchStart < cvesPerSoftware; batchStart += batchSize {
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerSoftware {
				batchEnd = cvesPerSoftware
			}

			for i := batchStart; i < batchEnd; i++ {
				cveNum := int(softwareID-1)*cvesPerSoftware + i
				// Use wider CVE number range to avoid duplicates
				cve := fmt.Sprintf("CVE-2023-%05d", cveNum)

				_, err := ds.InsertSoftwareVulnerability(ctx, fleet.SoftwareVulnerability{
					SoftwareID: softwareID,
					CVE:        cve,
				}, fleet.NVDSource)
				require.NoError(t, err)
			}
		}

		if softwareID%5 == 0 {
			t.Logf("  Progress: %d/%d software packages completed", softwareID, 15)
		}
	}
}

// createOSVulnerabilities creates OS vulnerabilities
func createOSVulnerabilities(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()

	// Create CVEs for each OS type
	// OS ID 1 = Windows, 2 = macOS, 3 = Ubuntu
	osIDs := []uint{1, 2, 3}
	cvesPerOS := numCVEs / len(osIDs)

	t.Logf("  Creating %d CVEs per OS type (3 OS types)", cvesPerOS)

	for _, osID := range osIDs {
		// Insert in batches to avoid memory issues with large slices
		batchSize := 5000
		totalBatches := (cvesPerOS + batchSize - 1) / batchSize

		for batchNum := 0; batchNum < totalBatches; batchNum++ {
			batchStart := batchNum * batchSize
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerOS {
				batchEnd = cvesPerOS
			}

			vulns := make([]fleet.OSVulnerability, batchEnd-batchStart)
			for i := 0; i < len(vulns); i++ {
				actualIndex := batchStart + i
				// Use different year to avoid overlap with software CVEs
				// (but still create some overlap)
				cveNum := int(osID-1)*cvesPerOS + actualIndex
				cve := fmt.Sprintf("CVE-2022-%05d", cveNum)

				// 10% overlap with software CVEs for realism
				if actualIndex%10 == 0 {
					cve = fmt.Sprintf("CVE-2023-%05d", cveNum)
				}

				vulns[i] = fleet.OSVulnerability{
					OSID: osID,
					CVE:  cve,
				}
			}

			_, err := ds.InsertOSVulnerabilities(ctx, vulns, fleet.MSRCSource)
			require.NoError(t, err)

			if (batchNum+1)%2 == 0 || batchNum == totalBatches-1 {
				t.Logf("  Progress: OS %d - batch %d/%d completed", osID, batchNum+1, totalBatches)
			}
		}
	}
}

// createCVEMetadata creates CVE metadata entries
func createCVEMetadata(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()
	mockTime := time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC)

	// Create metadata in batches of 500 for better performance
	batchSize := 500
	totalBatches := (numCVEs + batchSize - 1) / batchSize
	t.Logf("  Creating CVE metadata in %d batches", totalBatches)

	for start := 0; start < numCVEs; start += batchSize {
		end := start + batchSize
		if end > numCVEs {
			end = numCVEs
		}

		batch := make([]fleet.CVEMeta, end-start)
		for i := start; i < end; i++ {
			// Alternate between 2022 and 2023 CVEs to match our created vulnerabilities
			year := 2023
			if i >= 50000 {
				year = 2022
			}

			// Use 5-digit format to match our CVE creation
			cve := fmt.Sprintf("CVE-%d-%05d", year, i%100000)

			// 30% have CISA known exploit
			cisaExploit := (i % 10) < 3

			batch[i-start] = fleet.CVEMeta{
				CVE:              cve,
				CVSSScore:        ptr.Float64(5.0 + float64(i%50)/10.0),
				EPSSProbability:  ptr.Float64(float64(i%100) / 100.0),
				CISAKnownExploit: ptr.Bool(cisaExploit),
				Published:        ptr.Time(mockTime.Add(time.Duration(i) * time.Hour)),
				Description:      fmt.Sprintf("Test vulnerability %s", cve),
			}
		}

		err := ds.InsertCVEMeta(ctx, batch)
		require.NoError(t, err)

		// Report progress every 10 batches
		batchNum := (start / batchSize) + 1
		if batchNum%10 == 0 || batchNum == totalBatches {
			t.Logf("  Progress: %d/%d batches completed (%d CVEs)", batchNum, totalBatches, end)
		}
	}
}

// getHostIDs extracts host IDs from a slice of hosts
func getHostIDs(hosts []*fleet.Host) []uint {
	ids := make([]uint, len(hosts))
	for i, h := range hosts {
		ids[i] = h.ID
	}
	return ids
}

// CreateMySQLDSForBenchmark creates a datastore for benchmarking
func CreateMySQLDSForBenchmark(b *testing.B) *Datastore {
	return CreateMySQLDS(b)
}
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Performance Improvements**
* Faster loading of the vulnerabilities list via optimized database
queries for the vulnerabilities API endpoint.

* **Bug Fixes**
* More accurate “created at” timestamps for vulnerabilities, improving
sorting and consistency.
* More consistent source attribution for vulnerabilities when multiple
sources are available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-10-17 09:57:47 -05:00
Lucas Manuel Rodriguez
15518d2893
Optimize software title reconciliation in vulnerabilities job (#34146) 
Resolves #34055

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Refactor**
* Optimized software title reconciliation used during vulnerability
processing, improving scan performance and reducing database load. More
efficient cleanup of orphaned titles and updates to title names.
* **Tests**
  * Corrected a test name typo for clarity.
* Streamlined MDM integration test by removing redundant title
recreation steps.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-10-14 17:36:45 -05:00
Konstantin Sykulev
bb4b62bd0f
Adding name to software checksum for mac software (#34097) 
**Related issue:** Resolves #28788

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* macOS app checksums now include the app name, improving grouping,
deduplication, and preventing mis-linking or duplicate entries when
multiple names share a bundle ID.
* More stable title handling when bundle IDs are missing, reducing
unintended renames and mismatches.

* **Tests**
* Re-enabled related host-software tests and added a
longest-common-prefix test to validate name reconciliation.

* **Chores**
* Database migration added to recalculate checksums for affected macOS
app records.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-10-14 17:36:34 -05:00
Dante Catalfamo
23bef25ab3
Add hash_sha256 to list hosts software response (#33657) 
**Related issue:** Resolves #33410

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
 2025-10-13 13:43:36 -05:00
Jahziel Villasana-Espinoza
0a3c6c35d3
Android software ingestion (#33826) 
> Closes #33581 


<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)


## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).

---------

Co-authored-by: RachelElysia <rachel@fleetdm.com>
 2025-10-08 10:24:38 -04:00
Zach Wasserman
41c53860e3
Add support for VSCode fork extensions in software inventory (#33595) 
**Related issue:** Resolves #31397

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
 2025-10-07 14:05:22 -07:00
Victor Lyuboslavsky
563bcdf18b
Handle multiple software entries with the same bundle ID during renames. (#33479) 
- Adjusted logic to support multiple software versions sharing a bundle
ID.
- Extended tests to validate scenarios involving renamed software across
versions.

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33468

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] Manually QA'd using osquery `--common_software_name_suffix` switch


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- Bug Fixes
- Improves handling of apps that share the same bundle ID, ensuring all
versions are correctly linked and consistently renamed across hosts.
- Reduces duplicate software entries and keeps host associations intact
during rename operations.
- Delivers more reliable software inventory views with accurate app
names derived from bundle IDs.

- Tests
- Adds comprehensive coverage for scenarios with multiple versions per
bundle ID to validate linking and renaming behavior across hosts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-10-06 11:32:26 -05:00
Konstantin Sykulev
8c91c03eea
Software renaming lock contention (#33791) 
**Related issue:** Resolves #33668

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
 2025-10-06 11:30:10 -05:00
Victor Lyuboslavsky
ecb8ee19e2
Fixed bad software ingestion debug message and added filter for invalid software with missing names. (#33682) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33681

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually
 2025-10-01 12:17:23 -05:00
Victor Lyuboslavsky
48afef8a27
Removing the software renaming fix introduced in 4.73.3 due to MySQL DB performance issues. (#33616) 
Needs to be cherry picked into 4.73.4 and/or 4.74.0 and 4.75.0

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33612

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
 2025-09-30 10:35:47 -04:00
Victor Lyuboslavsky
1ae9597b65
Software ingestion fixes (#33399) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:**
Resolves #29053
Resolves #33298

For reference, the diffs for merging Konstantin's changes into my
original PR are here: https://github.com/fleetdm/fleet/pull/33390

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- Bug Fixes
- Fixed duplicate macOS software entries caused by users renaming apps,
ensuring accurate, consolidated inventory.

- Documentation
- Documented improved software ingestion performance by pre-inserting
data in smaller batches to reduce database lock times during host
check-ins.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-09-24 17:38:13 -05:00
Victor Lyuboslavsky
d6695bf299
Fixed MySQL DB performance regressions (#33184) 
Resolves #33147 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked table schema to confirm autoupdate


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- Bug Fixes
- Resolved MySQL performance regressions from 4.73.0/4.73.1 affecting OS
versions and software titles views, improving load times and reducing
timeouts.

- Refactor
- Optimized OS vulnerabilities fetching by batching multiple OS versions
in a single request.
- Added a supporting database index to speed kernel-related
vulnerability queries.

- Tests
- Added comprehensive tests for multi-OS vulnerability retrieval, CVSS
enrichment, team-scoped data, and service endpoint behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-09-19 15:35:05 -05:00
Victor Lyuboslavsky
19014bfd8f
Added support for retry logic in setup experience software installations. (#32823) 
Fixes #32580

- Added retry logic for software installs
- Added sending intermediate results to Fleet server

I QA'd this on Linux (see video below). For macOS and Windows, I QA'd it
by having the server force retries on normal (non-setup experience
software installs).

Demo video: https://youtu.be/dbu78G6bXf8

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## fleetd/orbit/Fleet Desktop

- [x] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [x] Verified that fleetd runs on macOS, Linux and Windows
- [x] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- New Features
- Automatic retries for software installations, with exponential backoff
on transient/network errors.
- Intermediate failures are recorded without closing the original
request; subsequent attempts continue automatically.
- Activity feed entries are created for intermediate failures, including
install identifiers.
- Setup experience installs now retry automatically (up to 3 attempts).

- Tests
- Expanded test coverage for retry behavior, error classification, and
intermediate failure reporting.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-09-16 12:26:14 -05:00
Konstantin Sykulev
db0b92f738
software library page fetches vpp icons properly (#32828) 
Fixes https://github.com/fleetdm/fleet/issues/32819


## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
 2025-09-10 18:24:58 -05:00
Jonathan Katz
2d4ea255a1
Change LastOpenedAt logging (#32767) 
Fixes: #32067
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
 2025-09-09 13:47:58 -04:00
Ian Littman
0d29f2bfc0
Add custom software icons (#32652) 
For #29478, sans GitOps.

---------

Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Konstantin Sykulev <konst@sykulev.com>
 2025-09-05 17:31:03 -05:00
Jonathan Katz
7c375c6e54
#30403 Fix fleet installed host count discrepancy (#32455) 
Fixes: #30403 
Keys for deletedTitles map were generated differently, causing the same
software title to be marked removed even when a new version of the same
title was inserted.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.


## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:
 2025-09-02 15:05:42 -04:00
Konstantin Sykulev
9a821efe8d
When iterating through softwares LastOpenedAt timestamp is copied as to not modify original records (#31946) 
fixes #31932

The problem here was that `hostInstalledSoftware` returns a slice of
pointers (`[]*hostSoftware`), so when iterating through and assigning
`LastOpenedAt` the original records were getting modified. This code
duplicates the records being put into `bySoftwareTitleID` so that the
records being stored in `bySoftwareID` are the original records.

As a side benefit to this I modified the logic to store the most recent
`LastOpenedAt` for the software title. I think we may be doing something
similar to this on the front end to show the "last used" column when we
have multiple version of a software installed. But this can potentially
be fetched from the API now.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
 2025-08-15 09:44:01 -05:00
Jahziel Villasana-Espinoza
153f73c8ca
30738 linux vulns (#31893) 
- **linux vulns API changes (#31490)**
- **31214 linux vulns optimization (#31722)**

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
 2025-08-14 10:13:37 -04:00
Lucas Manuel Rodriguez
12811546ee
Fix server panic with all teams software titles (#31746) 
For #31571.

Steps to reproduce at the end of the description here:
https://github.com/fleetdm/fleet/issues/31571.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
 2025-08-08 17:49:32 -03:00
Konstantin Sykulev
2fae481a25
Retain vpp apps last install information after inventoried by osquery (#31520) 
fixes #31459

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

---------

Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
 2025-08-01 13:39:51 -05:00
Konstantin Sykulev
828504d038
Automatic install policies in ListHostSoftware (#31469) 
Fixes #30197
# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
 2025-08-01 10:22:14 -05:00
Ian Littman
89ca35c66b
Switch vulns cron false positive clear to clear vulns based on when the vulns run started, rather than based on periodicity (#31364) 
Fixes #26404.

This means that for long vulns runs vulns will stick around longer, so
we don't wind up nuking vulns that were added earlier in the run, and in
cases where the vulns run takes less than 2h we'll see vulns clear
cleanly more quickly.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

- [ ] QA'd all new/changed functionality manually

---------

Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
 2025-07-29 10:14:14 -05:00
Konstantin Sykulev
8234e46e7b
Exclude pending software installs from host inventory request (#30855) 
Fixes https://github.com/fleetdm/fleet/issues/30747

- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [ ] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
 2025-07-21 16:25:00 -05:00
Ian Littman
694f67a26c
Filter out DEB/RPM installers in ListHostSoftware when they're incompatible with the target host's distro (#30852) 
Fixes #29849.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
 2025-07-15 15:41:42 -05:00
Konstantin Sykulev
6957f84f28
Manual labels no longer factor in created_at time for exclusions (#30745) 
Fixes https://github.com/fleetdm/fleet/issues/29315

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* The "created_at" label no longer affects manual label scoping for
software packages, ensuring more accurate filtering.
* Device authentication tokens are now validated solely by their value,
not by their expiration time.

* **Tests**
* Added new tests to verify label scoping logic, ensuring correct
handling of dynamic and manual labels based on timestamps.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-07-11 12:18:34 -05:00
Ian Littman
5ef6904b13
Skip software_id=0 and log, but otherwise complete counts, when counting host software on a host_software table including rows with software ID zero (#30523) 
Fixes #30522.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [ ] Manual QA for all new/changed functionality

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Resolved an issue where host software counts were not updated if the
database contained rows with a zero software ID.

* **Tests**
* Enhanced tests to verify correct handling of host software records
with a zero software ID.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-07-02 13:47:51 -05:00
Konstantin Sykulev
f008d72107
available_for_install false hides uninstalled software (#30404) 
https://github.com/fleetdm/fleet/issues/30188

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
 2025-07-01 11:08:15 -05:00
Jahziel Villasana-Espinoza
4c5432c7eb
vpp verify followup (#30389) 
> Fixes #29851
> Fixes #29902
> Mainly followups from https://github.com/fleetdm/fleet/pull/30295,
plus improved integration testing

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
 2025-07-01 11:19:42 -04:00
Jahziel Villasana-Espinoza
0c4af0b985
Verify VPP: core implementation (#30295) 
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- For database migrations:
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
 2025-06-26 17:55:43 -04:00
Konstantin Sykulev
1c2db0007f
Host installed vulnerable vpp apps (#29369) 
If the user has a software installed on their machine that happens to
match a vpp app in our catalog. When searching by vulnerable attributes
do not link that software to a vpp app. Just treat it like a non-fleet
installed application.

https://github.com/fleetdm/fleet/issues/29308


- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
 2025-05-29 09:37:10 -05:00
Konstantin Sykulev
a42167462f
Added SHA256 hash from mac apps on install paths (#29280) 
https://github.com/fleetdm/fleet/issues/25545

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Make sure fleetd is compatible with the latest released version of
Fleet (see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md)).
- [x] Orbit runs on macOS, Linux and Windows. Check if the orbit
feature/bugfix should only apply to one platform (`runtime.GOOS`).
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
 2025-05-20 23:38:59 -05:00
Konstantin Sykulev
a2a66af6e4
Fleet installed vpp apps, prior to being inventoried, correctly pull self service flag (#29272) 
https://github.com/fleetdm/fleet/issues/29240

Prior to being accounted for in software inventory we store vpp app
installs in `host_vpp_software_installs`. The query pull data from this
table was not getting self service flag information which was necessary
for label scoping logic later on while fetching software available for
install on the service service page.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
 2025-05-20 10:44:27 -05:00
Konstantin Sykulev
1797120639
Non fleet installed software properly filtered by labels (#29116) 
https://github.com/fleetdm/fleet/issues/28328

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
 2025-05-13 18:41:33 -05:00
Konstantin Sykulev
e134691da3
Handled installed vpp apps when mdm is turned off (#29063) 
https://github.com/fleetdm/fleet/issues/28985

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
 2025-05-13 09:09:39 -05:00
Konstantin Sykulev
ec9f6c3947
Handled fleet installed & inventoried vpp app versions (#29051) 
https://github.com/fleetdm/fleet/issues/28985

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
 2025-05-12 22:20:33 -05:00
Konstantin Sykulev
e374972133
Ensured vpp apps return installed version if exists (#29007) 
https://github.com/fleetdm/fleet/issues/28985

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
 2025-05-09 15:11:27 -05:00
Konstantin Sykulev
4d754aacbb
Fixed search with self service filter (#28991) 
For https://github.com/fleetdm/fleet/issues/28880

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.
 2025-05-09 14:52:11 -05:00
Konstantin Sykulev
b2c1ddaafa
Rewrite label scoping logic for listing host software, fix other ListHostSoftware bugs (#28592) 
https://github.com/fleetdm/fleet/issues/28328

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
- [x] For unreleased bug fixes in a release candidate, confirmed that
the fix is not expected to adversely impact load test results or alerted
the release DRI if additional load testing is needed.

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
 2025-05-06 12:32:35 -05:00
Tim Lee
14967172a7
Downgrade sotware paths error (#28736) 2025-05-05 14:47:59 -06:00