4bed761f77
for #21304 # Checklist for submitter - [X] Manual QA for all new/changed functionality ## Details This PR adds a new validator for NVD feed files to be run as part of the nvd repo workflow. The intention is for that workflow to fail if any of the files it creates are not valid (i.e. they would not be parseable by the Fleet server) so that we don't publish and tag a release with bad files in it. This follows the pattern from https://github.com/fleetdm/fleet/issues/21300 as suggested by @iansltx. ## Testing I downloaded all of the latest release files to my local system using ```bash gh release download 202505190037 -D ~/Downloads/nvd ``` and then ran the validator on them with ```bash go run cmd/cpe/validate/main.go --db_dir ~/Downloads/nvd ``` To simulate file issues, I modified one section of each file to change a value into the wrong type, and validated that this caused the validator to panic. Examples: ``` panic: failed to load CPE translations: decode json: json: cannot unmarshal string into Go struct field CPETranslation.filter.vendor of type []string goroutine 1 [running]: main.checkCPETranslations({0x16dc975f9?, 0x14000192190?}) /Users/scott/Development/fleet/cmd/cpe/validate/main.go:34 +0xa8 main.main() /Users/scott/Development/fleet/cmd/cpe/validate/main.go:24 +0xb0 exit status 2 ``` --- ``` panic: failed to parse MacOffice release notes fleet_macoffice_release_notes_macoffice-2025_05_19.json: parsing time "xyz" as "2006-01-02T15:04:05Z07:00": cannot parse "xyz" as "2006" goroutine 1 [running]: main.checkMacOfficeNotes({0x16f7af5f9, 0x1a}) /Users/scott/Development/fleet/cmd/cpe/validate/main.go:56 +0x1f0 main.main() /Users/scott/Development/fleet/cmd/cpe/validate/main.go:25 +0xbc exit status 2 ``` --- ``` panic: failed to parse MSRC feed fleet_msrc_Windows_Server_2012_R2-2025_05_19.json: json: cannot unmarshal array into Go struct field Vulnerability.Vulnerabities.RemediatedBy of type bool goroutine 1 [running]: main.checkMSRCVulnerabilities({0x16f49b5f9, 0x1a}) /Users/scott/Development/fleet/cmd/cpe/validate/main.go:74 +0x1ac main.main() /Users/scott/Development/fleet/cmd/cpe/validate/main.go:26 +0xc8 exit status 2 ``` Additionally I tried the validator in [a run of the NVD workflow](https://github.com/fleetdm/nvd/actions/runs/15121687898/job/42505283781) and it executed successfully. |
||
|---|---|---|
| .. | ||
| sync | ||
| tools | ||
| cpe.go | ||
| cpe_matching_rule.go | ||
| cpe_matching_rule_test.go | ||
| cpe_matching_rules.go | ||
| cpe_test.go | ||
| cpe_translations.go | ||
| cpe_translations.json | ||
| cve.go | ||
| cve_test.go | ||
| db.go | ||
| indexed_cpe_item.go | ||
| README.md | ||
| sanitize.go | ||
| sanitize_test.go | ||
| sync.go | ||
| sync_test.go | ||
| testing_utils.go | ||
CPE Translations
CPE Translations are rules to address bugs when translating Fleet software to Common Platform Enumerations (CPEs) which are used to identify software in the National Vulnerability Database (NVD)
To improve accuracy when mapping software to CVEs, we can add data to cpe_translations.json
How CPE translations work
CPE Translations are defined in cpe_translations.json and currently released in
GitHub once a day. The rules are specified in JSON format and
and each rule consists of a software and a filter object.
software defines matching logic on what Fleet Software this rule should apply to. You can use one
or more of the below attributes to match on. Each attribute is an array of string or regex
matches (a regex string is identified by a leading and trailing /).
A match on the attribute is found if at least 1 item in the array matches. If multiple
attributes are defined, then a match is needed for each attribute. (ie. name == Zoom.app &&
source == apps)
software attributes:
name: A software name attributebundle_identifier: A software bundle_identifier attribute (macOS only)source: A software source attribute (ie.apps,chrome_extensions, etc...)
example: Search Fleet software for items that match: (bundle_identifier == us.zoom.xos) AND (source = apps)
"software": {
"bundle_identifier": ["us.zoom.xos"],
"source": ["apps"]
}
If the software rule matches, then Fleet will search known NVD CPEs (stored in a local sqlite database) using the
specified filters or skip the software item based on the filter specified.
filter attributes:
product: array of strings to search by product fieldvendor: array of strings to search by vendor fieldtarget_sw: array of strings to search by target_sw fieldpart: string to override the default "a" Part valueskip: boolean; software is skipped iftrue. This overrides any other filters set.
Like the software matching logic, filter items are matched by OR within the array, and AND between filter items
example: Query the CPE database for a CPE that matches: (product == zoom OR product == meetings) AND (vendor == zoom) AND (target == macos OR target == mac_os)
"filter": {
"product": ["zoom", "meetings"],
"vendor": ["zoom"],
"target_sw": ["macos", "mac_os"]
}
Testing CPE Translations (end-to-end)
-
make the appropriate changes to cpe_translations
-
host this file on a local web server
go run ./tools/file-server/main.go 8082 ./server/vulnerabilities/nvd/ -
(re)launch your local fleet server with one of the following
Config method
vulnerabilities: cpe_translations_url: "http://localhost:8082/cpe_translations.json"Environment method
FLEET_VULNERABILITIES_CPE_TRANSLATIONS_URL="http://localhost:8082/cpe_translations.json" ./build/fleet serve --dev --dev_license --logging_debug -
trigger a vulnerabilities scan
fleetctl trigger --name vulnerabilities