#20934
This is tied to https://github.com/fleetdm/vulnerabilities/pull/14; for
supported OS versions (currently Amazon Linux 1/2/2022/2023) we'll pull
XZ'd sqlite files from the vulnerabilities repo and query them to
determine what's vulnerable. See the associated issue for how I
self-QA'd this.
This replaced OVAL parsing for Amazon Linux 2, as we were using the
wrong data source there (Amazon has backported a bunch of fixes to their
own-named releases, so any RHEL fixes don't match).
Some checklist items are missing here; getting this set up in draft to
get code feedback now, and I'll push updates with e.g. docs changes, as
well ass an addition to the changes file.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Add tests to oval_platform
- [x] Add sync_test
- [x] Add database_test
- [x] Manual QA for all new/changed functionality
- [x] Update vulnerability management docs
We may need more effort to alert on this in a place where we can
see/action it, but for that alerting we can just catch warn (or
warn-and-above) logs, as I just ran the artifact build command and
didn't get any warnings.
Confirmed that this works by starting without the "es" case (since I
wasn't sure which language tags other than "en" and "en-US" we were
seeing) and seeing a bunch of CVEs from 2004 with "es" language tags
reported as warnings. Also confirmed (`gzcat
cvefeed/nvdcve-1.1-2024.json.gz | grep -A 5 -B 5 "description_data"`)
that language tags were correctly set in the resulting feed (caught a
bug locally due to Golang having an implied break at the end of each
switch case).
# Checklist for submitter
- [x] Manual QA for all new/changed functionality
#21242
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
QA'd manually (see repro scenario in the linked bug). Happy to dig into
adding tests here if it's worth the time to build them now that the
upstream data feed has been patchd.
#20463 and #21173
- During vulnerability scanning, use 'macos' SW target for CPEs of
homebrew packages
- During vulnerability scanning, don't ignore software with non-ASCII en
dash and em dash characters
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#21239
This PR fixes the generated vuln JSON files, which would get the current
customers up and running.
QA Done:
- New vuln JSON files generated at:
https://github.com/getvictor/vulnerabilities/releases
- Ran local server with new vuln JSON files by setting `export
TEST_VULN_GITHUB_OWNER=getvictor`
- Diff of JSON files from https://github.com/fleetdm/vulnerabilities and
https://github.com/getvictor/vulnerabilities
Steps for diff:
```
mkdir new
cd new
gh release download cve-202408111650 -D . -R getvictor/vulnerabilities
gunzip *.gz
cd ../
mkdir old
cd old
gh release download cve-202408111637 -D . -R fleetdm/vulnerabilities
gunzip *.gz
cd ..
diff old new
```
Diff results also remove a few Rejected CVEs from JSON files. The 2024
results also have a few minor diffs that don't seem significant.
#18913
This is a test to ensure Rejected CVEs do not match against software.
The related PR needs to be merged first, otherwise this test will fail:
https://github.com/fleetdm/fleet/pull/19972
- [X] Added/updated tests
#19853
Fixing Homebrew intellij/pycharm CE false negative.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
`go-kit/kit/log` was deprecated and generating warnings
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Manual QA for all new/changed functionality
#18733
Added a rule for the affected CVEs and made a change to the `IfIgnore`
func. Manually tested CVE removals with `/tools/nvd/nvdvuln`. Existing
test coverage ensures the rule doesn't cause false negatives on vscode
extensions.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [] Added/updated tests
- [X] Manual QA for all new/changed functionality
#18126
Ignoring Valve Corporation's Steam client's vulnerabilities on Windows
and macOS
- On Windows and macOS, the true version of the Steam client (like
2021-04-10) cannot be retrieved by standard methods used on other
software. We would need to create custom logic to retrieve the version
of the Steam client.
- Steam client automatically updates itself, so security risk is
somewhat mitigated.
QA note: Items from software_cve table are cleared after 2 hours.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
#17061
TODO: Need to also merge this fix into patch branch.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#17003
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [X] Manual QA for all new/changed functionality
#15176
This was already fixed earlier in the sprint by
https://github.com/fleetdm/fleet/pull/15187
Adding docs and a test.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
#14888
@getvictor This is ready for review, but keeping as draft as there are
probably many tests that need amending.
I used the new version of the `./tools/nvd/nvdvuln/nvdvuln.go` to
compare the current vulnerabilities found in our dogfood environment
with the vulnerabilities found by the code in this PR and both results
match:
```
go run -race -tags fts5 ./tools/nvd/nvdvuln/nvdvuln.go --debug --db_dir ./local --software_from_url <dogfood URL> --software_from_api_token <API_TOKEN> --sync 2>&1 | tee out.txt
[...]
CVEs found and expected matched!
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: Victor Lyuboslavsky <victor@fleetdm.com>
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
Loom explaining changes (hit 5 min limit):
https://www.loom.com/share/e59b63bf638e4d9cad7984ef589b878d?sid=111fff75-115a-4a44-ae4f-6f25fede0d51#14887
- [x] Need to merge fleetdm/nvd PR
https://github.com/fleetdm/nvd/pull/25 before this one.
# Checklist for submitter
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Manually tested (with corresponding fleetdm/fleet changes) in my
personal fork: https://github.com/getvictor/nvd/releases
# QA Plan (must be done before merging this PR, and after merging the
nvd PR)
- [ ] Fork https://github.com/fleetdm/nvd and point `generate.yml` to
this branch.
[example](9d8e54930b/.github/workflows/generate.yml (L26))
- [ ] Add NVD_API_KEY to nvd secrets, and run the the nvd generate
GitHub action. Get key:
https://nvd.nist.gov/developers/request-an-api-key
- [ ] Compare the generated `cpe-###.sqlite.gz` to the previous one. One
way is to open it up with sqlite3 and `select * from cpe_2 order by
cpe23;` and dump results to a CSV file. Known differences are:
- New file has ~2,500 more records
- Backslashes are handled differently for `Backpack\CRUD` and `Philips
In.Sight B120\37` products -- not a new issue since we do not support
those products right now
- `cpe:2.3🅰️moodle:moodle:4.2.0:*:*:*:*:*:*:*` -- this appears OK.
Also, it is a PHP plugin, and we don't support these currently.
- [ ] Record the existing vulnerabilities of current hosts.
- [ ] Stop any running fleet server. Delete `/tmp/vulndbs/cpe.sqlite`.
Can also delete other files there, or not delete this file -- it should
be overwritten by the new file. Also delete all rows in software_cpe and
software_cve DB tables. (Or can just spin up a fresh fleet server with
fresh DB, and re-enroll hosts (after setting the new env variable
below))
- [ ] Find the path to the generated `cpe-###.sqlite.gz` file
- [ ] Set `FLEET_VULNERABILITIES_CPE_DATABASE_URL` environment variable
to the above path, and start fleet server.
- [ ] After server's vulnerabilities cron job runs, the new
vulnerabilities should match the previous vulnerabilities
#15143 and #15162
Previous fix for #13889 caused false positives on software with similar
names. Tightening the matching to reduce false positive rate.
- Google Chrome Helper.app no longer matches Google Chrome.app
- Acrobat Uninstaller.app no longer matches Acrobat.app
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- Fix many vulnerability false positives on Chrome and Firefox
extensions. This is related to #11924 and also related to the following
5 false positives found in dogfood around Chrome extensions.
- Reduced `TestTranslateCPEToCVE` run time considerably:
```
--- PASS: TestTranslateCPEToCVE (8.59s)
--- PASS: TestTranslateCPEToCVE/find_vulns_on_cpes (38.45s)
--- PASS: TestTranslateCPEToCVE/recent_vulns (62.03s)
vs.
=== RUN TestTranslateCPEToCVE/recent_vulns
=== NAME TestTranslateCPEToCVE
nettest.go:36: network test done: TestTranslateCPEToCVE
--- PASS: TestTranslateCPEToCVE (269.86s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️1password:1password:3.9.9:*:*:*:*:macos:*:* (31.31s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️1password:1password:3.9.9:*:*:*:*:*:*:* (29.00s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️pypa:pip:9.0.3:*:*:*:*:python:*:* (52.59s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️mozilla:firefox:93.0:*:*:*:*:windows:*:* (34.29s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️mozilla:firefox:93.0.100:*:*:*:*:windows:*:* (28.17s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️apple:icloud:1.0:*:*:*:*:macos:*:* (28.44s)
--- PASS: TestTranslateCPEToCVE/recent_vulns (56.95s)
```
<img width="868" alt="Screenshot 2023-10-30 at 10 21 42"
src="https://github.com/fleetdm/fleet/assets/2073526/284a2373-09bc-44f7-952b-1e53650232ff">
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Adding vulnerability data support for JetBrains applications (like IDEA,
PyCharm, etc.) that have similar names.
- For example: IntelliJ IDEA.app and IntelliJ IDEA Ultimate.app
Resolves#13889
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#14543
Python's certifi package is ingested with the following version string:
`2023.7.22`.
The NVD dataset uses the following versioning: `2023.07.22`.
This PR updates the nvdtools package.
This is the fix in nvdtools that fixes this exact false positive:
c0d18738cd
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
#13615
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#11923
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [x] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
For #13715, this:
- Upgrades the Go version to `1.21.1`, infrastructure changes are
addressed separately at https://github.com/fleetdm/fleet/pull/13878
- Upgrades the linter version, as the current version doesn't work well
after the Go upgrade
- Fixes new linting errors (we now get errors for memory aliasing in
loops! 🎉 )
After this is merged people will need to:
1. Update their Go version. I use `gvm` and I did it like:
```
$ gvm install go1.21.1
$ gvm use go1.21.1 --default
```
2. Update the local version of `golangci-lint`:
```
$ go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.54.2
```
3. (optional) depending on your setup, you might need to re-install some
packages, for example:
```
# goimports to automatically import libraries
$ go install golang.org/x/tools/cmd/goimports@latest
# gopls for the language server
$ go install golang.org/x/tools/gopls@latest
# etc...
```
#11922
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Related to #9386 - this should fix one of the three reported problems.
* Add the ability to add exclusion rules to cpe_translations.
* Added exclusion rule for Docs chrome extension.
* WIP
* Add more logging
* Check rate limit at end of action
* Add github client in more places
* Add new published firefox 93 vulnerabilities to tests
* Remove fmt printfs
* Restore CI check settings
* Readd newline
