Commit graph

200 commits

Author SHA1 Message Date
Tim Lee
09235486b4
Process all vulncheck data (#24318) 2024-12-18 10:53:46 -07:00
Scott Gress
edc68d3042
Add versions to product names in MSRC bulletins to aid Windows vulnerability matching (#24172)
for #24041 

This PR addresses an issue that can cause Windows vulnerability checks
to fail (possibly causing false negatives). We determine whether a
vulnerability in an MSRC bulletin applies to any hosts in a Fleet
instance by attempting to matching the data in [each row of the
`operating_systems`
table](65e374c85c/cmd/fleet/cron.go (L297-L303))
with [at least one "product" in a
bulletin](e2d9a9016c/server/vulnerabilities/msrc/analyzer.go (L39)),
including [matching architecture and "display
version"](76f5baced9/server/vulnerabilities/msrc/parsed/product.go (L26-L39)).
However a subset of products listed in these bulletins do not include
the display version, so for example a host whose OS was listed as
`Microsoft Windows Server 2022 Datacenter 21H2` (21H2 being the "display
version") would match nothing in the bulletins because no listed Server
2022 products include "21H2" in their names.

The fix made here is to add relevant version info to the products list
when we do our ETL of the MSRC bulletins. The version info was gleaned
from https://en.wikipedia.org/wiki/List_of_Microsoft_Windows_versions.

We see logs related to this issue a lot, so cleaning this up will
alleviate some noise and infra costs as well.
2024-12-17 09:46:03 -06:00
Tim Lee
47a43850f0
Skip python vulnerabilities test (#24287) 2024-12-02 14:33:03 -07:00
Konstantin Sykulev
1446d28029
github cli false negative vulnerability (#24100)
Added a cpe translation for the `gh` command. The software is identified
as `gh`, however, the cpe (`cpe:2.3🅰️github:cli:2.62.0:*:*:*:*:*:*:*`)
name is labeled as `cli`, thus the mismatch.

https://github.com/fleetdm/fleet/issues/24009

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-11-25 12:32:10 -06:00
RachelElysia
79a3549e67
Quick spelling/grammar fixes (#23859)
I noticed a spelling issue and a grammar issue and fixed similar ones
elsewhere in our docs

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->
2024-11-18 13:36:59 -06:00
Jahziel Villasana-Espinoza
6a26b894b3
fix: update test with new vulns (#23775)
> No ticket, just fixing a broken test

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Added/updated tests
2024-11-13 14:15:31 -05:00
Victor Lyuboslavsky
3cf99f8039
Enable staticcheck Go linter. (#23487)
#23486 

Linter: https://staticcheck.dev/
2024-11-05 11:16:24 -06:00
Victor Lyuboslavsky
e2d9a9016c
Add gosimple linter (#23250)
#23249

Add gosimple linter to golangci-lint CI job.
2024-10-29 14:17:51 -05:00
Victor Lyuboslavsky
f85b6f776f
Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
Ian Littman
de537692a1
Update MSRC feed URL in test (#22494)
Whoops, _this_ was the failing test from earlier. Related to #22492
#22493.

- [x] Added/updated tests
2024-09-28 09:18:55 -05:00
Ian Littman
e8a6456f13
Don't reuse GitHub HTTP client to pull MSRC feeds (#22493)
See #22492

As of today, MSRC's API will 401 if you pass it a JWT it doesn't expect,
and by reusing the GitHub API HTTP client for MSRC pulls we were passing
the API an unexpected JWT. Wasn't able to reproduce this locally because
I didn't need a GitHub token locally to pull release details, while the
token is populated in Actions. Was able to repro both the issue and this
fix inside Actions on my fork.

This also updates to call v3.0 of the API directly, which v2.0 has been
redirecting to for awhile.

Finally, adds slightly better logging so we know which part of the feed
generation process we're in when we're running this in Actions.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
2024-09-27 21:23:48 -05:00
Lucas Manuel Rodriguez
f52e0a0e8c
Fix to not panic if MSRC for current month is not available (#21749)
See #21745.

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
2024-09-27 16:59:43 -05:00
Tim Lee
839106c572
Hotfix CVE test (#22349) 2024-09-24 17:45:37 -06:00
Tim Lee
1deccdb68d
Add cpe translation for macos vbox (#22082) 2024-09-13 10:57:50 -06:00
Tim Lee
419433fb44
Homebrew git false negative vulnerability (#22002) 2024-09-13 06:00:12 -06:00
Tim Lee
8542a823a7
macOS VirtualBox false negative (#21646) 2024-09-04 14:22:36 -06:00
Ian Littman
e96c70e4c0
Pull xz'd goval-dictionary sqlite files to evaluate vulnerabilities on Amazon Linux hosts (#21506)
#20934

This is tied to https://github.com/fleetdm/vulnerabilities/pull/14; for
supported OS versions (currently Amazon Linux 1/2/2022/2023) we'll pull
XZ'd sqlite files from the vulnerabilities repo and query them to
determine what's vulnerable. See the associated issue for how I
self-QA'd this.

This replaced OVAL parsing for Amazon Linux 2, as we were using the
wrong data source there (Amazon has backported a bunch of fixes to their
own-named releases, so any RHEL fixes don't match).

Some checklist items are missing here; getting this set up in draft to
get code feedback now, and I'll push updates with e.g. docs changes, as
well ass an addition to the changes file.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
    - [x] Add tests to oval_platform
    - [x] Add sync_test
    - [x] Add database_test
- [x] Manual QA for all new/changed functionality
- [x] Update vulnerability management docs
2024-08-26 14:07:42 -05:00
Victor Lyuboslavsky
fdfc12982b
Improvements to go tests in CI (#21545)
#21546 
Some improvements to overall go test CI run time.
2024-08-26 08:55:53 -05:00
Tim Lee
e376d3ddbb
Python Vulnerability False Positive (#21493) 2024-08-23 11:59:25 -06:00
Lucas Manuel Rodriguez
415cccc121
Add matching rules for Microsoft 365 for July and August 365 (#21410)
#20409

I used `Current Channel`'s build version from
[here](https://learn.microsoft.com/en-us/officeupdates/microsoft365-apps-security-updates)
for `ResolvedInVersion`. Please @mostlikelee let me know if that's a-ok
(mimicked from June's change).

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
2024-08-20 11:35:44 -03:00
Ian Littman
605779eee3
Warn on CVE description language tags from NVD feed that we haven't seen before (#21310)
We may need more effort to alert on this in a place where we can
see/action it, but for that alerting we can just catch warn (or
warn-and-above) logs, as I just ran the artifact build command and
didn't get any warnings.

Confirmed that this works by starting without the "es" case (since I
wasn't sure which language tags other than "en" and "en-US" we were
seeing) and seeing a bunch of CVEs from 2004 with "es" language tags
reported as warnings. Also confirmed (`gzcat
cvefeed/nvdcve-1.1-2024.json.gz | grep -A 5 -B 5 "description_data"`)
that language tags were correctly set in the resulting feed (caught a
bug locally due to Golang having an implied break at the end of each
switch case).

# Checklist for submitter

- [x] Manual QA for all new/changed functionality
2024-08-19 10:42:24 -05:00
Ian Littman
49300bc844
Don't panic on zero-length NVD description_data array fields (#21250)
#21242

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

QA'd manually (see repro scenario in the linked bug). Happy to dig into
adding tests here if it's worth the time to build them now that the
upstream data feed has been patchd.
2024-08-14 10:53:47 -05:00
Victor Lyuboslavsky
488115d366
CPE fixes (#21244)
#20463 and #21173
- During vulnerability scanning, use 'macos' SW target for CPEs of
homebrew packages
- During vulnerability scanning, don't ignore software with non-ASCII en
dash and em dash characters

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-08-12 16:14:25 +02:00
Victor Lyuboslavsky
1176afbb0b
Vuln cron fail fix (#21240)
#21239 

This PR fixes the generated vuln JSON files, which would get the current
customers up and running.

QA Done:

- New vuln JSON files generated at:
https://github.com/getvictor/vulnerabilities/releases
- Ran local server with new vuln JSON files by setting `export
TEST_VULN_GITHUB_OWNER=getvictor`
- Diff of JSON files from https://github.com/fleetdm/vulnerabilities and
https://github.com/getvictor/vulnerabilities

Steps for diff:
```
mkdir new
cd new
gh release download cve-202408111650 -D . -R getvictor/vulnerabilities
gunzip *.gz
cd ../
mkdir old
cd old
gh release download cve-202408111637 -D . -R fleetdm/vulnerabilities
gunzip *.gz
cd ..
diff old new
```

Diff results also remove a few Rejected CVEs from JSON files. The 2024
results also have a few minor diffs that don't seem significant.
2024-08-11 20:57:29 +02:00
Victor Lyuboslavsky
647ffc6bf1
Exclude ios_apps and ipados_apps from normal vuln processing. (#21143)
#21142
2024-08-07 19:46:33 +02:00
Victor Lyuboslavsky
e0d36366fe
Moving Go integration tests to integration test job (#21126)
#20929
Moving Go integration tests to integration test job, which runs tests
that start with `TestIntegrations`
Test changes only. No product changes
2024-08-07 14:00:25 +02:00
Tim Lee
e3c3c870c9
Ignore Rejected CVEs Test (#19974)
#18913

This is a test to ensure Rejected CVEs do not match against software.
The related PR needs to be merged first, otherwise this test will fail:

https://github.com/fleetdm/fleet/pull/19972

- [X] Added/updated tests
2024-08-02 07:33:57 -03:00
Tim Lee
362a0e545e
18913 ignore rejected NVD vulnerabilities (#20193)
#18913 Recreating PR (ref: https://github.com/fleetdm/fleet/pull/19972)
2024-07-24 10:53:33 -06:00
Victor Lyuboslavsky
5cc5e527e5
Fixing Homebrew intellij/pycharm CE false negative. (#20461)
#19853
Fixing Homebrew intellij/pycharm CE false negative.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-07-15 12:47:10 -05:00
Tim Lee
2ea369c093
Custom Vulnerability Matching (#20118) 2024-07-09 11:50:22 -06:00
Tim Lee
587a42f2a3
Revert "Ignore Rejected CVEs" (#20183) 2024-07-03 05:58:05 -06:00
Tim Lee
47b7794fe8
Ignore Rejected CVEs (#19972) 2024-07-02 18:42:38 -06:00
Tim Lee
4dce1f6623
Windows Vuln False Positives (#20012) 2024-06-26 09:43:57 -06:00
Tim Lee
cb5fb65c5e
fix app vuln false positives (#19905) 2024-06-21 08:05:18 -06:00
Tim Lee
2d6458b3f8
Custom Ubuntu Kernel Vuln Scanning (#19588) 2024-06-17 15:44:01 -06:00
Tim Lee
b137bcbef2
Hotfix - Vuln matching race (#19786) 2024-06-17 11:02:47 -06:00
Roberto Dip
1cc13a09fb
🧹 friday cleanup party: substitute deprecated import of go-kit (#19774)
`go-kit/kit/log` was deprecated and generating warnings

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Manual QA for all new/changed functionality
2024-06-17 10:27:31 -03:00
Tim Lee
0516cd61d0
App OS Vulnerability Matching (#19486) 2024-06-13 11:20:28 -04:00
Tim Lee
6d2471d5e5
vscode false positivies (#19508)
#18733 

Added a rule for the affected CVEs and made a change to the `IfIgnore`
func. Manually tested CVE removals with `/tools/nvd/nvdvuln`. Existing
test coverage ensures the rule doesn't cause false negatives on vscode
extensions.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [] Added/updated tests
- [X] Manual QA for all new/changed functionality
2024-06-13 11:03:53 -04:00
Roberto Dip
fa3f87a9a5
skip vuln tests until API keys are restored (#19446) 2024-06-03 14:06:41 -03:00
Tim Lee
d1db2e3650
Ubuntu Kernel Vulns Part 2: Matching (#19303) 2024-05-29 06:59:12 -06:00
Tim Lee
14fa3e2cbe
Ubuntu Kernel Vulns Part 1: Generate OVAL JSON (#19210) 2024-05-28 13:40:32 -06:00
Tim Lee
6f008b40f2
Fix backoff in vuln stream logging (#19148) 2024-05-28 08:13:46 -06:00
Tim Lee
7bba061ad2
Bugfix: Use hyphen delimiter in resolved_in_version (#17683) 2024-05-09 06:49:55 -06:00
Tim Lee
536ccc91c0
18601 add ubuntu sources (#18602) 2024-05-02 13:20:00 -06:00
Tim Lee
648628d3be
Update Windows OS Version Reporting (#17682) 2024-05-01 12:02:16 -06:00
Jacob Shandling
d7821685a8
Migrate logic from nvdtools into Fleet (#18244)
## Addresses #15999

- [x] Added/updated tests
- [ ] Manual QA for all new/changed functionality

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
2024-04-24 15:25:59 -07:00
Tim Lee
9e1878e01c
18439 Vulncheck data processing bug (#18440) 2024-04-22 09:36:20 -06:00
Victor Lyuboslavsky
72cc82f60d
Ignoring Valve Corporation's Steam client's vulnerabilities on Windows and macOS (#18380)
#18126
Ignoring Valve Corporation's Steam client's vulnerabilities on Windows
and macOS
- On Windows and macOS, the true version of the Steam client (like
2021-04-10) cannot be retrieved by standard methods used on other
software. We would need to create custom logic to retrieve the version
of the Steam client.
- Steam client automatically updates itself, so security risk is
somewhat mitigated.

QA note: Items from software_cve table are cleared after 2 hours.

# Checklist for submitter

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-04-18 10:12:49 -05:00
Tim Lee
8194b6e379
Optimize cve/generate to use last release (#18269) 2024-04-15 13:17:28 -06:00
Tim Lee
a47f59ef36
Implement NVD Feed with Vulncheck Data (#18168) 2024-04-15 09:44:44 -06:00
George Karr
e73fd55c5b
Updating vuln check support (#15174) 2024-04-10 10:04:50 -05:00
Roberto Dip
f1b45a3865
friday tidy up party (#18106)
- add missing tests
- remove unused code
- remove unnecessary nested branches
2024-04-08 08:42:42 -03:00
Victor Lyuboslavsky
759003e37d
Fixing false negative vulnerabilities on macOS Homebrew python packages. (#17709)
#17061

TODO: Need to also merge this fix into patch branch.

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-03-19 14:12:07 -05:00
Lucas Manuel Rodriguez
cf64d85deb
Add visual studio extensions to software inventory (#17501)
#17003

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [X] Manual QA for all new/changed functionality
2024-03-14 16:33:12 -03:00
Tim Lee
8be1d4766f
MS Edge Translation (#17171) 2024-03-13 12:47:31 -06:00
Victor Lyuboslavsky
7bbb4a64bb
Reduced vulnerability processing time by grouping vulnerability dictionary by vendor. (#17298)
Reduced vulnerability processing time by grouping vulnerability
dictionary by vendor.
#16858 

Notice `cvefeed.(*Cache).Get`
Before:
<img width="1997" alt="image"
src="https://github.com/fleetdm/fleet/assets/2685025/a3cc5571-cee2-40a6-8b2d-cfed1323d96c">
After:

![image](https://github.com/fleetdm/fleet/assets/2685025/89484f87-a4e9-4c5c-943f-138e1c27a2c6)


# Checklist for submitter

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-03-04 16:16:32 -06:00
Tim Lee
7727978cb4
Skip nonASCII Software Names in Vuln Processing (#17193) 2024-03-01 10:31:46 -07:00
Jacob Shandling
1cf7362968 Merge branch 'main' into 15919-vulnerabilities-page 2024-02-21 10:58:58 -08:00
Victor Lyuboslavsky
35ca4ee32b
Add team filter to software detail APIs (#16876)
#16787
2024-02-18 07:14:20 -06:00
Gabriel Hernandez
119d1df76f
add permission check to software titles/versions endpoints (#16561)
relates to #16052

This adds a team permission check the `GET software/titles/:id`
endpoint. If the user should not be able to get the software title if it
is not on a host that is on the same team as the user (e.g. software
title 1 is on host 1, which is on team 1. A user who is only on team 2
should get a 403 response)

The UI is also updated to show the access denied error page when the we
receive a 403 response for the software title

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
Co-authored-by: Roberto Dip <me@roperzh.com>
2024-02-15 17:22:27 -03:00
Tim Lee
6d1eee6279
Add resolved_in_version for Windows OS Vulns (#16546) 2024-02-07 06:19:59 -07:00
Victor Lyuboslavsky
66ec651ec4
Switching from FixedBuild to FixedBuilds (part 2). (#16465)
Switching from FixedBuild to FixedBuilds (part 2).
#16412
2024-01-30 14:24:41 -06:00
Victor Lyuboslavsky
72705ef93a
Switching from FixedBuild to FixedBuilds (part 1). (#16454)
Switching from FixedBuild to FixedBuilds (part 1).
#16412 

Converting msrc files from using FixedBuild to FixedBuilds. In part 2,
FixedBuild will be completely removed.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
2024-01-30 10:43:28 -06:00
Tim Lee
ae87519938
16356 Unreleased bugfix for osversions filter (#16391) 2024-01-29 13:04:26 -07:00
Tim Lee
79b5baa297
4345 OS Vulnerabilities Backend (#16303)
#4345 

This backend feature branch includes the following PRs:

macOS Vuln Matching:
#15837 
#15990 
#16077 

Bugs / Issues:
#16004
 #15905 
#16226 

Windows Vuln Matching
#16047 
#16049 
#16085 
#16099 

API:
#16215
2024-01-24 12:18:57 -07:00
Tim Lee
26c070eb00
fix vulnerabilities resolved in value (#15905) 2024-01-11 09:22:35 -07:00
Tim Lee
27073720c5
15254 oval scan err (#15499) 2023-12-14 10:39:28 -07:00
Tim Lee
b2568ad475
HotFix - ambiguous policy search name (#15312) 2023-11-27 12:21:39 -07:00
Victor Lyuboslavsky
fbbd81ff77
UmbrellaMenu.app no longer matches Cisco Umbrella (#15262)
#15176 
This was already fixed earlier in the sprint by
https://github.com/fleetdm/fleet/pull/15187
Adding docs and a test.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
2023-11-21 16:29:02 -06:00
Lucas Manuel Rodriguez
4194c44131
Use NVD API 2.0 to download CVE information (#15102)
#14888

@getvictor This is ready for review, but keeping as draft as there are
probably many tests that need amending.

I used the new version of the `./tools/nvd/nvdvuln/nvdvuln.go` to
compare the current vulnerabilities found in our dogfood environment
with the vulnerabilities found by the code in this PR and both results
match:
```
go run -race -tags fts5 ./tools/nvd/nvdvuln/nvdvuln.go --debug --db_dir ./local --software_from_url <dogfood URL> --software_from_api_token <API_TOKEN> --sync 2>&1 | tee out.txt
[...]
CVEs found and expected matched!
```

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [X] Manual QA for all new/changed functionality

---------

Co-authored-by: Victor Lyuboslavsky <victor@fleetdm.com>
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
2023-11-21 12:30:07 -06:00
Tim Lee
97e88ae32e
hotfix: update CPE tests (#15250) 2023-11-21 10:25:27 -07:00
Victor Lyuboslavsky
eada583ff1
Updating CPE generator to use new NVD API. (#15018)
Loom explaining changes (hit 5 min limit):
https://www.loom.com/share/e59b63bf638e4d9cad7984ef589b878d?sid=111fff75-115a-4a44-ae4f-6f25fede0d51

#14887

- [x] Need to merge fleetdm/nvd PR
https://github.com/fleetdm/nvd/pull/25 before this one.

# Checklist for submitter

- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Manually tested (with corresponding fleetdm/fleet changes) in my
personal fork: https://github.com/getvictor/nvd/releases

# QA Plan (must be done before merging this PR, and after merging the
nvd PR)
- [ ] Fork https://github.com/fleetdm/nvd and point `generate.yml` to
this branch.
[example](9d8e54930b/.github/workflows/generate.yml (L26))
- [ ] Add NVD_API_KEY to nvd secrets, and run the the nvd generate
GitHub action. Get key:
https://nvd.nist.gov/developers/request-an-api-key
- [ ] Compare the generated `cpe-###.sqlite.gz` to the previous one. One
way is to open it up with sqlite3 and `select * from cpe_2 order by
cpe23;` and dump results to a CSV file. Known differences are:
   - New file has ~2,500 more records
- Backslashes are handled differently for `Backpack\CRUD` and `Philips
In.Sight B120\37` products -- not a new issue since we do not support
those products right now
- `cpe:2.3🅰️moodle:moodle:4.2.0:*:*:*:*:*:*:*` -- this appears OK.
Also, it is a PHP plugin, and we don't support these currently.
- [ ] Record the existing vulnerabilities of current hosts.
- [ ] Stop any running fleet server. Delete `/tmp/vulndbs/cpe.sqlite`.
Can also delete other files there, or not delete this file -- it should
be overwritten by the new file. Also delete all rows in software_cpe and
software_cve DB tables. (Or can just spin up a fresh fleet server with
fresh DB, and re-enroll hosts (after setting the new env variable
below))
- [ ] Find the path to the generated `cpe-###.sqlite.gz` file
- [ ] Set `FLEET_VULNERABILITIES_CPE_DATABASE_URL` environment variable
to the above path, and start fleet server.
- [ ] After server's vulnerabilities cron job runs, the new
vulnerabilities should match the previous vulnerabilities
2023-11-20 16:10:00 -06:00
Victor Lyuboslavsky
8ae88cfe1a
Tightening the CPE matching to reduce false positive rate. (#15187)
#15143 and #15162 

Previous fix for #13889 caused false positives on software with similar
names. Tightening the matching to reduce false positive rate.
- Google Chrome Helper.app no longer matches Google Chrome.app
- Acrobat Uninstaller.app no longer matches Acrobat.app

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
2023-11-20 11:59:31 -06:00
Victor Lyuboslavsky
9e3215eb2b
Fixing TestTranslateCPEToCVE (#14931) 2023-11-03 12:58:59 -05:00
Lucas Manuel Rodriguez
ae6c387059
Fix false positive vulnerabilities on Chrome and Firefox extensions (#14791)
- Fix many vulnerability false positives on Chrome and Firefox
extensions. This is related to #11924 and also related to the following
5 false positives found in dogfood around Chrome extensions.
- Reduced `TestTranslateCPEToCVE` run time considerably:
```
--- PASS: TestTranslateCPEToCVE (8.59s)
    --- PASS: TestTranslateCPEToCVE/find_vulns_on_cpes (38.45s)
    --- PASS: TestTranslateCPEToCVE/recent_vulns (62.03s)

vs.

=== RUN   TestTranslateCPEToCVE/recent_vulns
=== NAME  TestTranslateCPEToCVE
    nettest.go:36: network test done: TestTranslateCPEToCVE
--- PASS: TestTranslateCPEToCVE (269.86s)
    --- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️1password:1password:3.9.9:*:*:*:*:macos:*:* (31.31s)
    --- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️1password:1password:3.9.9:*:*:*:*:*:*:* (29.00s)
    --- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️pypa:pip:9.0.3:*:*:*:*:python:*:* (52.59s)
    --- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️mozilla:firefox:93.0:*:*:*:*:windows:*:* (34.29s)
    --- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️mozilla:firefox:93.0.100:*:*:*:*:windows:*:* (28.17s)
    --- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️apple:icloud:1.0:*:*:*:*:macos:*:* (28.44s)
    --- PASS: TestTranslateCPEToCVE/recent_vulns (56.95s)
```

![Screenshot 2023-10-30 at 09 34
25](https://github.com/fleetdm/fleet/assets/2073526/30ff798d-362d-4df4-876e-53619d2d8802)
![Screenshot 2023-10-30 at 09 34
35](https://github.com/fleetdm/fleet/assets/2073526/ae65583f-26d3-403a-93e3-39c1393bc471)
![Screenshot 2023-10-30 at 09 34
47](https://github.com/fleetdm/fleet/assets/2073526/aba99efe-b744-4f05-927c-981c490fc02f)
![Screenshot 2023-10-30 at 09 35
41](https://github.com/fleetdm/fleet/assets/2073526/aaff0725-ceca-494e-b64f-c30ff5e63aec)
<img width="868" alt="Screenshot 2023-10-30 at 10 21 42"
src="https://github.com/fleetdm/fleet/assets/2073526/284a2373-09bc-44f7-952b-1e53650232ff">

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-11-01 09:26:26 -03:00
Victor Lyuboslavsky
160755ad1d
Adding CPE support for different Jetbrains IDEA and PyCharm app names. (#14806)
Adding vulnerability data support for JetBrains applications (like IDEA,
PyCharm, etc.) that have similar names.
- For example: IntelliJ IDEA.app and IntelliJ IDEA Ultimate.app

Resolves #13889 

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-10-31 16:18:24 -05:00
Lucas Manuel Rodriguez
512699614d
Update facebookincubator/nvdtools package to fix false positive (#14798)
#14543

Python's certifi package is ingested with the following version string:
`2023.7.22`.
The NVD dataset uses the following versioning: `2023.07.22`.

This PR updates the nvdtools package.

This is the fix in nvdtools that fixes this exact false positive:

c0d18738cd

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-10-30 17:20:28 -03:00
guangwu
33858d7301
chore: remove refs to deprecated io/ioutil (#14485)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).

Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-27 15:28:54 -03:00
Lucas Manuel Rodriguez
5d827133c2
Add debug log for troubleshooting (#14602)
This PR just adds a debug log for #11924.
2023-10-27 14:14:41 -03:00
Lucas Manuel Rodriguez
2fd6fa4e04
Vulnerability processing should ignore software without version (#14612)
#13615

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-10-26 18:28:08 -03:00
Lucas Manuel Rodriguez
7d13769de6
Fix false positive vulnerability for python on macOS (#14531)
#11923

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [x] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-10-13 14:24:35 -03:00
Tim Lee
338c64d78b
Add version_resolved_in to software API (#13939) 2023-09-18 16:53:32 -06:00
Tim Lee
0bde133831
add CVE-2013-0340 to ignore list (#13942) 2023-09-15 16:38:33 -06:00
Tim Lee
5bc6d30aa8
Add Description text to CVE Metadata (#13856) 2023-09-15 11:24:10 -06:00
Tim Lee
e80ea50240
Flock vulnerability false positive (#13827) 2023-09-14 10:38:36 -06:00
Roberto Dip
ea6b59f179
upgrade Go version to 1.21.1 (#13877)
For #13715, this:

- Upgrades the Go version to `1.21.1`, infrastructure changes are
addressed separately at https://github.com/fleetdm/fleet/pull/13878
- Upgrades the linter version, as the current version doesn't work well
after the Go upgrade
- Fixes new linting errors (we now get errors for memory aliasing in
loops! 🎉 )

After this is merged people will need to:

1. Update their Go version. I use `gvm` and I did it like:

```
$ gvm install go1.21.1
$ gvm use go1.21.1 --default
```

2. Update the local version of `golangci-lint`:

```
$ go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.54.2
```

3. (optional) depending on your setup, you might need to re-install some
packages, for example:

```
# goimports to automatically import libraries
$  go install golang.org/x/tools/cmd/goimports@latest

# gopls for the language server
$ go install golang.org/x/tools/gopls@latest

# etc...
```
2023-09-13 15:59:35 -03:00
Lucas Manuel Rodriguez
31598ab721
Fix CVE-2020-10146 false positive being detected on all Microsoft Teams versions (#13839)
#11922

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-09-11 16:51:53 -03:00
Juan Fernandez
91f048e98a
Fixed iCloud false positives (#12551)
Added new type `CPEMatchingRule` used for fixing false positives caused by 'bad' entries in the NVD dataset.
2023-06-30 13:05:34 -04:00
Juan Fernandez
009a87d33e
Feature 10196: Add filepath to end-points and third party integrations (#11285)
Adds the software installed path property to the proper end-points and third party integrations (webhook, Zendesk and Jira).
2023-05-17 16:53:15 -04:00
Juan Fernandez
7f83135aa1
Feature: Store installed file path when ingesting software (#11214)
Store software installed paths into the host_software_installed_paths table when ingesting osquery software data.
2023-05-17 14:49:09 -04:00
Juan Fernandez
d353a3deb4
Handle flaky vulnerability tests (#11262)
- Refactored some of the vulnerabilities tests to help with flakiness.
- Don't load NVD assets if local assets have a timestamp of today.
2023-04-21 19:37:29 -04:00
Juan Fernandez
4c2ddba2e4
Clean out-of-date NVD results. (#10514)
Keep the vulnerabilities detected via NVD and stored in the DB in sync. with the results from the NVD vulnerability process.
2023-04-03 13:45:18 -04:00
Juan Fernandez
56ed2727b5
Updated translation rules so that Docker Desktop can be mapped to the proper CPE (#10326)
Updated translation rules so that Docker Desktop can be mapped to the proper CPE.
2023-03-09 17:46:57 -04:00
Juan Fernandez
6ff2c449cf
Improve test coverage and bug fix (#10301)
Addresses https://github.com/fleetdm/fleet/issues/10112

Improved test coverage, also fixed a bug related to vulnerability processing for mac office apps.
2023-03-06 15:07:27 -04:00
Juan Fernandez
7e366272c0
Feature 9386: Parse the Mac Office release notes for vulnerability processing (#9993)
This PR adds the capability of parsing the release notes posted in https://learn.microsoft.com/en-us/officeupdates/release-notes-office-for-mac into a JSON metadata file (to be released in the NVD repo) and use it for detecting vulnerabilities on Mac Office apps.
2023-02-24 14:18:25 -04:00
Benjamin Edwards
0db432f7e3
vuln processing distinct command (#9813)
closes https://github.com/fleetdm/fleet/issues/3723

Add new vuln processing command, configs, and documentation on how to utilize.
2023-02-17 10:00:57 -05:00
Juan Fernandez
e376d53294
Bug 9386: Docs chrome extension has the wrong CPE. (#9524)
Related to #9386 - this should fix one of the three reported problems.

* Add the ability to add exclusion rules to cpe_translations.
* Added exclusion rule for Docs chrome extension.
2023-01-27 15:11:20 -04:00
Lucas Manuel Rodriguez
f64ba83b2e
Make test more stable (#9162) 2023-01-03 16:39:28 -03:00
Lucas Manuel Rodriguez
ac22aadc13
Fleet server and tooling to use NETWORK_TEST_GITHUB_TOKEN when environment variable is set. (#9143)
* WIP

* Add more logging

* Check rate limit at end of action

* Add github client in more places

* Add new published firefox 93 vulnerabilities to tests

* Remove fmt printfs

* Restore CI check settings

* Readd newline
2023-01-03 14:56:11 -03:00