This fixes the deadlock reported in #14779.
We found a deadlock in software ingestion during load tests performed in
October:
```
2023-10-26T17:20:41.719627Z 0 [Note] [MY-012468] [InnoDB] Transactions deadlock detected, dumping detailed information. (lock0lock.cc:6482)
2023-10-26T17:20:41.719661Z 0 [Note] [MY-012469] [InnoDB] *** (1) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069866646, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 8 lock struct(s), heap size 1136, 18 row lock(s), undo log entries 10
MySQL thread id 95, OS thread handle 70431326097136, query id 340045 10.12.3.105 fleet executing
DELETE FROM software WHERE id IN (165, 79, 344, 47, 212, 21, 60, 127, 173, 145) AND
NOT EXISTS (
SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
)
2023-10-26T17:20:41.719700Z 0 [Note] [MY-012469] [InnoDB] *** (1) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866646 lock_mode X locks rec but not gap
Record lock, heap no 22 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 0000000000000015; asc ;;
1: len 6; hex 0000a74c4a7c; asc LJ|;;
2: len 7; hex 82000000d00264; asc d;;
3: len 26; hex 616e74692d76697275735f666f725f736f70686f735f686f6d65; asc anti-virus_for_sophos_home;;
4: len 5; hex 322e322e36; asc 2.2.6;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 48 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000002f; asc /;;
1: len 6; hex 0000a74c4aad; asc LJ ;;
2: len 7; hex 81000000e30220; asc ;;
3: len 10; hex 7265616c706c61796572; asc realplayer;;
4: len 11; hex 31322e302e312e31373338; asc 12.0.1.1738;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 61 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000003c; asc <;;
1: len 6; hex 0000a74c4afb; asc LJ ;;
2: len 7; hex 820000017501ba; asc u ;;
3: len 7; hex 636f6e6e656374; asc connect;;
4: len 5; hex 332e322e37; asc 3.2.7;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000004f; asc O;;
1: len 6; hex 0000a74c4b32; asc LK2;;
2: len 7; hex 820000008a01cb; asc ;;
3: len 7; hex 68697063686174; asc hipchat;;
4: len 4; hex 342e3330; asc 4.30;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
2023-10-26T17:20:41.720564Z 0 [Note] [MY-012469] [InnoDB] *** (1) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866646 lock mode S waiting
Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 8; hex 000000000000004f; asc O;;
1: len 4; hex 0000000c; asc ;;
2023-10-26T17:20:41.720650Z 0 [Note] [MY-012469] [InnoDB] *** (2) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069866680, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 7 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 8
MySQL thread id 98, OS thread handle 70375801900784, query id 340524 10.12.3.9 fleet executing
DELETE FROM software WHERE id IN (49, 113, 183, 187, 223, 79, 81, 116) AND
NOT EXISTS (
SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
)
2023-10-26T17:20:41.720682Z 0 [Note] [MY-012469] [InnoDB] *** (2) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866680 lock_mode X locks rec but not gap
Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 8; hex 000000000000004f; asc O;;
1: len 4; hex 0000000c; asc ;;
2023-10-26T17:20:41.720760Z 0 [Note] [MY-012469] [InnoDB] *** (2) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866680 lock_mode X locks rec but not gap waiting
Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000004f; asc O;;
1: len 6; hex 0000a74c4b32; asc LK2;;
2: len 7; hex 820000008a01cb; asc ;;
3: len 7; hex 68697063686174; asc hipchat;;
4: len 4; hex 342e3330; asc 4.30;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
2023-10-26T17:20:41.720984Z 0 [Note] [MY-012469] [InnoDB] *** WE ROLL BACK TRANSACTION (2) (lock0lock.cc:6496)
```
I was able to reproduce this issue on `main` with the added test. The
solution is to remove the deletion (cleanup) of `software` to a separate
transaction after the main transaction is done.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
https://github.com/fleetdm/confidential/issues/2118
API doc changes are at: https://github.com/fleetdm/fleet/pull/15092
This change affects Chrome-based and Firefox browsers. I tried Safari,
but it appears that latest Safari extensions simply show up as regular
apps on macOS.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Tested detail query changes on macOS, Windows, Ubuntu, and ChromeOS
- Tested Chrome, Edge, Brave, and Firefox browsers
Testing notes:
1. Install some browser extensions.
2. Update fleet to this change.
3. Refresh hosts, and trigger vulnerabilities job (which updates host
software).
4. Do something like `GET
https://localhost:8080/api/v1/fleet/software?query=Password%20Manager`
to see `extension_id` and `browser` for the hosts that have updated.
#14554
For the following endpoints:
/api/v1/fleet/software
/api/v1/fleet/software/count
- added validation on `page`, `per_page`, `order_key`, `order_direction`
-- invalid values will now return 400 HTTP status code
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Should fix the issue reported in #12230. For Wireshark, osquery was
reporting a `vendor` value larger than what we allowed storing in the
`vendor` column (32 bytes). But recently we enlarged the `vendor` column
to fit `114` chars. The direct software ingestion routine was inserting
a new software entry every time because the incoming software vendor was
different to what Fleet had stored in the previous ingestion (`vendor`
column trimmed from `The Wireshark developer community,
https://www.wireshark.org/` to `The Wireshark developer communit`).
I've now made sure that all fields are trimmed as soon as they are
received by osquery thus not triggering any re-inserts when any field is
larger than what Fleet supports.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
https://github.com/fleetdm/confidential/issues/1968
It's ready for review but I still need to load test this.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
This only applies to Premium users, we want to show the vulnerabilities' published date anywhere vulnerabilities are shown including API endpoints and third party integrations.
The OVAL analyzer falsely assumes that any vulnerabilities detected on a
host only come from OVAL. However, it is possible that NVD detects
vulnerabilities on these hosts even though it excludes software from
deb_packages and rpm_packages. For example, a python package twisted
v22.20 has a vulnerability CVE-2022-39348 detected by NVD. The OVAL
analyzer would delete this vulnerability, and it would be re-inserted by
the NVD scanner on the next run. This creates a loop.
The fix is to only delete vulnerabilities that are actually detected
using OVAL. We already store this in the source column in the
software_cve table.
* Bug 7320: Fixed dulp. vulns. on software details.
The software details page was showing duplicated vulnerabilities if the software was being used by many hosts.
* Do not use golangci action for better reproducibility
* Add fix to trigger build
* Fix all reported issues
* fix more lint errors
* Add missing import
* Remove unused method
* Remove change not necessary
Feature: Improve our capability to detect vulnerable software on Ubuntu hosts
To improve the capability of detecting vulnerable software on Ubuntu, we are now using OVAL definitions to detect vulnerable software on Ubuntu hosts. If data sync is enabled (disable_data_sync=false) OVAL definitions are automatically kept up to date (they are 'refreshed' once per day) - there's also the option to manually download the OVAL definitions using the 'fleetctl vulnerability-data-stream' command. Downloaded definitions are then parsed into an intermediary format and then used to identify vulnerable software on Ubuntu hosts. Finally, any 'recent' detected vulnerabilities are sent to any third-party integrations.
* Add CentOS parsing and post-processing in fleet
* Add tests and amend SyncCPEDatabase
* Add test for centosPostProcessing
* Changes from PR comments
* Amend software test
* Fix sync test
* Add index to source and vendor
* Use os.MkdirTemp
* Rearrange migrations
* Regenerate test schema
* Add support for testing migrations (#4112)
* Add support for testing migrations
* Rename migration in tests
* Changes suggested in PR
* Go mod tidy
* WIP
* Amend tests
* Do not load aggregated stats for packs
* Add option to host lite
* Fix remaining TODOs
* Fix osquery_utils tests
* Fix SQL
* Fix SQL (bis)
* Restore AuthenticateHost to load once
* Code improvements and re-add deferred host save
* More fixes to the PR
* Wrap users table update on tx
* Add caching to ListPacksForHost and ListScheduledQueriesInPack
* Remove SaveHostSoftware (replaced by UpdateHostSoftware)
* Add unit tests for new functionality
* Add changes file
* Fix scheduled queries test
