#14779
This PR fixes the deadlock when upserting to `host_batteries`.
Which probably happens because InnoDB uses row-locking.
I was able to reproduce in main with the new test
`TestHosts/ReplaceHostBatteriesDeadlock`.
I refactored `ds.ReplaceHostBatteries` to use the same upsert pattern as
`ds.ReplaceHostDeviceMapping` (given `battery` is assumed to return just
a few rows per host). With such pattern the tests does not fail with
deadlock errors anymore.
Here are some of the techniques MySQL recommends:
https://dev.mysql.com/doc/refman/5.7/en/innodb-deadlocks-handling.html
Basically by changing the upsert pattern the deadlock goes away (It's
hard to know exactly why the original code deadlocks).
Here's the deadlock trace from load test performed in October:
```
2023-10-26T17:19:17.244707Z 0 [Note] [MY-012468] [InnoDB] Transactions deadlock detected, dumping detailed information. (lock0lock.cc:6482)
2023-10-26T17:19:17.244756Z 0 [Note] [MY-012469] [InnoDB] *** (1) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069771944, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 75, OS thread handle 70369297350384, query id 658 10.12.3.201 fleet update
INSERT INTO
host_batteries (
host_id,
serial_number,
cycle_count,
health
)
VALUES
(27472, '0000', 505, 'Good'),(27472, '0001', 730, 'Good')
ON DUPLICATE KEY UPDATE
cycle_count = VALUES(cycle_count),
health = VALUES(health),
updated_at = CURRENT_TIMESTAMP
2023-10-26T17:19:17.244800Z 0 [Note] [MY-012469] [InnoDB] *** (1) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771944 lock_mode X locks gap before rec
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245027Z 0 [Note] [MY-012469] [InnoDB] *** (1) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771944 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245239Z 0 [Note] [MY-012469] [InnoDB] *** (2) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069771958, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 9, OS thread handle 70369296809712, query id 708 10.12.2.156 fleet update
INSERT INTO
host_batteries (
host_id,
serial_number,
cycle_count,
health
)
VALUES
(59161, '0000', 1384, 'Fair'),(59161, '0001', 396, 'Good')
ON DUPLICATE KEY UPDATE
cycle_count = VALUES(cycle_count),
health = VALUES(health),
updated_at = CURRENT_TIMESTAMP
2023-10-26T17:19:17.245272Z 0 [Note] [MY-012469] [InnoDB] *** (2) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771958 lock_mode X locks gap before rec
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245504Z 0 [Note] [MY-012469] [InnoDB] *** (2) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771958 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245730Z 0 [Note] [MY-012469] [InnoDB] *** WE ROLL BACK TRANSACTION (2) (lock0lock.cc:6496)
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
#15168
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests.
- [X] Manual QA for all new/changed functionality.
The issue can be reproduced by running `osqueryd` with
`--config_plugin=filesystem --config_path=/path/to/config.json`
This means the osquery config is fetched from a file rather than from
Fleet's agent settings.
The `/path/to/config.json` has the agent settings, e.g.:
```
{
"decorators": {
"load": [
"SELECT uuid AS host_uuid FROM system_info;",
"SELECT hostname AS hostname FROM system_info;"
]
},
"options": {
"disable_distributed": false,
"distributed_interval": 10,
"distributed_plugin": "tls",
"distributed_tls_max_attempts": 3,
"logger_tls_endpoint": "/api/osquery/log",
"logger_tls_period": 10,
"pack_delimiter": "/"
},
"schedule": {
"USB devices": {
"query": "SELECT * FROM usb_devices;",
"interval": 15
},
"OS version": {
"query": "SELECT * FROM os_version;",
"interval": 10
}
},
"packs": {
"Elsewhere": {
"queries": {
"Osquery Info": {
"query": "SELECT * FROM osquery_info;",
"interval": 30,
"platform": "",
"version": "",
"snapshot": true
}
}
}
}
}
```
The three queries should be logged to Fleet's configured result logging
destination (default is `filesystem`).
for #15362, this adjusts the query we use to get MDM details for windows
to account for hosts that might have more than one matching value in the
registry for any of the items we query.
## Addresses #14970
Command fails for correct reason (no real path provided), instead of
reading it as multiple commands due to line breaks:
<img width="801" alt="Screenshot 2023-11-27 at 4 27 21 PM"
src="https://github.com/fleetdm/fleet/assets/61553566/c30b183a-d8e4-4b90-85be-9097dcd7315c">
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
https://github.com/fleetdm/confidential/issues/2118
API doc changes are at: https://github.com/fleetdm/fleet/pull/15092
This change affects Chrome-based and Firefox browsers. I tried Safari,
but it appears that latest Safari extensions simply show up as regular
apps on macOS.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Tested detail query changes on macOS, Windows, Ubuntu, and ChromeOS
- Tested Chrome, Edge, Brave, and Firefox browsers
Testing notes:
1. Install some browser extensions.
2. Update fleet to this change.
3. Refresh hosts, and trigger vulnerabilities job (which updates host
software).
4. Do something like `GET
https://localhost:8080/api/v1/fleet/software?query=Password%20Manager`
to see `extension_id` and `browser` for the hosts that have updated.
> Addresses #14851
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
for #15068
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Relates to #14958
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
#15037
For endpoint fleet/hosts, filtering hosts with invalid team_id now
returns 400 error.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#15176
This was already fixed earlier in the sprint by
https://github.com/fleetdm/fleet/pull/15187
Adding docs and a test.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
#14888
@getvictor This is ready for review, but keeping as draft as there are
probably many tests that need amending.
I used the new version of the `./tools/nvd/nvdvuln/nvdvuln.go` to
compare the current vulnerabilities found in our dogfood environment
with the vulnerabilities found by the code in this PR and both results
match:
```
go run -race -tags fts5 ./tools/nvd/nvdvuln/nvdvuln.go --debug --db_dir ./local --software_from_url <dogfood URL> --software_from_api_token <API_TOKEN> --sync 2>&1 | tee out.txt
[...]
CVEs found and expected matched!
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: Victor Lyuboslavsky <victor@fleetdm.com>
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
related to #14359
Implements the UI for windows custom profiles. This includes:
- uploading, downloading, viewing, and deleting windows profiles
- updating the profile status aggregate to new endpoint that includes
windows profiles.
- UI updates and refactoring of some code to be more maintainable
## Addresses #14102
- Enable closing this menu on clicking its header when open
- Other small UX and code improvements around this component
https://github.com/fleetdm/fleet/assets/61553566/b848b2d1-533f-4aa0-9827-e841d3d840e8
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#15143 and #15162
Previous fix for #13889 caused false positives on software with similar
names. Tightening the matching to reduce false positive rate.
- Google Chrome Helper.app no longer matches Google Chrome.app
- Acrobat Uninstaller.app no longer matches Acrobat.app
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
Part of #15135
This needs to be removed ASAP because software inventory queries will
start failing when the table is removed from osquery and the agent
updates to that new version.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux. (macOS only)
### Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
### Summary
This PR creates a pluralized activity log in the case where a user
deletes multiple saved queries.
I had considered updating the default template to better support
pluralization, but when reviewing the various activity log types, I
think queries are the only type that would benefit from this. So I chose
the easier option here which felt less risky.
cc @sharon-fdm as the currently assigned person on the issue (just
trying to save y'all some cycles by contributing and slowly ramping up
on the codebase 😄)
ref: #15050
### Test Plan
I added a unit test for this.
Additionally, I deleted multiple saved queries in my local installation.
Here's the resulting log, showing the proper pluralization:
<img width="1381" alt="Screenshot 2023-11-10 at 9 54 40 PM"
src="https://github.com/fleetdm/fleet/assets/1317288/f40414e2-7a9b-4478-b6cf-bb9d4ab6d8f0">
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
When Munki is deleted and reinstalled on the host, Fleet will show Munki
info again.
#12409
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Query report now shows the host display name instead of hostname.
#14763
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
### Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
### Summary
This PR adds the [latest osquery
versions](https://github.com/osquery/osquery/releases) to the list of
versions suitable for use in a query's "minimum osquery version"
setting.
As far as I could tell, this was solely a frontend bug and did not
require any backend changes.
ref: https://github.com/fleetdm/fleet/issues/14991
### Test Plan
I updated a standard query to include:
```yaml
min_osquery_version: 5.10.2
```
And then applied the query with `fleetctl apply`.
I then saw the version reflected in the UI:
<img width="1381" alt="Screenshot 2023-11-12 at 10 01 32 PM"
src="https://github.com/fleetdm/fleet/assets/1317288/4c9922aa-5afc-4c54-8929-1f64e3931e34">
I also manually set the minimum osquery version in the UI and confirmed
it was reflected in the API response and when refreshing the page.
Longer term, I'd encourage considering whether setting up some kind of
GH Action or, easier but more manual, an engineering ritual might make
sense here to update this list as osquery releases happen. 🤷
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/REST API/rest-api.md or
docs/Contributing/API-for-contributors.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
final step of #14361 , this integrates the work of the other PRs.
currently branched off `14361-windows-custom-settings-configs`
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
for #14361 this adds the bits related to saving a slice of strings with
paths to configuration profiles.
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Marko Lisica <83164494+marko-lisica@users.noreply.github.com>
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
