#15472
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Issue #13034
TODO: Frontend requirements will be covered in a separate PR.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
This fixes the deadlock reported in #14779.
We found a deadlock in software ingestion during load tests performed in
October:
```
2023-10-26T17:20:41.719627Z 0 [Note] [MY-012468] [InnoDB] Transactions deadlock detected, dumping detailed information. (lock0lock.cc:6482)
2023-10-26T17:20:41.719661Z 0 [Note] [MY-012469] [InnoDB] *** (1) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069866646, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 8 lock struct(s), heap size 1136, 18 row lock(s), undo log entries 10
MySQL thread id 95, OS thread handle 70431326097136, query id 340045 10.12.3.105 fleet executing
DELETE FROM software WHERE id IN (165, 79, 344, 47, 212, 21, 60, 127, 173, 145) AND
NOT EXISTS (
SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
)
2023-10-26T17:20:41.719700Z 0 [Note] [MY-012469] [InnoDB] *** (1) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866646 lock_mode X locks rec but not gap
Record lock, heap no 22 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 0000000000000015; asc ;;
1: len 6; hex 0000a74c4a7c; asc LJ|;;
2: len 7; hex 82000000d00264; asc d;;
3: len 26; hex 616e74692d76697275735f666f725f736f70686f735f686f6d65; asc anti-virus_for_sophos_home;;
4: len 5; hex 322e322e36; asc 2.2.6;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 48 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000002f; asc /;;
1: len 6; hex 0000a74c4aad; asc LJ ;;
2: len 7; hex 81000000e30220; asc ;;
3: len 10; hex 7265616c706c61796572; asc realplayer;;
4: len 11; hex 31322e302e312e31373338; asc 12.0.1.1738;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 61 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000003c; asc <;;
1: len 6; hex 0000a74c4afb; asc LJ ;;
2: len 7; hex 820000017501ba; asc u ;;
3: len 7; hex 636f6e6e656374; asc connect;;
4: len 5; hex 332e322e37; asc 3.2.7;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000004f; asc O;;
1: len 6; hex 0000a74c4b32; asc LK2;;
2: len 7; hex 820000008a01cb; asc ;;
3: len 7; hex 68697063686174; asc hipchat;;
4: len 4; hex 342e3330; asc 4.30;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
2023-10-26T17:20:41.720564Z 0 [Note] [MY-012469] [InnoDB] *** (1) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866646 lock mode S waiting
Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 8; hex 000000000000004f; asc O;;
1: len 4; hex 0000000c; asc ;;
2023-10-26T17:20:41.720650Z 0 [Note] [MY-012469] [InnoDB] *** (2) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069866680, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 7 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 8
MySQL thread id 98, OS thread handle 70375801900784, query id 340524 10.12.3.9 fleet executing
DELETE FROM software WHERE id IN (49, 113, 183, 187, 223, 79, 81, 116) AND
NOT EXISTS (
SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
)
2023-10-26T17:20:41.720682Z 0 [Note] [MY-012469] [InnoDB] *** (2) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866680 lock_mode X locks rec but not gap
Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 8; hex 000000000000004f; asc O;;
1: len 4; hex 0000000c; asc ;;
2023-10-26T17:20:41.720760Z 0 [Note] [MY-012469] [InnoDB] *** (2) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866680 lock_mode X locks rec but not gap waiting
Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000004f; asc O;;
1: len 6; hex 0000a74c4b32; asc LK2;;
2: len 7; hex 820000008a01cb; asc ;;
3: len 7; hex 68697063686174; asc hipchat;;
4: len 4; hex 342e3330; asc 4.30;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
2023-10-26T17:20:41.720984Z 0 [Note] [MY-012469] [InnoDB] *** WE ROLL BACK TRANSACTION (2) (lock0lock.cc:6496)
```
I was able to reproduce this issue on `main` with the added test. The
solution is to remove the deletion (cleanup) of `software` to a separate
transaction after the main transaction is done.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
> #14920
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#14779
This PR fixes the deadlock when upserting to `host_batteries`.
Which probably happens because InnoDB uses row-locking.
I was able to reproduce in main with the new test
`TestHosts/ReplaceHostBatteriesDeadlock`.
I refactored `ds.ReplaceHostBatteries` to use the same upsert pattern as
`ds.ReplaceHostDeviceMapping` (given `battery` is assumed to return just
a few rows per host). With such pattern the tests does not fail with
deadlock errors anymore.
Here are some of the techniques MySQL recommends:
https://dev.mysql.com/doc/refman/5.7/en/innodb-deadlocks-handling.html
Basically by changing the upsert pattern the deadlock goes away (It's
hard to know exactly why the original code deadlocks).
Here's the deadlock trace from load test performed in October:
```
2023-10-26T17:19:17.244707Z 0 [Note] [MY-012468] [InnoDB] Transactions deadlock detected, dumping detailed information. (lock0lock.cc:6482)
2023-10-26T17:19:17.244756Z 0 [Note] [MY-012469] [InnoDB] *** (1) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069771944, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 75, OS thread handle 70369297350384, query id 658 10.12.3.201 fleet update
INSERT INTO
host_batteries (
host_id,
serial_number,
cycle_count,
health
)
VALUES
(27472, '0000', 505, 'Good'),(27472, '0001', 730, 'Good')
ON DUPLICATE KEY UPDATE
cycle_count = VALUES(cycle_count),
health = VALUES(health),
updated_at = CURRENT_TIMESTAMP
2023-10-26T17:19:17.244800Z 0 [Note] [MY-012469] [InnoDB] *** (1) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771944 lock_mode X locks gap before rec
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245027Z 0 [Note] [MY-012469] [InnoDB] *** (1) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771944 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245239Z 0 [Note] [MY-012469] [InnoDB] *** (2) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069771958, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 9, OS thread handle 70369296809712, query id 708 10.12.2.156 fleet update
INSERT INTO
host_batteries (
host_id,
serial_number,
cycle_count,
health
)
VALUES
(59161, '0000', 1384, 'Fair'),(59161, '0001', 396, 'Good')
ON DUPLICATE KEY UPDATE
cycle_count = VALUES(cycle_count),
health = VALUES(health),
updated_at = CURRENT_TIMESTAMP
2023-10-26T17:19:17.245272Z 0 [Note] [MY-012469] [InnoDB] *** (2) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771958 lock_mode X locks gap before rec
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245504Z 0 [Note] [MY-012469] [InnoDB] *** (2) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771958 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245730Z 0 [Note] [MY-012469] [InnoDB] *** WE ROLL BACK TRANSACTION (2) (lock0lock.cc:6496)
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
#15168
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests.
- [X] Manual QA for all new/changed functionality.
The issue can be reproduced by running `osqueryd` with
`--config_plugin=filesystem --config_path=/path/to/config.json`
This means the osquery config is fetched from a file rather than from
Fleet's agent settings.
The `/path/to/config.json` has the agent settings, e.g.:
```
{
"decorators": {
"load": [
"SELECT uuid AS host_uuid FROM system_info;",
"SELECT hostname AS hostname FROM system_info;"
]
},
"options": {
"disable_distributed": false,
"distributed_interval": 10,
"distributed_plugin": "tls",
"distributed_tls_max_attempts": 3,
"logger_tls_endpoint": "/api/osquery/log",
"logger_tls_period": 10,
"pack_delimiter": "/"
},
"schedule": {
"USB devices": {
"query": "SELECT * FROM usb_devices;",
"interval": 15
},
"OS version": {
"query": "SELECT * FROM os_version;",
"interval": 10
}
},
"packs": {
"Elsewhere": {
"queries": {
"Osquery Info": {
"query": "SELECT * FROM osquery_info;",
"interval": 30,
"platform": "",
"version": "",
"snapshot": true
}
}
}
}
}
```
The three queries should be logged to Fleet's configured result logging
destination (default is `filesystem`).
for #15362, this adjusts the query we use to get MDM details for windows
to account for hosts that might have more than one matching value in the
registry for any of the items we query.
## Addresses #14970
Command fails for correct reason (no real path provided), instead of
reading it as multiple commands due to line breaks:
<img width="801" alt="Screenshot 2023-11-27 at 4 27 21 PM"
src="https://github.com/fleetdm/fleet/assets/61553566/c30b183a-d8e4-4b90-85be-9097dcd7315c">
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
https://github.com/fleetdm/confidential/issues/2118
API doc changes are at: https://github.com/fleetdm/fleet/pull/15092
This change affects Chrome-based and Firefox browsers. I tried Safari,
but it appears that latest Safari extensions simply show up as regular
apps on macOS.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Tested detail query changes on macOS, Windows, Ubuntu, and ChromeOS
- Tested Chrome, Edge, Brave, and Firefox browsers
Testing notes:
1. Install some browser extensions.
2. Update fleet to this change.
3. Refresh hosts, and trigger vulnerabilities job (which updates host
software).
4. Do something like `GET
https://localhost:8080/api/v1/fleet/software?query=Password%20Manager`
to see `extension_id` and `browser` for the hosts that have updated.
> Addresses #14851
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
for #15068
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Relates to #14958
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
#15037
For endpoint fleet/hosts, filtering hosts with invalid team_id now
returns 400 error.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#15176
This was already fixed earlier in the sprint by
https://github.com/fleetdm/fleet/pull/15187
Adding docs and a test.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
#14888
@getvictor This is ready for review, but keeping as draft as there are
probably many tests that need amending.
I used the new version of the `./tools/nvd/nvdvuln/nvdvuln.go` to
compare the current vulnerabilities found in our dogfood environment
with the vulnerabilities found by the code in this PR and both results
match:
```
go run -race -tags fts5 ./tools/nvd/nvdvuln/nvdvuln.go --debug --db_dir ./local --software_from_url <dogfood URL> --software_from_api_token <API_TOKEN> --sync 2>&1 | tee out.txt
[...]
CVEs found and expected matched!
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: Victor Lyuboslavsky <victor@fleetdm.com>
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
related to #14359
Implements the UI for windows custom profiles. This includes:
- uploading, downloading, viewing, and deleting windows profiles
- updating the profile status aggregate to new endpoint that includes
windows profiles.
- UI updates and refactoring of some code to be more maintainable
## Addresses #14102
- Enable closing this menu on clicking its header when open
- Other small UX and code improvements around this component
https://github.com/fleetdm/fleet/assets/61553566/b848b2d1-533f-4aa0-9827-e841d3d840e8
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#15143 and #15162
Previous fix for #13889 caused false positives on software with similar
names. Tightening the matching to reduce false positive rate.
- Google Chrome Helper.app no longer matches Google Chrome.app
- Acrobat Uninstaller.app no longer matches Acrobat.app
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
Part of #15135
This needs to be removed ASAP because software inventory queries will
start failing when the table is removed from osquery and the agent
updates to that new version.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux. (macOS only)
### Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
### Summary
This PR creates a pluralized activity log in the case where a user
deletes multiple saved queries.
I had considered updating the default template to better support
pluralization, but when reviewing the various activity log types, I
think queries are the only type that would benefit from this. So I chose
the easier option here which felt less risky.
cc @sharon-fdm as the currently assigned person on the issue (just
trying to save y'all some cycles by contributing and slowly ramping up
on the codebase 😄)
ref: #15050
### Test Plan
I added a unit test for this.
Additionally, I deleted multiple saved queries in my local installation.
Here's the resulting log, showing the proper pluralization:
<img width="1381" alt="Screenshot 2023-11-10 at 9 54 40 PM"
src="https://github.com/fleetdm/fleet/assets/1317288/f40414e2-7a9b-4478-b6cf-bb9d4ab6d8f0">
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
When Munki is deleted and reinstalled on the host, Fleet will show Munki
info again.
#12409
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Query report now shows the host display name instead of hostname.
#14763
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
### Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
### Summary
This PR adds the [latest osquery
versions](https://github.com/osquery/osquery/releases) to the list of
versions suitable for use in a query's "minimum osquery version"
setting.
As far as I could tell, this was solely a frontend bug and did not
require any backend changes.
ref: https://github.com/fleetdm/fleet/issues/14991
### Test Plan
I updated a standard query to include:
```yaml
min_osquery_version: 5.10.2
```
And then applied the query with `fleetctl apply`.
I then saw the version reflected in the UI:
<img width="1381" alt="Screenshot 2023-11-12 at 10 01 32 PM"
src="https://github.com/fleetdm/fleet/assets/1317288/4c9922aa-5afc-4c54-8929-1f64e3931e34">
I also manually set the minimum osquery version in the UI and confirmed
it was reflected in the API response and when refreshing the page.
Longer term, I'd encourage considering whether setting up some kind of
GH Action or, easier but more manual, an engineering ritual might make
sense here to update this list as osquery releases happen. 🤷
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/REST API/rest-api.md or
docs/Contributing/API-for-contributors.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
final step of #14361 , this integrates the work of the other PRs.
currently branched off `14361-windows-custom-settings-configs`
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
for #14361 this adds the bits related to saving a slice of strings with
paths to configuration profiles.
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Marko Lisica <83164494+marko-lisica@users.noreply.github.com>
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Loom explaining changes:
https://www.loom.com/share/f05f241a77304c19bc6ba1d0702c7bd8?sid=ea86b282-0bda-4ba4-a6cf-4520f0db610d#14571
Enabled support and validation of 'after' parameter for the following
endpoints:
- GET /api/v1/fleet/carves
Setting 'after' parameter no longer returns SQL syntax error for the
following endpoints:
- GET /api/v1/fleet/carves
- GET /api/v1/fleet/invites
- GET /api/v1/fleet/labels
- GET /api/v1/fleet/packs
- GET /api/v1/fleet/global/policies
- GET /api/v1/fleet/teams/{id}/policies
- GET /api/v1/fleet/queries
- GET /api/v1/fleet/packs/{id}/scheduled
- GET /api/v1/fleet/teams
- GET /api/v1/fleet/users
API doc changes PR: https://github.com/fleetdm/fleet/pull/15061
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
Fixes to /fleet/queries/run endpoint:
- now returns 403 for an unauthorized user
- now returns 400 when query_ids or host_ids are not specified
#11446 and #11901
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
API clarifications are in a separate PR
https://github.com/fleetdm/fleet/pull/14956
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
After 30 seconds, the 'Delete host' modal closes and the delete
operation continues in the background. The following text has been added
to the modal when deleting 500 or more hosts: "When deleting a large
volume of hosts, it may take some time for this change to be reflected
in the UI."
#14097
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
- [x] Backend test added
/fleet/sso endpoint now returns 400 status code (as opposed to 500) when
SSO Metadata URL returns invalid data or SSO Metadata is invalid
#12559
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Fixed 500 return code from several endpoints.
/api/v1/fleet/perform_required_password_reset
- Now returns 403 when Authorization token is missing
/api/v1/fleet/hosts_summary
- Now returns 400 when low_disk_space parameter is invalid
/api/v1/fleet/demologin
- Now returns 403
/api/v1/fleet/sessions/*
- Now returns 400 on invalid input
#12274
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#14554
For the following endpoints:
/api/v1/fleet/software
/api/v1/fleet/software/count
- added validation on `page`, `per_page`, `order_key`, `order_direction`
-- invalid values will now return 400 HTTP status code
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Fix many vulnerability false positives on Chrome and Firefox
extensions. This is related to #11924 and also related to the following
5 false positives found in dogfood around Chrome extensions.
- Reduced `TestTranslateCPEToCVE` run time considerably:
```
--- PASS: TestTranslateCPEToCVE (8.59s)
--- PASS: TestTranslateCPEToCVE/find_vulns_on_cpes (38.45s)
--- PASS: TestTranslateCPEToCVE/recent_vulns (62.03s)
vs.
=== RUN TestTranslateCPEToCVE/recent_vulns
=== NAME TestTranslateCPEToCVE
nettest.go:36: network test done: TestTranslateCPEToCVE
--- PASS: TestTranslateCPEToCVE (269.86s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️1password:1password:3.9.9:*:*:*:*:macos:*:* (31.31s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️1password:1password:3.9.9:*:*:*:*:*:*:* (29.00s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️pypa:pip:9.0.3:*:*:*:*:python:*:* (52.59s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️mozilla:firefox:93.0:*:*:*:*:windows:*:* (34.29s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️mozilla:firefox:93.0.100:*:*:*:*:windows:*:* (28.17s)
--- PASS: TestTranslateCPEToCVE/cpe:2.3🅰️apple:icloud:1.0:*:*:*:*:macos:*:* (28.44s)
--- PASS: TestTranslateCPEToCVE/recent_vulns (56.95s)
```
<img width="868" alt="Screenshot 2023-10-30 at 10 21 42"
src="https://github.com/fleetdm/fleet/assets/2073526/284a2373-09bc-44f7-952b-1e53650232ff">
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Adding vulnerability data support for JetBrains applications (like IDEA,
PyCharm, etc.) that have similar names.
- For example: IntelliJ IDEA.app and IntelliJ IDEA Ultimate.app
Resolves#13889
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
relates to #14418
This fixes an issue where the team dropdown would change when a host
that was on a team is selected.
We fixed this by passing the host team into the host actions dropdown
component instead of changing the global current team context attribute.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#14522
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#14543
Python's certifi package is ingested with the following version string:
`2023.7.22`.
The NVD dataset uses the following versioning: `2023.07.22`.
This PR updates the nvdtools package.
This is the fix in nvdtools that fixes this exact false positive:
c0d18738cd
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#13998
Cleaned up REST API input validation for hosts, carves, users endpoints.
rest-api.md changes explained:
https://www.loom.com/share/9cd82653bacb4528bdaac117ec85a976
For the following endpoints:
/api/v1/fleet/hosts
/api/v1/fleet/hosts/count
/api/v1/fleet/hosts/report
- converted validation errors from 500 to 400 HTTP status code
- added validation that policy_id must be present when policy_response
is specified
- added validation that policy_response must be `passing` or `failing`
- added validation that `os_name` must be specified with `os_version`
For the following endpoint:
/api/v1/fleet/users
- converted team_id validation error from 500 to 400 HTTP status code
For the following endpoint:
/api/v1/fleet/carves
- added rest-api.md documentation for `page`, `per_page`, `order_key`,
`order_direction`, and `expired`
- converted `expired` validation error from 500 to 400 HTTP status code
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
relates to #13012
Updates the copy of the ABM terms banner message to reflect that Apple
may take a long time to update that the terms have been accepted.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#12888
Updated code so that macOS version returns RSR, if available. For
example: `14.3.1 (a)` as opposed to simply `14.3.1`
This feature is supported by osquery with the `extra` column in
`os_version` table.
Since I could not get a real macOS running a version with RSR, I did the
following manual checks:
- ensured that `extra` column from osquery `os_version` table was being
received for processing
- hard coded the `extra` column result on the server and verified the OS
version was correct in GUI and REST APIs
Main testing was done by Zach following these
[instructions](https://docs.google.com/document/d/18xsCEFlpcBL-5EbQhvmUNuJz3XM0Ak-NfCCwxEQXVic/edit)
- I confirmed that the one incorrect query result was due to a bad query
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#13615
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
For #14056, per the [mdn web
docs](https://developer.mozilla.org/en-US/docs/Web/API/File/type)
> Note: Based on the current implementation, browsers won't actually
> read the bytestream of a file to determine its media type. It is
assumed
> based on the file extension; a PNG image file renamed to .txt would
give
> "text/plain" and not "image/png". Moreover, file.type is generally
> reliable only for common file types like images, HTML documents, audio
> and video. Uncommon file extensions would return an empty string.
Client
> configuration (for instance, the Windows Registry) may result in
> unexpected values even for common types. Developers are advised not to
> rely on this property as a sole validation scheme.
for #13703 and #13992, this updates the logic used by the functions that
gather hosts that need DEP profile updates to use hosts directly from
`host_dep_assignments`, regardless of their MDM status.
I'm not sure this is the best place to put this information or if this
is the best format. But we need to inform users of these changes
somewhere in the v4.39.0 release notes. Am all ears.
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
#11923
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [x] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Removes a changefile that was included in a previous release and
accidentally re-introduced to main.
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#14383
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
(tested the fix with my reproduction setup)
# Checklist for submitter
#13486
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Eric <eashaw@sailsjs.com>
Co-authored-by: Sampfluger88 <108141731+Sampfluger88@users.noreply.github.com>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Tyler Diderich <15862572+tdiderich@users.noreply.github.com>
Co-authored-by: Dave Herder <27025660+dherder@users.noreply.github.com>
Co-authored-by: Rachael Shaw <r@rachael.wtf>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Sharon Katz <121527325+sharon-fdm@users.noreply.github.com>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
Co-authored-by: Nathanael Holliday <100959072+hollidayn@users.noreply.github.com>
Co-authored-by: Katheryn Satterlee <me@ksatter.com>
Co-authored-by: Mo Zhu <mo@fleetdm.com>
Co-authored-by: Mo Zhu <mozhu888@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Robert Fairburn <8029478+rfairburn@users.noreply.github.com>
Co-authored-by: Sabrina Coy <13890648+sabrinabuckets@users.noreply.github.com>
Co-authored-by: Isabell Reedy <113355639+ireedy@users.noreply.github.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Alex Mitchell <105945793+alexmitchelliii@users.noreply.github.com>
- fix: use function that returns params
- fix: missing integration test
fixes: #13809
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#13287
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#13926
I re-added the old queries back and we are now using them when not using
pagination or if the page size is larger than 100 (UI always uses
per_page=50)
TODO: We need to run actual load tests with high number of hosts (50k
hosts with 140 policies each)
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Found these bugs while testing the extensions feature for #13287.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [x] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
relates to #12926
This implements the changes to the Controls page that add the windows
Bitlocker functionality.
There is some work that needs to be complete when the API is done. For
now we are mocking the new disk encryption API response
**new column for windows hosts:**
also includes various other changes behind the scenes that include
windows hosts into the disk encryption as well as changes to the
profiles status summary to use StatusIndicatorWithIcon
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
This relates to #13875
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Manual QA for all new/changed functionality
For #13715, this:
- Upgrades the Go version to `1.21.1`, infrastructure changes are
addressed separately at https://github.com/fleetdm/fleet/pull/13878
- Upgrades the linter version, as the current version doesn't work well
after the Go upgrade
- Fixes new linting errors (we now get errors for memory aliasing in
loops! 🎉 )
After this is merged people will need to:
1. Update their Go version. I use `gvm` and I did it like:
```
$ gvm install go1.21.1
$ gvm use go1.21.1 --default
```
2. Update the local version of `golangci-lint`:
```
$ go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.54.2
```
3. (optional) depending on your setup, you might need to re-install some
packages, for example:
```
# goimports to automatically import libraries
$ go install golang.org/x/tools/cmd/goimports@latest
# gopls for the language server
$ go install golang.org/x/tools/gopls@latest
# etc...
```
#11922
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Should fix the issue reported in #12230. For Wireshark, osquery was
reporting a `vendor` value larger than what we allowed storing in the
`vendor` column (32 bytes). But recently we enlarged the `vendor` column
to fit `114` chars. The direct software ingestion routine was inserting
a new software entry every time because the incoming software vendor was
different to what Fleet had stored in the previous ingestion (`vendor`
column trimmed from `The Wireshark developer community,
https://www.wireshark.org/` to `The Wireshark developer communit`).
I've now made sure that all fields are trimmed as soon as they are
received by osquery thus not triggering any re-inserts when any field is
larger than what Fleet supports.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#13318
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [x] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#13527
(Adding @mna to double check the changes in the async implementation of
policy result storage)
This PR also adds the osquery-perf changes needed to define the count of
macOS and Windows hosts.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Test with 80k hosts: 70k simulated macOS, 10k simulated Windows.
Apply Windows policies first, then apply macOS policies:
```
fleetctl apply -f ee/cis/win-10/cis-policy-queries.yml
# Leave running for some time
fleetctl apply -f ee/cis/macos-13/cis-policy-queries.yml
```
After applying CIS policies previous to these changes:
After applying these changes and applying the same policies:
related to #12847
This changes the authentication method for windows mdm enrollment. We
were using `HostByIndentifier ` method but have changed to
`LoadHostByOrbitNodeKey`.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
relates to #13308
Implements the UI for the activity item for script ran, and the script
details modal.
NOTE: Still have to do API integration and will do when API is ready in
another PR.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
Found while working on #12696.
This was caught be a recent check added by @mostlikelee to `fleetctl
apply` (#13294).
Sample error:
```sh
$ fleetctl apply --context loadtest -f ee/cis/win-10/cis-policy-queries.yml
Error: applying policies: policy names must be globally unique. Please correct policy "CIS - Ensure 'Windows Firewall: Public: Firewall state' is set to 'On (recommended)'\n" and try again.
```
#11939
- This PR fixes typos in three CIS Windows queries (the queries were
failing with `invalid SQL syntax`).
- Also adds tooling to perform similar testing that we ran for macOS
(using `fleetd_tables` as an extension).
changelog for the version bump can be found here:
https://github.com/macadmins/osquery-extension/releases/tag/v0.0.15
related to #13158 as this fixes a bug in the extension causing `profiles
show --type enrollment` to be run almost every time we queried something
from the `mdm` table.
I couldn't find any new tables. Other than that, some dependencies were
updated on their repo as well, most notably `osquery/osquery-go`
relates to #12986
This is a fix for a case where the `os_versions` endpoint was returning
a 404 when searching for `no teams`.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
related to #12483, we have found out that in distributed scenarios, the
URL of the Puppet server used for the request is appended to the
identifier, and it can be different between `/preassign` and `/match`
calls.
to account for this, we're only grabbing the first 36 characters of the
identifier.
Prior to 4.35.0, some rows in the scheduled_query table might have a
`NULL` value due to a race condition with database replicas and the way
`ds.EnsureGlobalPack` and `ApplyPackSpecs` work together.
This is no longer the case, but some databases are left in weird states,
which were not accounted by this migration.
Chaning the migration in-place because that's the approach we took in
previous migrations with similar problems.
relates to #12907
implement adding an enroll secret when creating a new team with
`fleetctl apply` if none is provided.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
This relates to #12600
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [X] Manual QA for all new/changed functionality
## Addresses #12999
<img width="1282" alt="Screenshot 2023-07-27 at 11 59 01 AM"
src="https://github.com/fleetdm/fleet/assets/61553566/b60d3b41-3d7b-4550-ba7c-8615bae085a6">
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
