> Issue found during testing
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
> Related issue: #24534
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
for #24131
This PR fixes an issue that can occur when reconciling entries in the
`software` and `software_titles` tables. This can cause
vulnerability-scanning cron jobs to exit early. The error is:
```
upsert software titles: Error 1869 (HY000): Auto-increment value in UPDATE conflicts with internally generated values
```
I haven't had any luck reproducing this locally; the relevant query
seems to cover all the bases as far as ensuring there are no duplicates.
I suspect it may be due to data in these instances that existed prior to
the introduction of the `additional_identifier` column in
`software_titles`. In any case, replacing the `id = id` no-op with a
clause that updates non-auto-incrementing fields is the standard
practice here (in fact it's done in other ON DUPLICATE KEY UPDATE
queries in the same file).
There are automated tests for `ReconcileSoftwareTitles`. I'd gladly add
one to test this case if I could figure out how to reliably cause it,
but if that involves altering the database to put in bad data, it's so
far from the happy path that it's probably not worth it.
Changes to add `team_identifier` signing information to macOS
applications on the `/api/latest/fleet/hosts/:id/software` API endpoint.
Docs: https://github.com/fleetdm/fleet/pull/23743
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
- [X] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ X Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Orbit runs on macOS, Linux and Windows. Check if the orbit
feature/bugfix should only apply to one platform (`runtime.GOOS`).
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [X] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
---------
Co-authored-by: Tim Lee <timlee@fleetdm.com>
Co-authored-by: Ian Littman <iansltx@gmail.com>
> Related issue: #23247
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
The intention here should be to initialize a slice with a capacity of
totalTitlesToProcess rather than initializing the length of this slice.
The only demo: https://go.dev/play/p/q1BcVCmvidW
#20757
API endpoint `/api/v1/fleet/mdm/profiles/batch` will now not log an
activity for profile types that did not change in the database (Apple
configuration profiles, Windows configuration profiles, or Apple
declarations).
Demo video: https://www.loom.com/share/8b75cbd8e7394c12ac6b56746b72c244
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- [x] Manual QA for all new/changed functionality
#21427
Also fixed#21488
- Demo video:
https://www.loom.com/share/0a97afa680c74fa884ac07e87857098a
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Manual QA for all new/changed functionality
Unreleased bug for iOS/iPadOS VPP #19447
# Checklist for submitter
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
Missing part of #19447
Added `available_for_install` query parameter to `GET
/api/v1/fleet/hosta/:id/software`
# Checklist for submitter
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#19447
iOS and iPadOS apps can be installed using Apple's VPP (Volume Purchase
Program)
VPP apps are now using a composite primary key (Adam ID and platform)
because we want to keep iOS/iPadOS/macOS separate. It is possible for
one app to be installable on all Apple platforms.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
Part 2 of https://github.com/fleetdm/fleet/issues/19447
- iOS and iPadOS user-installed apps are loaded into Fleet
- Added an additional identifier into software_titles table to
differentiate between iOS/iPadOS apps
- Updated nano queue timestamp precision
Note: TestIntegrationsMDM/TestVPPApps fails when run as part of the
suite, but passes standalone. I'd like to proceed with merging this PR,
and figure out the issue next week.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Roberto Dip <rroperzh@gmail.com>
> Related issue: #20662
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
during load testing we found this query to be a bottleneck, locally
splitting in to two different statements makes a difference since we can
effectively use the indexes.
```
mysql> explain UPDATE software s
-> JOIN software_titles st
-> ON COALESCE(s.bundle_identifier, '') = '' AND s.name = st.name AND s.source = st.source AND s.browser = st.browser
-> SET s.title_id = st.id
-> WHERE (s.title_id IS NULL OR s.title_id != st.id)
-> AND COALESCE(s.bundle_identifier, '') = '';
+----+-------------+-------+------------+-------+-------------------------------------------------------------------------------------+----------------------------+---------+------------------------------------------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+-------------------------------------------------------------------------------------+----------------------------+---------+------------------------------------------------+------+----------+-------------+
| 1 | SIMPLE | st | NULL | index | idx_sw_titles | idx_sw_titles | 2302 | NULL | 765 | 100.00 | Using index |
| 1 | UPDATE | s | NULL | ref | software_listing_idx,software_source_vendor_idx,title_id,idx_sw_name_source_browser | idx_sw_name_source_browser | 2302 | fleet.st.name,fleet.st.source,fleet.st.browser | 1 | 91.00 | Using where |
+----+-------------+-------+------------+-------+-------------------------------------------------------------------------------------+----------------------------+---------+------------------------------------------------+------+----------+-------------+
2 rows in set (0.00 sec)
mysql> explain UPDATE software s
-> JOIN software_titles st
-> ON s.bundle_identifier = st.bundle_identifier
-> SET s.title_id = st.id
-> WHERE s.title_id IS NULL
-> OR s.title_id != st.id;
+----+-------------+-------+------------+------+-----------------------------------------------------+---------------------------------------+---------+---------------------------+------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+-----------------------------------------------------+---------------------------------------+---------+---------------------------+------+----------+--------------------------+
| 1 | UPDATE | s | NULL | ALL | title_id,idx_software_bundle_id | NULL | NULL | NULL | 788 | 100.00 | Using where |
| 1 | SIMPLE | st | NULL | ref | idx_software_titles_bundle_identifier,idx_composite | idx_software_titles_bundle_identifier | 1023 | fleet.s.bundle_identifier | 1 | 100.00 | Using where; Using index |
+----+-------------+-------+------------+------+-----------------------------------------------------+---------------------------------------+---------+---------------------------+------+----------+--------------------------+
2 rows in set (0.00 sec)
```
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Manual QA for all new/changed functionality
for #19144
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
> Related issue: #20050
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
`go-kit/kit/log` was deprecated and generating warnings
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Manual QA for all new/changed functionality
#19348
Fixed host details page and device details page not showing the latest
software.
- During software ingestion, software titles are now added if needed and
software items have their title_id field populated.
- In addition, after refreshing via UI, the software will be re-fetched
if it has been modified.
Added `exclude_software` query parameter to the
`/api/latest/fleet/hosts/:id` endpoint to exclude software from the
response.
PR for API doc change: #19617
Related issue filed for the Device User Page:
https://github.com/fleetdm/fleet/issues/19618
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#11942
To check if you have duplicate software, install multiple python
versions, like:
```shell
brew install python@3.11
brew install python@3.12
```
Then query: `SELECT * FROM python_packages` to see the packages with the
same versions but different paths.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
