Feature cleanup
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
No related issue, just cleanup work on the feature
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18838 and #18986
Optimized master DB accesses during host software ingestion.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18766
Fixed a bug where updating user via `/api/v1/fleet/users/:id` endpoint
sometimes did not update the activity feed and returned the un-updated
user object.
You must use a DB configuration with a replica to reproduce the issue.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18944
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18477
Built-in labels can now be applied via `fleetctl apply` as long as no
changes are made to them. This allows the following workflow:
1. `fleetctl get labels --yaml > labels.yml`
2. (Optional) Edit/add non-built in labels in labels.yml
3. fleetctl apply -f labels.yml
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Tim Lee <timlee@fleetdm.com>
#18467
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- ~[ ] If database migrations are included, checked table schema to
confirm autoupdate~
- ~For database migrations:~
- ~[ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.~
- ~[ ] Confirmed that updating the timestamps is acceptable, and will
not cause unwanted side effects.~
- ~[ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#18221
Broke apart the hourly host_software count query to reduce the
individual query runtime. This fixes timeouts seen when host_software
table has over 25 million records.
I recommend hiding whitespace during review:
<img width="240" alt="image"
src="https://github.com/fleetdm/fleet/assets/2685025/6da9b643-8582-4d2f-bf32-8a1cc38f1032">
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
> Related issue: #18330
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
## Notes
- ~I added an `includeTitle bool` parameter to
`ds.GetSoftwareInstallerMetadata`. This allows for the title of the
software (from the `software_titles` page) to be fetched in
`svc.DeleteSoftwareInstaller` without an additional call to the DB.~ We
wound up deciding to just fetch the title every time.
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
#18808
Added the new `sofa_security_release_info` and `sofa_unpatched_cves`
tables from `macadmins/osquery-extension` 1.0.1
These tables do not have detailed documentation in macadmins repo, so
not adding documentation at this point.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
#12619
Fixed MySQL sort buffer overflow when fetching activities. This issue
happened when activities contained very large details, such as large SQL
queries.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- Existing tests provide sufficient coverage
- [x] Manual QA for all new/changed functionality
#16562
Optimized policy_stats updates to NOT lock the policy_membership table.
This should improve deployment performance with many global policies and
team hosts.
The original implementation that used INSERT ... SELECT (SELECT
COUNT(*)) ... caused performance issues. Given 50 global policies, 10
teams, and 10,000 hosts per team, the INSERT query took 30-60 seconds to
complete. Since it was an INSERT query, it blocked other hosts from
updating their policy results in policy_membership.
Now, we separate the INSERT from the SELECT, since SELECT by itself does
not block other hosts from updating their policy results. In addition,
we process one global policy at a time, which reduces the time to
complete the SELECT query to <2 seconds, and limits the memory usage. We
are not using a transaction to reduce locks. This means that INSERT may
fail if the policy was deleted by a parallel process. Also, the INSERT
may overwrite a clearing of the stats. This is acceptable, since these
are very rare cases. We log and proceed in that case.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18079
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- ~[ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.~
- ~[ ] Confirmed that updating the timestamps is acceptable, and will
not cause unwanted side effects.~
- ~[ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#18039
Fixed ingestion of private IPv6 address from agent. Host details page
can now display private IPv6 address if private IPv4 does not exist.
How to set up an IPv6-only host:
https://www.loom.com/share/5e205549b6484df88702c27a5aa5a3ee
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
relates to #18326
Add ability to add software from the UI. This includes
- new button on software page to open add software modal
- new add software modal to add software.
> Note: still need to do form error validation but will do on another PR
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
#16677
Improvements to `fleetctl gitops` command:
- Added the ability to pass multiple files, like `fleetctl gitops -f
file1 -f file2`, where the first file must be the global configuration
- Added the ability to remove teams that were not specified in team
configs using the switch `--delete-other-teams`
- When passing a global config and team config during initial
configuration, the `org_settings.mdm.apple_bm_default_team` value can be
set to match the team that will be created by the provided team config.
After these changes are released to prod, we can update
https://github.com/fleetdm/fleet-gitops to use the new switches: #18692
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18464
Added `/fleet/autofill/policy` endpoint to get autogenerated policy
description and resolution for a given SQL query.
Added `server_settings.ai_features_disabled` setting to disable the
above endpoint.
For Google calendar integration,
- changed the event title to: "💻🚫 Scheduled maintenance"
- updated event description to include policy description and resolution
if only one policy is failing
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Luke Heath <luke@fleetdm.com>
