Additional telemetry stats to report to Heroku.
FYI - @eashaw, we could later add to datadog as well.
---------
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
## Addresses #18881 and #18858
- Fix the bugs, memoize various props to optimize table rendering
## QA
In addition to the bugs outlined in the addressed issue, this PR
contains rendering optimizations. Please check these functionalities,
which should be unaffected (and were so in my own testing) but should be
double-checked:
- [ ] create query
- [ ] delete query
- [ ] empty state
- [ ] changing platform dropdown
- [ ] different sort orders
- [ ] delete query from table action
- [ ] results count change
## Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#18838
1. During software ingestion, switched to updating `last_updated_at` as
a batch for 1 host.
2. Removed `DELETE FROM software` statement that ran for every host
update (where software was deleted). This is only done now during the
vulnerability job.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
For #16795, this:
- Updates Go to go1.22.3
- Per
https://github.com/fleetdm/fleet/issues/16795#issuecomment-2100450618, I
also ran the following to update the versions requested by @getvictor
```
go get github.com/kataras/golog@v0.1.12
go get github.com/kataras/iris/v12@v12.2.11
go get github.com/sethvargo/go-password@v0.3.0
```
**Notes**
After this is merged people will need to update their Go version. I use
gvm and I did it like:
```
$ gvm install go1.22.3
$ gvm use go1.22.3 --default
```
**Relevant changes**
The release notes mention:
> Previously, the variables declared by a “for” loop were created once
> and updated by each iteration. In Go 1.22, each iteration of the loop
> creates new variables, to avoid accidental sharing bugs.
However, we already have a lint rule (see
https://github.com/fleetdm/fleet/pull/13877) for this scenario, so it
shouldn't affect us.
> Related issue: #19153
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Feature cleanup
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
No related issue, just cleanup work on the feature
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18838 and #18986
Optimized master DB accesses during host software ingestion.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18766
Fixed a bug where updating user via `/api/v1/fleet/users/:id` endpoint
sometimes did not update the activity feed and returned the un-updated
user object.
You must use a DB configuration with a replica to reproduce the issue.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18944
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
#18477
Built-in labels can now be applied via `fleetctl apply` as long as no
changes are made to them. This allows the following workflow:
1. `fleetctl get labels --yaml > labels.yml`
2. (Optional) Edit/add non-built in labels in labels.yml
3. fleetctl apply -f labels.yml
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Tim Lee <timlee@fleetdm.com>
#18467
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- ~[ ] If database migrations are included, checked table schema to
confirm autoupdate~
- ~For database migrations:~
- ~[ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.~
- ~[ ] Confirmed that updating the timestamps is acceptable, and will
not cause unwanted side effects.~
- ~[ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#18221
Broke apart the hourly host_software count query to reduce the
individual query runtime. This fixes timeouts seen when host_software
table has over 25 million records.
I recommend hiding whitespace during review:
<img width="240" alt="image"
src="https://github.com/fleetdm/fleet/assets/2685025/6da9b643-8582-4d2f-bf32-8a1cc38f1032">
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
> Related issue: #18330
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
## Notes
- ~I added an `includeTitle bool` parameter to
`ds.GetSoftwareInstallerMetadata`. This allows for the title of the
software (from the `software_titles` page) to be fetched in
`svc.DeleteSoftwareInstaller` without an additional call to the DB.~ We
wound up deciding to just fetch the title every time.
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
#18808
Added the new `sofa_security_release_info` and `sofa_unpatched_cves`
tables from `macadmins/osquery-extension` 1.0.1
These tables do not have detailed documentation in macadmins repo, so
not adding documentation at this point.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
#12619
Fixed MySQL sort buffer overflow when fetching activities. This issue
happened when activities contained very large details, such as large SQL
queries.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- Existing tests provide sufficient coverage
- [x] Manual QA for all new/changed functionality
#16562
Optimized policy_stats updates to NOT lock the policy_membership table.
This should improve deployment performance with many global policies and
team hosts.
The original implementation that used INSERT ... SELECT (SELECT
COUNT(*)) ... caused performance issues. Given 50 global policies, 10
teams, and 10,000 hosts per team, the INSERT query took 30-60 seconds to
complete. Since it was an INSERT query, it blocked other hosts from
updating their policy results in policy_membership.
Now, we separate the INSERT from the SELECT, since SELECT by itself does
not block other hosts from updating their policy results. In addition,
we process one global policy at a time, which reduces the time to
complete the SELECT query to <2 seconds, and limits the memory usage. We
are not using a transaction to reduce locks. This means that INSERT may
fail if the policy was deleted by a parallel process. Also, the INSERT
may overwrite a clearing of the stats. This is acceptable, since these
are very rare cases. We log and proceed in that case.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
