related to #10441, inspired by the prior work done in
https://github.com/kolide/fleet/pull/1360, this PR:
1. Adds a migration to use `utf8mb4_general_ci` as the default collation
for the database and all the tables. From [MySQL's documentation][1]:
> To change the table default character set and all character columns
> (CHAR, VARCHAR, TEXT) to a new character set, use a statement like
> this:
>
> ```
> ALTER TABLE tbl_name CONVERT TO CHARACTER SET charset_name;
> ```
> The statement also changes the collation of all character columns. If
> you specify no COLLATE clause to indicate which collation to use, the
> statement uses default collation for the character set.
2. Changes the connection settings to use `utf8mb4_general_ci` as the
default collation, from the [driver docs][2]:
> Sets the collation used for client-server interaction on
connection. In contrast to charset, collation does not issue additional
queries. If the specified collation is unavailable on the target server,
the connection will fail.
[1]: https://dev.mysql.com/doc/refman/5.7/en/alter-table.html
[2]: https://github.com/go-sql-driver/mysql
**TODO:** discuss how we can enforce this, is setting the database
default collation enough? should we add some kind of custom lint rule to
all migrations?
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#9132
The actual fix for the empty hosts is adding the `--database_path`
argument in the initial `osqueryd -S` invocation when retrieving the
UUID. Osquery attempts to retrieve the UUID from OS files/APIs, when not
possible (which is what happens on some linux distributions), then it
resorts to generating a new random UUID and storing it in the
`osquery.db`. The issue was Orbit's first invocation of `osqueryd -S`
was not using the same `osquery.db` as the main daemon invocation of
`osqueryd`.
I'm also adding a `hostname` + `platform` to the orbit enroll phase so
that if there are any issues in the future we can avoid the "empty" host
and have some information to help us troubleshoot.
## How to reproduce
On Linux, osquery reads `/sys/class/dmi/id/product_uuid` to load the
hardware UUID.
Some Linux distributions running on specific hardware or container
environments do not have such file available.
The way to reproduce on a Linux VM is to do the following:
```sh
$ sudo su
# chmod -r /sys/class/dmi/id/product_uuid
```
which will turn the file inaccessible to root.
## Checklist
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
#9486
Now cron jobs should recover from a Fleet outage after ~ two hours.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#8129
Apart from fixing the issue in #8129, this change also introduces UUIDs
to Fleet errors. To be able to match a returned error from the API to a
error in the Fleet logs. See
https://fleetdm.slack.com/archives/C019WG4GH0A/p1677780622769939 for
more context.
Samples with the changes in this PR:
```
curl -k -H "Authorization: Bearer $TEST_TOKEN" -H 'Content-Type:application/json' "https://localhost:8080/api/v1/fleet/sso" -d ''
{
"message": "Bad request",
"errors": [
{
"name": "base",
"reason": "Expected JSON Body"
}
],
"uuid": "a01f6e10-354c-4ff0-b96e-1f64adb500b0"
}
```
```
curl -k -H "Authorization: Bearer $TEST_TOKEN" -H 'Content-Type:application/json' "https://localhost:8080/api/v1/fleet/sso" -d 'asd'
{
"message": "Bad request",
"errors": [
{
"name": "base",
"reason": "json decoder error"
}
],
"uuid": "5f716a64-7550-464b-a1dd-e6a505a9f89d"
}
```
```
curl -k -X GET -H "Authorization: Bearer badtoken" "https://localhost:8080/api/latest/fleet/teams"
{
"message": "Authentication required",
"errors": [
{
"name": "base",
"reason": "Authentication required"
}
],
"uuid": "efe45bc0-f956-4bf9-ba4f-aa9020a9aaaf"
}
```
```
curl -k -X PATCH -H "Authorization: Bearer $TEST_TOKEN" "https://localhost:8080/api/latest/fleet/users/14" -d '{"name": "Manuel2", "password": "what", "new_password": "p4ssw0rd.12345"}'
{
"message": "Authorization header required",
"errors": [
{
"name": "base",
"reason": "Authorization header required"
}
],
"uuid": "57f78cd0-4559-464f-9df7-36c9ef7c89b3"
}
```
```
curl -k -X PATCH -H "Authorization: Bearer $TEST_TOKEN" "https://localhost:8080/api/latest/fleet/users/14" -d '{"name": "Manuel2", "password": "what", "new_password": "p4ssw0rd.12345"}'
{
"message": "Permission Denied",
"uuid": "7f0220ad-6de7-4faf-8b6c-8d7ff9d2ca06"
}
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Just improving logs to detect platforms that Fleet does not recognize
yet (mostly Linux distributions)
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
* Change order of returned json fields
* Change field "failed" to "failing"
- [x] Manual QA
- [x] Updated docs
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
Related to #9495, this adds the underlying methods to send a
configuration profile that enables FileVault and FileVault Escrow, so we
can fetch and decrypt the encryption key later on.
These methods still need to be called somewhere, and they might need to
be moved outside of `Service`, but at least this gives us a start.
Issue #9599
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
We have code that builds conditionally depending on the platform (mostly
Orbit code) so we should run `golangci-lint` checks on all OSs.
This adds it to run on macOS, for Windows see:
https://github.com/fleetdm/fleet/issues/9943
Related to https://github.com/fleetdm/fleet/issues/9653 I couldn't find
any documentation to back this up, but I have a strong suspicion that
the `os` field in the device sync response might come empty in some
scenarios (particularly, when a laptop is brand new, which is hard to
reproduce 😅)
My thoughts are:
1. For the recently purchased MacBooks,
`IngestMDMAppleDevicesFromDEPSync` didn't create an entry in the
database, BUT `nanodep.Assigner.ProcessDeviceResponse` correctly
assigned a DEP profile (the devices were able to enroll). Both methods
filter by `op_type` but only ours filters by `os`.
2. I think this is safe-ish to do, as you will normally assign a MDM
server per device type in ABM
3. I have added extra logs to try to prove this hypothesis next time a
brand new device comes in, let's keep an eye on and re-evaluate this
approach.
Related to #9571, this adds a new value to both responses which is
calculated when the Fleet server is started, and only set to `true` if
the server is properly configured for MDM.
This helps the UI to determine wether or not we should show certain UI
elements that we only want to show to servers with MDM enabled.
#8948
- Add more go:generate commands for MDM mocks
- Add unit and integration tests for MDM code
- Move interfaces from their PoC location to match existing patterns
See requirements in #8682.
Two assumptions on the implementation (@zayhanlon please take a look):
- Hosts explicitly selected to run always run the live query (no matter
the values on the selectors).
- When selecting `All hosts`, selecting any other platform or label is
kind of a no-op. We should look into graying out all the selectors if
the user selects `All hosts`.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
relates to https://github.com/fleetdm/fleet/issues/8928
This adds a new `meta` attribute to the "GET /activities" endpoint that
includes pagination metadata. This can allow clients to know if there
are additional items to request.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
