#8593
This PR adds a new role `gitops` to Fleet.
MDM capabilities for the role coming on a separate PR. We need this
merged ASAP so that we can unblock the UI work for this.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [X] Documented any permissions changes
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [x] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
This fixes the following flaky tests:
- `TestHosts/LoadHostByOrbitNodeKey`
- `TestIntegrationsEnterprise/TestListSoftware`
- `TestHosts/ListStatus`
I couldn't figure out what's wrong with `TestScanVulnerabilities` which
is also randomly failing, and it's super slow to run (since it has to
download assets from GitHub) maybe @juan-fdz-hawa can spot it?
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Added/updated tests
### Related tickets
https://github.com/fleetdm/fleet/issues/10775https://github.com/fleetdm/fleet/issues/10678https://github.com/fleetdm/fleet/issues/11024https://github.com/fleetdm/fleet/issues/11026
### What's happening
- Implemented the hashing mechanism defined by @mna in #10678, however
this mechanism is mainly relevant for batch profile updates via the CLI,
we can't leverage it when a host switches teams.
- Modified `BulkSetPendingMDMAppleHostProfiles` so when two profiles
with the same identifier are sheduled both for removal and update, the
function will now mark only the `install` as `pending` so it's picked by
the cron, and will `DELETE` the `remove` entry from the database so it's
not picked by the cron and never sent to the user.
- `GetHostMDMProfiles` and consequently the profiles returned in `GET
/api/_version_/fleet/hosts` return `host_mdm_apple_profiles.state =
NULL` as "Enforcing (pending", the distinction between `status =
'pending'` and `status IS NULL` is only useful for the cron, for users
both mean the same thing, and all our profile aggregations already
behave this way.
- Using the solution implemented by @gillespi314 in
https://github.com/fleetdm/fleet/pull/10998 we're now deleting the host
row from `host_disk_encryption_keys` if a host is moved from a team that
enforces disk encryption to a team that doesn't.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
This should get version numbers for more apps on macOS. Notably,
1Password includes helper apps that were getting vulnerability false
positives because we were not picking up the versions.
Addresses #10702.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
#8593
This PR adds a new role `observer_plus` to Fleet. (The `GitOps` role
will be added on a separate PR.)
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [X] Documented any permissions changes
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
https://github.com/fleetdm/confidential/issues/1968
It's ready for review but I still need to load test this.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
relates to #9436
Implementation of the API supporting filtering host by disk encryption
status. This adds this through a `macos_settings_disk_encryption` query
param that can be passed to these endpoints:
`GET /hosts`
`GET /hosts/count`
`GET /lables/:id/hosts`
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Related to #9459, this adds logic to the cron to add a
`com.fleetdm.fleetd.config` configuration profile to the
`apple_mdm_configuration_profiles` table.
As noted in the comments, this makes some assumptions:
- This profile will be applied to all hosts in the team (or "no team",)
but it will only be used by hosts that have a fleetd installation
without
an enroll secret and fleet URL (mainly DEP enrolled hosts).
- Once the profile is applied to a team (or "no team",) it's not removed
if
AppConfig.MDM.AppleBMDefaultTeam changes, this is to preserve existing
agents using the configuration (mainly ServerURL as EnrollSecret is used
only during enrollment)
relates to #10786
This fixes an issue where users would see the incorrect disk encryption
banners on the my device page. This included a change to the ingestion
logic of the `directIngestDiskEncryptionKeyDarwin` method to take into
account if a host was already being encrypted with filevault locally.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Related to #10787, this tries to find in the tables with High likelihood
described in the issue.
This successfully accounts for unique keys that contain leading/trailing
whitespace and are using a collation with a pad attribute set to `NO
PAD` (considers whitespace as any other character instead of ignoring
it)
I haven't found a way to successfully detect the same scenario for
special unicode characters, for example:
```
mysql> SELECT TABLE_NAME, TABLE_COLLATION FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 'software';
+------------+--------------------+
| TABLE_NAME | TABLE_COLLATION |
+------------+--------------------+
| software | utf8mb4_general_ci |
+------------+--------------------+
1 row in set (0.01 sec)
mysql> select vendor COLLATE utf8mb4_unicode_ci from software where name = 'zchunk-libs' GROUP BY vendor COLLATE utf8mb4_unicode_ci;
+-----------------------------------+
| vendor COLLATE utf8mb4_unicode_ci |
+-----------------------------------+
| vendor |
| vendor? |
+-----------------------------------+
2 rows in set (0.01 sec)
mysql> ALTER TABLE `software` CONVERT TO CHARACTER SET `utf8mb4` COLLATE `utf8mb4_unicode_ci`;
ERROR 1062 (23000): Duplicate entry 'zchunk-libs-1.2.1-rpm_packages--vendor\2007-x86_64' for key 'unq_name'
```
> **Note** that `?` in "vendor?" is an unicode character
This only applies to Premium users, we want to show the vulnerabilities' published date anywhere vulnerabilities are shown including API endpoints and third party integrations.
#9609
This PR also fixes#10777.
The issue is: We were using `svc.AppConfig` instead of
`svc.ds.AppConfig` to retrieve the SMTP credentials.
`svc.AppConfig` obfuscates credentials, whereas `svc.ds.AppConfig` does
not.
To help prevent this from happening again I've renamed `svc.AppConfig`
to `svc.AppConfigObfuscated`.
I've also added a new test SMTP server
(https://github.com/axllent/mailpit) that supports Basic Authentication
and tests that make use of it to catch these kind of bugs (the tests are
executed when running `go test` with `MAIL_TEST=1`).
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
relates to #9434
implements the `GET /fleet/mdm/apple/filevault/summary` aggregate
endpoint.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
closes https://github.com/fleetdm/fleet/issues/10778
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
#8957
To test this feature, build+run Fleet and then visit:
`https://localhost:8080/metrics`.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [ ] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
this cover edge cases when the `/var/db/FileVaultPRK.dat` is not
present, but the disk is still encrypted and thus we're not able to get
the encryption key.
https://github.com/fleetdm/fleet/issues/10672
- Broke up the single MDM doc into multiple ones organized by category
- Changed any links to point to the new docs
---------
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
related to #10441, inspired by the prior work done in
https://github.com/kolide/fleet/pull/1360, this PR:
1. Adds a migration to use `utf8mb4_general_ci` as the default collation
for the database and all the tables. From [MySQL's documentation][1]:
> To change the table default character set and all character columns
> (CHAR, VARCHAR, TEXT) to a new character set, use a statement like
> this:
>
> ```
> ALTER TABLE tbl_name CONVERT TO CHARACTER SET charset_name;
> ```
> The statement also changes the collation of all character columns. If
> you specify no COLLATE clause to indicate which collation to use, the
> statement uses default collation for the character set.
2. Changes the connection settings to use `utf8mb4_general_ci` as the
default collation, from the [driver docs][2]:
> Sets the collation used for client-server interaction on
connection. In contrast to charset, collation does not issue additional
queries. If the specified collation is unavailable on the target server,
the connection will fail.
[1]: https://dev.mysql.com/doc/refman/5.7/en/alter-table.html
[2]: https://github.com/go-sql-driver/mysql
**TODO:** discuss how we can enforce this, is setting the database
default collation enough? should we add some kind of custom lint rule to
all migrations?
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
