**Related issue:** Resolves #
## Description
Bumps RustFS image version from `1.0.0-alpha.72` to `1.0.0-alpha.73` in
docker-compose configurations.
**Files updated:**
- `docker-compose.yml` - root development environment
- `tools/osquery/in-a-box/docker-compose.yml` - Fleet preview
environment
RustFS provides S3-compatible object storage for file carving and
software installer features in development/testing environments.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
## fleetd/orbit/Fleet Desktop
- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
<!-- START COPILOT ORIGINAL PROMPT -->
<details>
<summary>Original prompt</summary>
> ## Summary
> Update RustFS Docker image references from version `1.0.0-alpha.72` to
the latest version `1.0.0-alpha.73` in all docker-compose configuration
files.
>
> ## Files to Update
> Based on the code search, the following files need to be updated:
>
> 1. `docker-compose.yml` (root level)
> 2. `docs/solutions/docker-compose/docker-compose.yml`
> 3. `tools/osquery/in-a-box/docker-compose.yml`
>
> ## Changes Required
> For each file, update the RustFS image reference:
> - **FROM:** `rustfs/rustfs:1.0.0-alpha.72`
> - **TO:** `rustfs/rustfs:1.0.0-alpha.73`
>
> ## Context
> RustFS is used as an S3-compatible object storage backend in Fleet's
development and testing environments. Keeping the version up-to-date
ensures we have the latest bug fixes and improvements from the RustFS
project.
>
> ## Verification
> After making these changes, verify that:
> 1. All docker-compose files can start successfully
> 2. The S3-compatible storage functionality works as expected
> 3. File carving and software installer storage features continue to
work properly
</details>
<!-- START COPILOT CODING AGENT SUFFIX -->
*This pull request was created as a result of the following prompt from
Copilot chat.*
> ## Summary
> Update RustFS Docker image references from version `1.0.0-alpha.72` to
the latest version `1.0.0-alpha.73` in all docker-compose configuration
files.
>
> ## Files to Update
> Based on the code search, the following files need to be updated:
>
> 1. `docker-compose.yml` (root level)
> 2. `docs/solutions/docker-compose/docker-compose.yml`
> 3. `tools/osquery/in-a-box/docker-compose.yml`
>
> ## Changes Required
> For each file, update the RustFS image reference:
> - **FROM:** `rustfs/rustfs:1.0.0-alpha.72`
> - **TO:** `rustfs/rustfs:1.0.0-alpha.73`
>
> ## Context
> RustFS is used as an S3-compatible object storage backend in Fleet's
development and testing environments. Keeping the version up-to-date
ensures we have the latest bug fixes and improvements from the RustFS
project.
>
> ## Verification
> After making these changes, verify that:
> 1. All docker-compose files can start successfully
> 2. The S3-compatible storage functionality works as expected
> 3. File carving and software installer storage features continue to
work properly
<!-- START COPILOT CODING AGENT TIPS -->
---
💬 We'd love your input! Share your thoughts on Copilot coding agent in
our [2 minute survey](https://gh.io/copilot-coding-agent-survey).
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: iansltx <472804+iansltx@users.noreply.github.com>
Co-authored-by: Ian Littman <iansltx@gmail.com>
Resolves#36909.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
Redis 5 has been EOL for a few years, and didn't get updates for the
latest high-severity CVEs. We're already using 6 in most places
(fleetctl preview, recommended reference architectures, managed cloud
environments) so it's safe to set 6 as the new minimum.
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Adds support for reading server `private_key` from AWS Secrets Manager.
Combined with #31075, this should allow removing all common sensitive
secrets from the environment/config (if I missed any let me know). This
works with localstack for local development (set
`AWS_ENDPOINT_URL=$LOCALSTACK_URL`, `AWS_ACCESS_KEY_ID=test`, and
`AWS_SECRET_ACCESS_KEY=test`).
I did not include config options for `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` because they are a bad practice vs role
credentials and defeat the purpose of this feature which is to remove
secrets from the environment/config.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Scott Gress <scott@fleetdm.com>
Fixes#31781
Only test-related changes.
I found this useful for agentic AI workflows. For example, you have an
AI agent debugging/rerunning a test. Meanwhile, you can spin up another
`mysql_test` instance in another workarea and work there in parallel.
#29482
[Migrate to the AWS SDK for Go
v2](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/migrate-gosdk.html)
documents how to migrate codebases.
QA on features that use AWS SDK Go:
- Bootstrap package:
- upload: ✅
- download: ✅
- cleanup: ✅
- Software (upload, download, installation, etc.) ✅
- Cloudfront: Luckly, this feature was already using aws-sdk-go-v2.
- Carves ✅
- Logging:
- Firehose ✅
- Kinesis ✅
- Lambda ✅ (tested result logs to a lambda function on our AWS Dogfood
account)
- Email:
- Amazon SES TODO ⚠️ (this is what Dogfood uses and a few customers)
- We cannot easily test locally, we can use dogfood or load testing
(AWS) environments.
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Manual QA for all new/changed functionality
For #25160.
Measured improvement by splitting the MySQL query into two queries to
use the indexes more efficiently:
- ~8s vs ~100ms for ~30k entries in software_titles for ~1.7k incoming
software without bundle_identifier (linux software).
- ~1.64s vs ~2ms for 25k entries in software_titles and ~500 incoming
new software with bundle_identifier + ~200 new software without
bundle_identifier (macOS software).
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [X] Manual QA for all new/changed functionality
#15881
This PR adds a script to test DB migrations with Percona XtraDB 5.7.25.
PS: To run this test before we merge this PR to `main` you will need to
change step 2 (`Make sure to be on latest main`), instead of `main` use
this branch `15881-test-migrations-with-percona`.
#9609
This PR also fixes#10777.
The issue is: We were using `svc.AppConfig` instead of
`svc.ds.AppConfig` to retrieve the SMTP credentials.
`svc.AppConfig` obfuscates credentials, whereas `svc.ds.AppConfig` does
not.
To help prevent this from happening again I've renamed `svc.AppConfig`
to `svc.AppConfigObfuscated`.
I've also added a new test SMTP server
(https://github.com/axllent/mailpit) that supports Basic Authentication
and tests that make use of it to catch these kind of bugs (the tests are
executed when running `go test` with `MAIL_TEST=1`).
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
* WIP
* Adding DEP functionality to Fleet
* Better organize additional MDM code
* Add cmdr.py and amend API paths
* Fix lint
* Add demo file
* Fix demo.md
* go mod tidy
* Add munki setup to Fleet
* Add diagram to demo.md
* Add fixes
* Update TODOs and demo.md
* Fix cmdr.py and add TODO
* Add endpoints to demo.md
* Add more Munki PoC/demo stuff
* WIP
* Remove proposals from PoC
* Replace prepare commands with fleetctl commands
* Update demo.md with current state
* Remove config field
* Amend demo
* Remove Munki setup from MVP-Dogfood
* Update demo.md
* Add apple mdm commands (#7769)
* fleetctl enqueue mdm command
* fix deps
* Fix build
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
* Add command to upload installers
* go mod tidy
* fix subcommands help
There is a bug in urfave/cli where help text is not generated properly when subcommands
are nested too deep.
* Add support for installing apps
* Add a way to list enrolled devices
* Add dep listing
* Rearrange endpoints
* Move DEP routine to schedule
* Define paths globally
* Add a way to list enrollments and installers
* Parse device-ids as comma-separated string
* Remove unused types
* Add simple commands and nest under enqueue-command
* Fix simple commands
* Add help to enqueue-command
* merge apple_mdm database
* Fix commands
* update nanomdm
* Split nanomdm and nanodep schemas
* Set 512 MB in memory for upload
* Remove empty file
* Amend profile
* Add sample commands
* Add delete installers and fix bug in DEP profile assigning
* Add dogfood.md deployment guide
* Update schema.sql
* Dump schema with MySQL 5
* Set default value for authenticate_at
* add tokens to enrollment profiles
When a device downloads an MDM enrollment profile, verify the token passed
as a query parameter. This ensures untrusted devices don't enroll with
our MDM server.
- Rename enrollments to enrollment profiles. Enrollments is used by nano
to refer to devices that are enrolled with MDM
- Rename endpoint /api/<version>/fleet/mdm/apple/enrollments to ../enrollmentprofiles
- Generate a token for authentication when creating an enrollment profile
- Return unauthorized if token is invalid when downloading an enrollment profile from /api/mdm/apple/enroll?token=
* remove mdm apple server url
* update docs
* make dump-test-schema
* Update nanomdm with missing prefix table
* Add docs and simplify changes
* Add changes file
* Add method docs
* Fix compile and revert prepare.go changes
* Revert migration status check change
* Amend comments
* Add more docs
* Clarify storage of installers
* Remove TODO
* Remove unused
* update dogfood.md
* remove cmdr.py
* Add authorization tests
* Add TODO comment
* use kitlog for nano logging
* Add yaml tags
* Remove unused flag
* Remove changes file
* Only run DEP routine if MDM is enabled
* Add docs to all new exported types
* Add docs
* more nano logging changes
* Fix unintentional removal
* more nano logging changes
* Fix compile test
* Use string for configs and fix config test
* Add docs and amend changes
* revert changes to basicAuthHandler
* remove exported BasicAuthHandler
* rename rego authz type
* Add more information to dep list
* add db tag
* update deps
* Fix schema
* Remove unimplemented
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* Make migrations compatible with GTID replication
Fixes an issue some deployments encountered when migrations used a
statement that is unsupported in GTID replication mode (#2462).
Local dev MySQL now enforces this consistency, so it should be easier to
maintain compatibility going forward.
* Update docker-compose formatting
* if exists
* Migrate all mysql tests to the new form
* Only dump sql if MYSQL_TEST is on
* Removing parallel until we get rid of this code
* Move TestMain to an actual _test file
* A little experiment with tmpfs to speed up the db
* Let's make sure the dump.sql file is also in ram
- Update names/roles of users in `make e2e-setup`.
- Update test SSO user info.
- Add Cypress commands for seeding users/Teams.
- Stub Cypress tests for team/tier matrix.
Instead of synchronously updating the seen_time column for a host on an update, batch these updates to be written together every 1 second.
This results in a ~33% reduction in MySQL CPU usage in a local test with 4,000 simulated hosts and MySQL running in Docker.
Improves MySQL test time (on my 2020 MBP) to ~18s from ~125s.
- Use separate databases for each test to allow parallelization.
- Run migrations only once at beginning of tests and then reload
generated schema.
- Add `--innodb-file-per-table=OFF` for ~20% additional speedup.
- Add --dev flag that will set default flag values. This simplifies the
invocation of Fleet in a development environment.
- Change defaults in docker-compose to use `fleet` in place of `kolide`.
- Skip prompt in `prepare db` when `--dev` specified.
- Update developer documentation.
Updates to MySQL configuration in docker-compose.yml may require
existing development containers and volumes to be deleted (this will
delete data in MySQL):
```shell
docker-compose rm -sf
docker volume rm fleet_mysql-persistent-volume
```
Closes#170
* Use YAML anchors to avoid repeating config blocks
* Use docker volumes to persist data for mysql
* Allow setting `FLEET_SERVER` (fixes#2127) when using the docker-compose file to spin up multiple osquery clients
