Commit graph

250 commits

Author SHA1 Message Date
Lucas Manuel Rodriguez
682202444c
Update go to 1.26.2 and update tooling to update it (#43771)
Golang 1.26.2 has been released. It fixes some CVEs:
https://github.com/golang/go/issues?q=milestone%3AGo1.26.2+label%3ACherryPickApproved

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated Go toolchain to 1.26.2 across the repository and build
configs.
  * Updated Docker build images to use Go 1.26.2.
* Expanded the set of tracked modules for the Go version update so
additional module files are included in automated updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-20 13:40:57 -03:00
Lucas Manuel Rodriguez
1f5e4a0674
Add a way to test osquery PR on local fleetd TUF (#43624)
This is a way to test osquery PRs as part of local fleetd TUF builds.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **Chores**
* Enhanced macOS build process to support creating application bundles
from pull request workflow artifacts in addition to released versions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-20 12:51:02 -03:00
Lucas Manuel Rodriguez
fa71b582d1
Add -s -w when building fleetd components to remove debugging information and reduce binary sizes (#43260)
Resolves #43259.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] QA'd all new/changed functionality manually

## fleetd/orbit/Fleet Desktop

- [x] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [x] Verified that fleetd runs on macOS, Linux and Windows
- [x] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
2026-04-08 18:30:05 -03:00
Scott Gress
c4aa6f5529
Use fleetctl new templates for new instances (#42768)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #41409 

# Details

This PR updates the `ApplyStarterLibrary` method and functionality to
rely on the same templates and mechanisms as `fleetctl new`. The end
result is that running `fleetctl new` and `fleetctl gitops` on a new
instance should be a no-op; no changes should be made. Similarly,
changing the templates in a Fleet release will automatically affect
`fleetctl new` and `ApplyStarterLibrary` in the same exact way for that
release.

> Note that this moves the template files out of `fleetctl` and into
their own shared package. This move comprises the majority of the file
changes in the PR.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [X] Added/updated automated tests
Note that 

<img width="668" height="44" alt="image"
src="https://github.com/user-attachments/assets/066cd566-f91d-4661-84fc-2aabbfce2ef9"
/>

will fail until the 4.83 Fleet docker image is published, since it's
trying to push 4.83 config (including `exceptions`) to a 4.82 server.

- [X] QA'd all new/changed functionality manually
- [X] Created a new instance and validated that the fleets, policies and
labels created matched the ones created by `fleetctl new`
- [X] Ran `fleetctl new` and verified that it created the expected
folders and files
- [X] Ran `fleetctl gitops` with the files created by `fleetctl new` and
verified that the instance was unchanged.
- [X] Ran `fleetctl preview` successfully using a dev build of the Fleet
server image (since it won't work against the latest published build,
which doesn't support `exceptions`). Verified it shows the expected
teams, policies and labels
2026-04-03 09:58:03 -05:00
Victor Lyuboslavsky
b1c85af75d
Updates to Android RELEASE.md (#42959)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Improved Android release process with automated changelog generation
and streamlined release workflow documentation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-03 09:00:39 -05:00
Lucas Manuel Rodriguez
0b8c29198b
Make orbit and Fleet Desktop not depend on server/service/ packages (#42231)
Resolves #40396.

No changes file because there should be no user visible changes.

## Testing

- [x] QA'd all new/changed functionality manually

## fleetd/orbit/Fleet Desktop

- [x] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [X] Verified that fleetd runs on macOS, Linux and Windows
- [X] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
2026-03-26 10:59:42 -03:00
Konstantin Sykulev
75c05aeb3f
adding gotestsum for better test output (#40753)
**Related issue:** Resolves #30896

## Testing

- [x] QA'd all new/changed functionality manually
2026-03-03 16:01:11 -06:00
Victor Lyuboslavsky
05648093f1
Added dedicated Go test workflow for activity bounded context (#40653) 2026-02-27 07:53:57 -06:00
Victor Lyuboslavsky
9a25a2b43d
Fix broken lint on main. (#40147)
`make lint-go` was not flagging issues in CI.
2026-02-20 07:15:23 -06:00
Ian Littman
6570e8846a
🤖 Update goreleaser snapshot build to pull version number from branch name when applicable (#39593)
For #39522.
2026-02-10 13:51:52 -06:00
Ian Littman
2b5eedac5e
🤖 Improve build version population based on git branch (#39523)
For #39522. Guessing this doesn't resolve goreleaser issues though?

Tested with this branch (shows a snapshot), `4.77.9-plus-sparklies`,
`v4.77.12-without-sparkles`, and `rc-patch-fleet-v4.77.7` locally to
make sure this behaves properly.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] QA'd all new/changed functionality manually

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: iansltx <472804+iansltx@users.noreply.github.com>
2026-02-09 08:43:02 -06:00
jacobshandling
79b7d83bf5
Use nilaway to incrementally check for unsafe nil pointer dereferences (#39030)
**Related issue:** Resolves #32999 

- Enhanced internal code quality tooling by implementing a custom
linting build configuration.
- Updated continuous integration workflow to utilize the new custom
build process for improved code analysis and consistency checks.

### Confirmed that running local custom `golangci-lint` build with
`nilaway` plugin catches lots of issues when run on `fleet/`:
<img width="1555" height="939" alt="Screenshot 2026-01-29 at 2 47 50 PM"
src="https://github.com/user-attachments/assets/c6a18400-fdf0-4104-97d8-e117efc28ed6"
/>
<img width="301" height="109" alt="Screenshot 2026-01-29 at 2 48 36 PM"
src="https://github.com/user-attachments/assets/b459ee7b-b391-457a-9191-17d56a80c783"
/>

### Confirmed that new incremental CI step using custom `golangci-lint`
build with `nilaway` plugin _does not_ check any `.go` files when none
have been modified, and so passes successfully (incremental check works
as expected):
<img width="337" height="197" alt="Screenshot 2026-01-29 at 2 45 24 PM"
src="https://github.com/user-attachments/assets/c7ae585e-2e10-4ebf-a3a3-96c26063f1e4"
/>

### Confirmed that new incremental CI step using custom `golangci-lint`
build with `nilaway` plugin _does_ check modified lines of `.go` files,
and so successfully flags a potentially unsafe dereference and fails the
job (incremental check works as expected):
<img width="825" height="491" alt="Screenshot 2026-01-29 at 5 50 01 PM"
src="https://github.com/user-attachments/assets/82bc5616-6fb9-4357-b8bc-c7eebc42c2d8"
/>

### Honorable mention:
`nilaway` agrees that `listHostSoftware` is a wild beast:
<img width="1277" height="190" alt="Screenshot 2026-01-29 at 5 52 32 PM"
src="https://github.com/user-attachments/assets/dfade2a8-fbcc-4bae-98f9-6bf1089620d2"
/>

- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Fleet dev cycle reliability improvements**


<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Victor Lyuboslavsky <2685025+getvictor@users.noreply.github.com>
2026-02-06 08:51:17 -06:00
Nico
2a23fe1724
Fix: GLIBC version incompatibility causes orbit agent failure on Ubuntu 20.04 during fleet-desktop update (#38648)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #35413 

Applied the same fix as in
https://github.com/fleetdm/fleet/pull/29186/files:
- Added musl-tools to the container image. This provides musl-gcc, a
compiler that links against musl libc instead of glibc. We use it for
static linking as explained below.
- Added static linking flags (CGO_ENABLED=1 CC=musl-gcc -linkmode
external -extldflags "-static"). This produces a self-contained binary
with all C library code embedded, eliminating runtime dependencies on
the host system's glibc version.

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

Steps:

- Started TUF server for the first time. Generated a Linux x86_64 image.

```
SYSTEMS="linux" \                                      
DEB_FLEET_URL=https://nicofleet.ngrok.io \
DEB_TUF_URL=http://nicotuf.ngrok.io \
GENERATE_DEB=1 \
ENROLL_SECRET=tm2CHBEF1I5BVuM1+4hzRRtpC5ZYV8vb \
FLEET_DESKTOP=1 \
DEBUG=1 \
./tools/tuf/test/main.sh
```

- Installed `fleet-osquery_26.1.46030_amd64.deb` generated by the
previous command on a Kubuntu 20.04 x86_64 VM.
- Ran `sudo journalctl -u orbit` to debug logs and verified that the
GLIBC incompatibility error was raised:

```
tammi 22 15:36:53 nicolas-Standard-PC-i440FX-PIIX-1996 sudo[125623]: pam_unix(sudo:session): session opened for user nicolas by (uid=0)

tammi 22 15:36:53 nicolas-Standard-PC-i440FX-PIIX-1996 orbit[125624]: /opt/orbit/bin/desktop/linux/stable/fleet-desktop/fleet-desktop: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/orbit/bin/desktop/linux/stable/fleet-desktop/fleet-desktop)

tammi 22 15:36:53 nicolas-Standard-PC-i440FX-PIIX-1996 orbit[125624]: /opt/orbit/bin/desktop/linux/stable/fleet-desktop/fleet-desktop: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /opt/orbit/bin/desktop/linux/stable/fleet-desktop/fleet-desktop)

tammi 22 15:36:53 nicolas-Standard-PC-i440FX-PIIX-1996 sudo[125623]: pam_unix(sudo:session): session closed for user nicolas

tammi 22 15:37:08 nicolas-Standard-PC-i440FX-PIIX-1996 orbit[125199]: 2026-01-22T15:37:08+02:00 INF killing any pre-existing fleet-desktop instances

tammi 22 15:37:09 nicolas-Standard-PC-i440FX-PIIX-1996 orbit[125199]: 2026-01-22T15:37:09+02:00 INF attempting to get user session type and display id=1000 user=nicolas

tammi 22 15:37:09 nicolas-Standard-PC-i440FX-PIIX-1996 orbit[125199]: 2026-01-22T15:37:09+02:00 ERR failed to get X11 display, using default :0 error="display not found on who output"
```

- Built a new version of the agent after applying the fixes on this PR
and pushed it to the TUF server:

```
# 1. Hardcode orbit to a higher version
export ORBIT_VERSION=26.1.46099

#2. Generate new package
FLEET_DESKTOP_VERSION=$ORBIT_VERSION make desktop-linux

#3. Update to TUF server
./tools/tuf/test/push_target.sh linux desktop desktop.tar.gz $ORBIT_VERSION
```

<img width="396" height="179" alt="Screenshot 2026-01-22 at 5 18 25 PM"
src="https://github.com/user-attachments/assets/56182580-1d54-4945-af03-98762e7795e9"
/>


- In the VM, verified that an update for fleet desktop was detected by
running `sudo journalctl -u orbit -g "update detected"`:

```
tammi 22 20:33:32 nicolas-Standard-PC-i440FX-PIIX-1996 orbit[4114]: 2026-01-22T20:33:32+02:00 INF update detected target=desktop
```

- Verified the new version is shown both on the desktop icon and the
Fleet UI:

<img width="1373" height="248" alt="Screenshot 2026-01-22 at 5 22 17 PM"
src="https://github.com/user-attachments/assets/9fac73ac-a3a6-49ed-82c7-261abec43798"
/>

<img width="1420" height="496" alt="Screenshot 2026-01-22 at 5 22 46 PM"
src="https://github.com/user-attachments/assets/55321108-9233-4799-b5e5-0713172c0138"
/>


## fleetd/orbit/Fleet Desktop

- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [x] Verified that fleetd runs on macOS, Linux and Windows
- [x] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
2026-01-23 12:02:03 -03:00
Ian Littman
fe2a9a867e
Swap minio to rustfs (#36851)
Resolves #36909.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
2025-12-10 10:03:48 -06:00
Jonathan Katz
b8d2ba371e
Update replica db setup and guide (#36918)
**Related issue:** Resolves #35937
Changes:
- Fix `make db-replica-setup`
- mention the tool in
`docs/contributing/getting-started/testing-and-local-development.md`.
2025-12-08 17:07:04 -05:00
Victor Lyuboslavsky
40022c5537
Add incremental lint run, with modernize as the linter. (#36711)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32999

The `modernize` linter was discussed some time ago in the backend sync.
We wanted to add it but it was not possible.

Now that it has been added to golangci-lint, we are adding it.

golangci-lint has incremental mode, where only changes vs the base
branch are linted. This is nice when adding new linters without needing
to fix the whole codebase. That said, it would be nice to `modernize`
the whole codebase.
2025-12-08 09:06:05 -06:00
Lucas Manuel Rodriguez
93bea23859
Add CVE scanning for fleetdm/wix and split out workflows (#33687)
Attempting to fix
https://github.com/fleetdm/fleet/actions/runs/18120473187/job/51564073671#step:11:38

Changes:
- The docker images are too big so they started to cause issues in
Github runners. Thus I'm splitting the one workflow into three separate
workflows, one for each image.
- While we are at it: start tracking vulnerabilities in fleetdm/wix and
fleetdm/bomutils.

New runs:
-
https://github.com/fleetdm/fleet/actions/runs/18161326970/job/51692559418
-
https://github.com/fleetdm/fleet/actions/runs/18161326953/job/51692559257
-
https://github.com/fleetdm/fleet/actions/runs/18161326952/job/51692559172
2025-10-01 19:37:22 -03:00
Victor Lyuboslavsky
e872f9a984
Update golangci-lint to v2.4.0 (#33251)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33250

Waived most new failures. Planning to come back and fix some of them in
subsequent PRs.
2025-09-22 13:17:11 -05:00
Jordan Montgomery
596fb70399
Update Makefile swift dialog versions and add github workflow (#32511)
For #31675
For #32099 

Adds a Github workflow to generate our packaged build of Swift
Dialog(following existing Nudge packager), updates the version to 2.5.6
and modifies the Migration dialog to render properly with the new Swift
Dialog version(it previously rendered it just didn't format as expected
due to changes in the markdown formatter)

Makefile changes are necessary not only because of the version bump but
because the latest package includes xattrs for some strange reason.
Extracting it verbatim on a system(at least with our Go implementation)
creates files that cause Gatekeeper to stop execution of swift dialog.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] QA'd all new/changed functionality manually

## fleetd/orbit/Fleet Desktop

- [x] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [x] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [x] Verified that fleetd runs on macOS, Linux and Windows
- [x] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
2025-09-05 10:49:21 -04:00
Carlo
8bc8d01f0a
Merge Android datastore into main Fleet datastore (#32233)
Resolves #31218
2025-08-25 11:41:28 -04:00
Victor Lyuboslavsky
98954e8113
Make MySQL test port configurable (#31782)
Fixes #31781 

Only test-related changes.

I found this useful for agentic AI workflows. For example, you have an
AI agent debugging/rerunning a test. Meanwhile, you can spin up another
`mysql_test` instance in another workarea and work there in parallel.
2025-08-12 15:45:43 +02:00
Lucas Manuel Rodriguez
4265bed7f1
Make sure VEX report is up-to-date with a CI check (#31759) 2025-08-11 14:55:31 -05:00
Victor Lyuboslavsky
0180cc8086
Add SCEP endpoint for host identity. (#30589)
Fixes #30458 

Contributor docs PR: https://github.com/fleetdm/fleet/pull/30651

# Checklist for submitter

- We will add changes file later.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- Did not do manual QA since the SCEP client I have doesn't support ECC.
Will rely on next subtasks for manual QA.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Introduced Host Identity SCEP (Simple Certificate Enrollment Protocol)
support, enabling secure host identity certificate enrollment and
management.
* Added new API endpoints for Host Identity SCEP, including certificate
issuance and retrieval.
* Implemented MySQL-backed storage and management for host identity SCEP
certificates and serials.
* Added new database tables for storing host identity SCEP certificates
and serial numbers.
* Provided utilities for encoding certificates and keys, and handling
ECDSA public keys.

* **Bug Fixes**
  * None.

* **Tests**
* Added comprehensive integration and unit tests for Host Identity SCEP
functionality, including certificate issuance, validation, and error
scenarios.

* **Chores**
* Updated test utilities to support unique test names and new SCEP
storage options.
* Extended mock datastore and interfaces for new host identity
certificate methods.

* **Documentation**
* Added comments and documentation for new SCEP-related interfaces,
methods, and database schema changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-07-11 11:44:07 -03:00
Lucas Manuel Rodriguez
5646062c85
Update go to 1.24.4 and add some automation (#29954)
Fixes CVE-2025-22874 reported by
https://github.com/fleetdm/fleet/actions/runs/15601368321/job/43941793647.

(IMO not a critical CVE, so it doesn't need to be cherry-picked into
v4.69.0.)

Added automation to make this easier next time.
2025-06-13 13:08:14 -05:00
Luke Heath
63596c761e
Update changelog command (#29382) 2025-05-27 15:17:47 -05:00
Lucas Manuel Rodriguez
7f85e529be
Iterate status.md for reporting vulnerability updates (#29062)
For #28805.

See scenario we want to support in the linked issue.

---------

Co-authored-by: Scott Gress <scottmgress@gmail.com>
2025-05-15 21:15:37 -03:00
Scott Gress
7225625bbb
Fix make so that it builds both binaries again (#28538)
Previous work on the makefile broke the default `make` action. This
restores it by setting default binaries to be built.
2025-04-24 16:03:21 -05:00
Lucas Manuel Rodriguez
895194d63b
Add scanning to released images and process to track vulnerabilities (#28087)
For #25902.

---------

Co-authored-by: Sharon Katz <121527325+sharon-fdm@users.noreply.github.com>
2025-04-16 11:50:10 -03:00
Dante Catalfamo
94f6127edc
Orbit for Windows ARM64 (#27882)
#27275 and #27274

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Make sure fleetd is compatible with the latest released version of
Fleet (see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/fleetd-development-and-release-strategy.md)).
- [x] Orbit runs on macOS, Linux and Windows. Check if the orbit
feature/bugfix should only apply to one platform (`runtime.GOOS`).
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).

---------

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
2025-04-11 10:18:28 -04:00
Victor Lyuboslavsky
7558d5afa1
Improvements to test-go CI job (#27916)
This PR makes several improvements to test-go CI job
- remove ZSH dependency from all test suites except for a new `scripts`
suite
- add a `fast` suite that does not have out-of-process dependencies,
which completes in ~1.5 minutes
- contributors can add their fast tests to this suite so they can see
the results in CI faster
- Rename `core` to `main` test suite to be consistent with Makefile. It
is the default bucket for tests.
- Cleaned up Makefile so that it is more straightforward to add new test
suites or move Go packages between suites
- Do not stop the test suites on a fail.
- We do not want to be blocked by a test fail that another product team
introduced
- Sometimes, we want to see all test failures so we can fix them all at
once.
- Removed `test-schema` and `mock` prerequisites for `test-go` since
they are not needed and just take up time.
- But also added `test-schema` run to one of the test suites just in
case.

Unfortunately, `fleetctl` is still the bottleneck and needs to be
refactored. New issue filed:
https://github.com/fleetdm/fleet/issues/27927

## Before
<img width="248" alt="image"
src="https://github.com/user-attachments/assets/110ffc1d-f090-4d3e-be77-0419b9577d20"
/>

## After
<img width="320" alt="image"
src="https://github.com/user-attachments/assets/8d01ea11-408f-4eb6-81d8-9c25410b8830"
/>
2025-04-07 12:52:26 -05:00
Lucas Manuel Rodriguez
8003227554
Use go 1.24 new tool feature (#27765)
Release notes: https://tip.golang.org/doc/go1.24

> Go modules can now track executable dependencies using tool directives
in go.mod. This removes the need for the previous workaround of adding
tools as blank imports to a file conventionally named “tools.go”. The go
tool command can now run these tools in addition to tools shipped with
the Go distribution. For more information see [the
documentation](https://tip.golang.org/doc/modules/managing-dependencies#tools).
The new -tool flag for go get causes a tool directive to be added to the
current module for named packages in addition to adding require
directives.

I ran:
```
go get -tool github.com/fleetdm/fleet/v4/server/goose
go get -tool github.com/kevinburke/go-bindata
go get -tool github.com/quasilyte/go-ruleguard/dsl
go rm tools.go
go mod tidy
```

`make deps-go` was failing in CI because of the removal of `tools.go`
(my guess is that `go get .` was a nop because there was nothing in `.`
to download).
So, taking the chance of removing `deps-go` because `go` will download
packages during the build process. AFAICS there's no need to download
everything beforehand.
2025-04-07 11:12:05 -03:00
Scott Gress
d51f2815ad
FDM updates: fdm serve, snapshot/restore improvements (#27890)
For #27889 

This PR introduces several improvements to the Makefile/`fdm` tool for
development:
 
### `fdm serve` (alias `fdm up`)

Starts a local Fleet server (building the binary first). The first time
this is called, it will start the server on `localhost:8080` with the
`--dev` and `--dev_license` flags, but the command accepts all of the
options that you can pass to `fleet serve`. If you pass options to `fdm
serve`, then subsequent invocations _without_ options will replay your
last command. Additionally, `fdm serve` supports the following:

- `--use-ip`: start the local server on your system's local IP address
rather than `localhost`. This makes it easier to point VMs on your
system to the fleet server to act as hosts.
- `--no-build`: don't rebuild the fleet binary before starting the
server.
- `--no-save`: don't save the current command for future invocations
(useful for scripting)
- `--show`: show options for the last-invoked `fdm serve` command
- `--reset`: reset the options for `fdm serve`. The next time `fdm
serve` is invoked, it will use the default options.
- `--help`: show all of the Fleet server options

### `fdm snapshot` improvements

* Added `fdm snap` alias
* Tracks the name of the last snapshot saved, to use as the default for
`fdm restore`
* Suppresses the "don't use password in CLI" warning when saving the
snapshot

### `fdm restore` improvements

* Added `--prep` / `--prepare` option to run db migrations after
restoring snapshot.
* Improved UI (more options displayed, and clearer indicator for
selected option)
* Now defaults to last snapshot restored
2025-04-07 09:10:15 -05:00
Victor Lyuboslavsky
527428f0b4
Speeding up Go tests in CI. (#27434)
Using `go get .` (~3s) is faster than `go mod download` (~13s) in CI.
2025-03-25 12:03:24 -05:00
Victor Lyuboslavsky
46719c9fb2
Auto-generate and check Android schema.sql (#26720)
For #26218 

This is an infrastructure change. No functional feature changes.
2025-02-28 16:30:40 -06:00
Scott Gress
f7e4a39915
Add help system to Makefile + FDM command (#25028) 2025-02-28 07:42:32 -06:00
Victor Lyuboslavsky
3d5666d4c6
Added GET enterprise API endpoint. (#26555)
For #26218 

- Added `GET /api/_version_/fleet/android_enterprise` andpoint and tests
- Set up some testing infrastructure for Android service tests -- see
new README.md

# Checklist for submitter

- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
2025-02-26 10:47:05 -06:00
Lucas Manuel Rodriguez
f9b08468d7
Fix automation for fleetd orbit/TUF.md (#26406)
Scheduled workflow to update the orbit/TUF.md has been failing since I
changed the tooling to use the new TUF:
https://github.com/fleetdm/fleet/actions/workflows/fleetd-tuf.yml.
2025-02-18 10:34:52 -03:00
Lucas Manuel Rodriguez
9114d0405b
Update TUF status generation to use new TUF repository (#26099)
For #25853.
2025-02-07 08:30:07 -03:00
Scott Gress
94eb573736
Add Fleet dev snapshot tool (#25909)
For #23750 

# Overview

This PR adds a basic tool for creating and restoring Fleet dev
snapshots. In this first iteration a snapshot is just a folder
containing a MySQL db dump made using the existing backup/restore
scripts, and the tool allows you to easily save and restore snapshots
interactively.

## Usage

* `make snapshot` to create a new snapshot
* `make restore` to select and restore a snapshot 

## Future plans

Future iterations can add metadata to snapshots to integrate things
like:

* node keys from osquery-perf, so you can easily reconnect to hosts
created in a previous session
* env vars from when the snapshot was made
* the branch from when the snapshot was made, to allow switching to that
branch and restarting the server as part of the restore process
*
![image](https://github.com/user-attachments/assets/6dc86581-5c12-4b57-b900-5034e00bc496)

## Demo


https://github.com/user-attachments/assets/1590c37a-3df9-4201-a42b-ccd1a36cb6cf
2025-02-05 09:52:10 -06:00
Lucas Manuel Rodriguez
a4db139e82
Remove fleetctl binary from fleetdm/fleet image and remove unused Dockerfile (#25749)
For #25748.

Manually tested by:
1. Building a `fleet` executable for Linux on a Ubuntu VM (with
`-extldflags "-static"`) for Alpine.
2. Placing the executable in `tools/fleet-docker`.
3. Building a local docker image using `docker build -t
fleetdm/fleet:v42.42.42 --platform=linux/amd64 .`.
4. Running the docker image (using `docker run`) and use Fleet on the
browser.
```
docker run -v $(pwd)/../osquery:/run -p 8412:8412 -e FLEET_MYSQL_ADDRESS=host.docker.internal:3306 -e FLEET_MYSQL_DATABASE=fleet -e FLEET_MYSQL_USERNAME=fleet -e FLEET_MYSQL_PASSWORD=insecure -e FLEET_REDIS_ADDRESS=host.docker.internal:6379 -e FLEET_SERVER_ADDRESS=0.0.0.0:8412 -e FLEET_SERVER_CERT=/run/fleet.crt -e FLEET_SERVER_KEY=/run/fleet.key -e FLEET_LOGGING_JSON='true' -e FLEET_VULNERABILITIES_CURRENT_INSTANCE_CHECKS="yes" -e FLEET_VULNERABILITIES_DATABASES_PATH=/vulndb -e FLEET_LOGGING_DEBUG='true' -it fleetdm/fleet:v42.42.42
```

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
2025-01-28 19:58:20 -03:00
Dante Catalfamo
8c338a1d77
Try splitting up integration tests (#25312)
Follow up to #25271 and #21774

Integration test failures will happen much faster of they occur, but now
the bottleneck is the `fleetctl` test suite.

It's trivial to continue splitting tests up now. We should look into
creating an action that checks that mock generation is up-to-date, run
it before all the tests, and then remove the mock generation step from
each test step. That would save about a minute and a half of runtime
from each test and help offset the cost of splitting the tests up.

![ci runtime
breakdown](https://github.com/user-attachments/assets/057b8ee1-782c-4e1f-9486-42c7d1169c81)
![ci runtime
max](https://github.com/user-attachments/assets/3a26995f-d9cb-490b-84d9-1a7fbb3cd6b3)

![image](https://github.com/user-attachments/assets/b4c888c8-867f-4bdd-9b69-0dc20d0d202a)
2025-01-10 12:52:13 -05:00
Dante Catalfamo
e6fb647d95
Run CI tests in parallel (#25271)
#21774

Improves run time by about 30%.

Things have been arranged in such a way that splitting modules out
further will be trivial in the future, such as breaking the different
integration test suited into their own units.


![image](https://github.com/user-attachments/assets/ead46e4c-6f14-406d-a29b-b25abc79c384)


![image](https://github.com/user-attachments/assets/3f7fd7f3-d7a8-4ff8-a184-646a72f1d015)
2025-01-09 13:38:24 -05:00
Scott Gress
bf7876af54
Add make commands to standardize running/debugging Go tests (#24606)
# Checklist for submitter

- [X] Manual QA for all new/changed functionality

# Details

This PR adds two new user-facing `make` targets:

* `run-go-tests`: run Go tests for one or more packages, optionally
filtering to specific tests
* `debug-go-tests`: debug (using Delve) Go tests for one or more
packages, optionally filtering to specific tests

Example usage:

```
# Run all tests in the mysql and gdmf packages
make run-go-tests PKG_TO_TEST="server/mdm/apple/gdmf server/datastore/mysql"
```

```
# Run all the TestMDMApple tests in the mysql package
make run-go-tests PKG_TO_TEST=server/datastore/mysql TESTS_TO_RUN="^TestMDMApple\$$" 
```

```
# Run only the TestMDMAppleProfileLabels test in the mysql package
make run-go-tests PKG_TO_TEST=server/datastore/mysql TESTS_TO_RUN="^TestMDMApple\$$/^TestMDMAppleProfileLabels\$$" 
```

# Notes

Two new "private" targets `.run-go-tests` and `.debug-go-tests` were
created as base commands for both `test-go` (used in CI) and the new
user-facing commands.
2024-12-11 11:06:56 -06:00
Lucas Manuel Rodriguez
c993e5a485
Add workflow to build fleetd_tables (#23873)
Workflow to build `fleetd_tables` in Github.
2024-11-15 19:05:07 -03:00
Victor Lyuboslavsky
d4525d8b94
Allow reusing docker container for building desktop-linux. (#23424) 2024-10-31 09:59:04 -05:00
Lucas Manuel Rodriguez
2e5bf75b6d
fleetd to start up when TUF signatures are expired (#23102)
#22740

Full QA is still a WIP but this is ready for review.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Orbit runs on macOS, Linux and Windows. Check if the orbit
feature/bugfix should only apply to one platform (`runtime.GOOS`).
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2024-10-28 20:40:19 -03:00
Victor Lyuboslavsky
f85b6f776f
Updating golangci-lint to 1.61.0 (#22973) 2024-10-18 12:38:26 -05:00
Ian Littman
92bb7ec666
Add policy ID and name to activity for automated software installs, list Fleet as author rather than installer uploader (#22747)
#22424, #22705

TODO: integration test updates

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
2024-10-09 18:15:56 -05:00
Ian Littman
e4df7abb67
Backend build for script automation (#22472)
#22115, #22116

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

No changes file, as FE changes file covers the entire feature

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
Co-authored-by: Tim Lee <timlee@fleetdm.com>
2024-10-03 20:03:40 -05:00
Ian Littman
518a4a1f98
Use previous package filename for activity if installer edit doesn't change the package (#22177)
Also adds a line in the makefile help for generate-doc, as it took me
way too long to find that command this time.

# Checklist for submitter

- [x] Manual QA for all new/changed functionality
2024-09-17 15:21:25 -05:00