This PR adds a new workflow called "Stress Test Go Test" (aka the
RandoKiller) that allows running one or more tests repeatedly up to a
set number of times, or until a test fails. This is useful for:
* Trying to diagnose and debug a flaky test
* Verifying that a proposed fix for a flaky test actually works.
To use:
1. Create a branch whose name ends with "-randokiller"
2. Modify the .github/workflows/config/randokiller.json file to your
specifications (choosing the packages and tests to run, the mysql
matrix, and the number of runs to do)
3. Push up the branch
Since the stress test is intended to run a branch that you'll never
merge, you should feel free to add whatever logs to your tests or code
that will help diagnose failures.
I used this to diagnose and fix
https://github.com/fleetdm/fleet/pull/24697!
for #19106
This PR adds a Slack notification when the GitOps run fails in the
dogfood-gitops workflow. Whenever the actual GitOps action fails, it
should notify #help-dogfooding with a link to the failed action. Note
that this will alert on both merges to main and scheduled runs, which I
think we want. Also note that this is [currently failing on
main](https://github.com/fleetdm/fleet/actions/runs/12154006118) so this
alert will start going off daily until the issue is fixed 😶
### > Note: this will need a new Slack incoming webhook for sending
messages to #help-dogfooding, and a new
`SLACK_G_HELP_DOGFOODING_WEBHOOK_URL` repo secret with the webhook URL.
I tested this on a personal private repo just to make sure I got all the
syntax right:
<img width="422" alt="image"
src="https://github.com/user-attachments/assets/74d188eb-5c03-471b-a5db-9f578a56e2ab">
I beleive we don't need this step anymore, since `fleetctl gitops` will
replace it with real value and send to the server. This should be done
in #17309.
[#8489](https://github.com/fleetdm/confidential/issues/8489)
We had the timestamp check.
Robert added the root check recently.
Am now duplicating the check for `snapshot` and `targets` metadata
files.
PS: Please review with whitespace changes disabled.
#23905
- Update with upstream nanomdm changes up to
825f2979a2
- Removed PostgeSQL folder from our nanomdm
- Added nanomdm MySQL test job to our CI
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
From discussions with @jahzielv.
QAing ADE flows:
1. New version of fleetd is pushed to `edge`
2. QA folks can trigger this new workflow and download the generated
`fleetd-base.pkg` and `fleetd-base-manifest.plist`.
3. Host the downloaded files (in `foobar/`) in their ngroks URLs (using
e.g. `go tools ./tools/file-server 8085 foobar/`)
4. Use Fleet's `FLEET_DEV_DOWNLOAD_FLEETDM_URL` to point the Fleet
server to their ngrok URL.
Closes: https://github.com/fleetdm/confidential/issues/8351
Changes:
- Deleted the "Deploy app to bulk operations dashboard pipeline on
Heroku" workflow. This dashboard is now hosted in Render, and deploys
are triggered manually via the Render dashboard.
Closes: #22931
Changes:
- Updated the deploy workflows for the Fleet website and the
vulnerability dashboard to run on Ubuntu 22.04 to prevent issues we've
been seeing with the Heroku deploy action and the latest version of
Ubuntu.
#22206
This was discussed in the backend weekly.
Currently the test-packaging.yml is extremely unreliable (it has more
failures than successes), because of issues with Docker and colima on
Github macOS runners (we tried docker then colima but both have issues,
timeouts, etc.).
This only removes testing of MSI package generation from macOS. IMO this
is low risk as almost all Fleet devs generate MSI packages from their
macOS workstations.
This fix applies to cases (e.g.
00ec402f18) where order of files being
added is e.g.:
1. Migration A
2. Migration B
3. Test for migration A
This also reorders workflow steps so the ones that don't require setting
up Go + compiling happen first, so if we have a migration issue it gets
reported sooner.
# Checklist for submitter
- [x] Manual QA for all new/changed functionality
Also mention that we test with 8.4.2 in a few more places.
Note that while I'm editing release articles, this isn't retconning
minimum requirements; we mention in 4.55.0 release notes further down
that we expect 8.0.36.
Done as part of oncall improvements.
`vars.GO_VERSION` can only be changed by admins and it's not public
(Fleet devs don't know the current value of the variable), this approach
uses the version specified in our `go.mod` file.
- `.github/workflows/push-osquery-perf-to-ecr.yml` has 0 workflow runs
(added but never used)
- `Dockerfile.osquery-perf` is only used by
`.github/workflows/push-osquery-perf-to-ecr.yml`.
Related to: #20296
Changes:
- Added `ee/bulk-operations-dashboard`, a Sails.js app that lets users
manage configuration profiles and scripts across multiple teams on a
Fleet instance.
- Added a Github workflow to deploy the app to Heroku
- Added a Github workflow to test changes to the bulk operations
dashboard.
#20571
## Summary of changes
We have a few moving parts in fleetctl land (`fleetdm/wix` is used to
build `msi`s and `fleetdm/bomutils` is used to build `pkg`s, and
`fleetdm/fleetctl` can be used to build packages using docker, no need
for fleetctl executable):
```mermaid
graph LR
fleetctl_exec[fleetctl<br>executable];
wix_image[fleetdm/wix<br>docker image];
bomutils_image[fleetdm/bomutils<br>docker image];
fleetctl_image[fleetdm/fleetctl<br>docker image];
fleetctl_exec -- uses --> wix_image;
fleetctl_image -- COPY dependencies<br>FROM --> wix_image;
fleetctl_exec -- uses --> bomutils_image;
fleetctl_image -- COPY dependencies<br>FROM --> bomutils_image;
```
So, we'll need to update the three images: `fleetdm/bomutils`,
`fleetdm/wix` & `fleetdm/fleetctl`.
- `tools/bomutils-docker/Dockerfile`, `tools/wix-docker/Dockerfile` and
`tools/fleetctl-docker/Dockerfile`: Updating the base image to fix the
CRITICAL vulnerabilities.
- Modified existing+unused
`.github/workflows/build-and-check-fleetctl-docker-and-deps.yml` to run
every day to check for CRITICAL vulnerabilities in `fleetdm/wix`,
`fleetdm/bomutils` and `fleetdm/fleetctl`.
- `.github/workflows/goreleaser-fleetctl-docker-deps.yaml`:
`fleetdm/bomutils` and `fleetdm/wix` were pushed manually a few years
ago (most likely by Zach), so I've added a new action to release them
when we have changes to release (like now). It will basically release
`fleetctl/bomutils` and `fleetdm/wix` when pushing a tag of the form
`fleetctl-docker-deps-*` (we'll need to protect such tag prefix).
- Changes in `.github/workflows/test-native-tooling-packaging.yml` to
build `fleetdm/bomutils` and `fleetdm/wix` for `fleetdm/fleetctl` to use
them instead of the ones in docker hub.
--
Build before upgrading `debian:stable-slim`:
https://github.com/fleetdm/fleet/actions/runs/10255391418/job/28372231837
Build after upgrading `debian:stable-slim`:
https://github.com/fleetdm/fleet/actions/runs/10255550034
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
This ensures `#help-engineering` is notified when a dogfood deploy is in
progress. It helps set people's expectations about what's going on while
the server is temporarily down.
#6085
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [x] Manual QA for all new/changed functionality
for #18297
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Manual QA for all new/changed functionality
#19182
Releasing new fleetd-base flow
- The flow has been QA'd
- The flow puts the generated files into new directories (`stable` and
`archive`), so risk is low
Oncall project to keep auto-generated documentation up-to-date.
- Auto-generated documentation from Golang source code.
- Auto-generated table JSON schema from yaml files in `schema/tables/`
(we currently remind the developer to run the sails.js commands in the
PR or ask Eric to do it).
Sample of the failure if the developer forgot to run `make generate-doc`
(similar to `make test-db-schema`):
A few days ago, a new major version of goreleaser was published, which
is currently breaking our workflows:
```
⨯ command failed error=unknown flag: --rm-dist
```
This locks the version to a max satisfying semver under 1 until we have
time to update to the new major.
#18811
* Fixed bug where fleetd-chrome sent multiple read requests to Fleet
server at the same time.
* Improved console log output messages when Fleet server is down.
The `macos-latest` runner is using `macos-14` + ARM now, which was
causing the Docker install to fail.
I switched to `macos-13` since seems to be a cheap x86_64 alternative
and figured what was the problem with Colima so we don't have to deal
with Docker anymore.
The Wine developer does have an Apple Develeoper certificate but the
"Wine Stable" app bundle is not code-signed or notarized post-install &
disables Gatekeeper for the install. This adds a warning to the script
user about the app not being signed. post-install
---------
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
Changes:
- Updated the deploy-vulnerability-dashboard workflow to use the correct
variables for the Heroku steps.
- Added GitHub maintainers to `website/config/custom.js` for the GitHub
workflows related to the vulnerability dashboard.
Emojis are back on Dogfood team names. Need to rename the teams in UI
before merging. Otherwise, GitOps will simply create new teams.
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Closes: https://github.com/fleetdm/confidential/issues/4057
Changes:
- Added the contents of the fleet-vulnerability-dashboard repo to
ee/vulnerability-dashboard
- Added a github workflow to deploy the vulnerability dashboard on
Heroku
- Added a github workflow to test changes to the vulnerability-dashboard
- Updated the website's custom configuration to enable
auto-approvals/review requests to files in the
ee/vulnerability-dashboard folder
Adds a minimum supported node and yarn version to the project.
Currently if you are on an unsupported version of node or yarn, there is
no messaging telling you that is the issue. The build just fails, and
you are left to figure out it's because of your node version. With this
change, it will be much clearer why any of the node required commands
(e.g. make deps, make generate-dev, make lint-js, make test-js) are not
working, and it will tell you exactly which minimum version of node or
yarn you need.
**After the console error is clear about using an unsupported node
version**
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Manual QA for all new/changed functionality
Moving mdm_profiles to it-and-security/lib/mdm_profiles so that they are
together with other gitops config files.
---------
Co-authored-by: Noah Talerman <noahtal@umich.edu>
for #16954
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Reasons:
- Smaller artifacts on
https://github.com/fleetdm/fleet/actions/workflows/goreleaser-orbit.yaml
(used when releasing fleetd).
- Less error prone (human performing the release has to be careful to
not pick the macOS amd64 or arm64 version of orbit, and pick the
universal one)
- Moves a small step forward to #16131
Should tackle #14026.
This will run a daily Github action and create a PR if there's a new
update in our TUF on `edge` or `stable`.
E.g. somebody releases 1.22.0 fleetd to `stable` on our TUF and the next
day this automation runs and will create a PR that updates the versions
in `orbit/TUF.md` (or they can run the workflow manually).
Am happy to amend the shape of `orbit/TUF.md` (or we can iterate later).
New flow for `fleetctl --package --type=msi` on macOS using arm64
processor (M1, M2, etc.)
- wine must be installed locally. See
./orbit/tools/build/install-wine-macos.sh and
https://wiki.winehq.org/MacOS for reference.
- --local-wix-dir can be used to point to a local Wix3 installation
(using this switch requires a current Fleet EE subscription)
#15463
PR for docs: https://github.com/fleetdm/fleet/pull/16459
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
#15881
This PR adds a script to test DB migrations with Percona XtraDB 5.7.25.
PS: To run this test before we merge this PR to `main` you will need to
change step 2 (`Make sure to be on latest main`), instead of `main` use
this branch `15881-test-migrations-with-percona`.
Add GitHub Actions for releasing fleetd-chrome beta and production. See
the included README updates for details.
This was tested with an `on: pull-request` trigger for the beta workflow
which is now removed for merging into the repo.
Closes: #14246
Changes:
- Added a new key to the rituals YAML configuration: `autoIssue.repo`.
This value should be a string that is the name of the GH repo that
issues for the ritual should be created in.
- Updated ritual validation in `build-static-content`.
- Added support for the "monthly" ritual frequency for rituals with an
`autoIssue` value.
- Updated the `create-issues-for-todays-rituals` script to create GitHub
issues for rituals.
---------
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
Co-authored-by: Sam Pfluger <108141731+Sampfluger88@users.noreply.github.com>
#15563
- [X] Manual QA for all new/changed functionality
Tested by running the following:
If the changes haven't been merged to `main`:
```sh
fleetctl preview --preview-config 15563-move-external-dep-osquery-in-a-box-to-monorepo
fleetctl preview stop
fleetctl preview reset
```
If the changes were already merged to `main`:
```sh
fleetctl preview
fleetctl preview stop
fleetctl preview reset
```
Implementing a safety measure to prevent issues like #15910 in
production.
Setting the macOS version explicitly avoids unexpected changes in the
builder runtime, ensuring the Fleet Desktop executable remains
compatible.
As of this commit, 'macos-latest' refers to 'macos-12'. We're aligning
the worker to this version, although building on macOS 13.x (presently
in GitHub workers' beta) should also be viable.
for https://github.com/fleetdm/fleet/issues/15584
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
Related to: #10209
Changes:
- Updated the go test in the Makefile to have the `--codepkg` flag.
- Added a newline to the `test-go` GH workflow to trigger a run for this
PR
> Note: I'm creating this as a draft PR to see the results of the "Test
Go" workflow
Update name to "Apply latest configuration profiles and macOS updates"
Because it used to say update MDM (workflow is for more than MDM), and
keeps it in line with workstation group.
These items fix the github action for use with the updates to the
monitoring module.
Additionally there were some changes needed to the monitoring module to
make it behave inside the GH action.
Once this is approved/merged, the new tag for them monitoring module
will be created as `tf-mod-addon-monitoring-v1.1.1`
Clsoes: https://github.com/fleetdm/fleet/issues/14162
Changes:
- Added two steps to the `deploy-fleet-website` workflow to prevent
errors when pushing to the Heroku git repo:
1. The first step runs a command to install the `heroku-repo` plugin in
the Heroku CLI.
2. The second step runs a command to reset the Heroku git repo for the
Fleet website. (This has no impact on the live Heroku app)
The
[goreleaser-orbit.yaml](https://github.com/fleetdm/fleet/actions/workflows/goreleaser-orbit.yaml)
workflow tends to timeout up to 9-10 times before successfully building
a macOS binary.
We have been using this workflow as a back-up, but it requires doing the
version bump and manually triggering the workflow, which can be error
prone.
This change follows the `workflows/generate-desktop-targets.yml` to
trigger the workflow when the workflow file itself is modified.
I need this change to load test #13926.
Due to recent changes we are not triggering a build on every
branch/commit pushed.
For load testing we need a way to trigger a build manually.
For #13715, this:
- Upgrades the Go version to `1.21.1`, infrastructure changes are
addressed separately at https://github.com/fleetdm/fleet/pull/13878
- Upgrades the linter version, as the current version doesn't work well
after the Go upgrade
- Fixes new linting errors (we now get errors for memory aliasing in
loops! 🎉 )
After this is merged people will need to:
1. Update their Go version. I use `gvm` and I did it like:
```
$ gvm install go1.21.1
$ gvm use go1.21.1 --default
```
2. Update the local version of `golangci-lint`:
```
$ go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.54.2
```
3. (optional) depending on your setup, you might need to re-install some
packages, for example:
```
# goimports to automatically import libraries
$ go install golang.org/x/tools/cmd/goimports@latest
# gopls for the language server
$ go install golang.org/x/tools/gopls@latest
# etc...
```
#13547
This is an attempt to stabilize this workflow that has been broken for
4-6 months.
# Issue and proposed solution
Github runner VMs re-use UUIDs, which is not supported by Orbit (this
causes a host to be enrolled as two hosts in Fleet), thus, until that is
fixed in https://github.com/fleetdm/fleet/issues/8021 I propose we
stabilize this workflow by testing all `stable` channels only (which is
better than having the build broken all the time IMO).
Once https://github.com/fleetdm/fleet/issues/8021 is fixed we can re-add
the edge channels.
After @rfairburn made the DNS change the clouldflared tunnel started
working again (after months of being broken).
#13547
Run: https://github.com/fleetdm/fleet/actions/runs/6025182774
This PR adds some fixes to the two workflows that make use of
cloudflared.
There are still some issues to fix but these are some changes needed to
continue/help troubleshooting.
#13182
[This PR](https://github.com/fleetdm/osquery-in-a-box/pull/18) in the
osquery-in-a-box repository recently added a new host to the simulated
host list which broke the CI job in the fleetdm/fleet repository.
PR run with this branch:
https://github.com/fleetdm/fleet/actions/runs/5866786432
PS: One of the reasons we had this osquery-in-a-box repository outside
the monorepo was to not break customers using `fleetctl preview`. But
now that we have Fleet Sandbox and we don't encourage users to use
`fleetctl preview`:
1. Does it make sense to have the separate repository?
2. Does it make sense to continue supporting this workflow in CI?
Context: The "Deploy Fleet website" workflow is currently failing
because the `build-storybook` step requires Node v16.
<img width="1013" alt="image"
src="https://github.com/fleetdm/fleet/assets/7445991/7681e11e-a94f-4a0b-8cd8-baa1ef5a37d8">
Changes:
- Changed the `deploy-fleet-website` and `test-website` workflows to use
Node 16.
- Updated the version of `actions/setup-node` to v3 to use node 16.
- added the `--legacy-peer-deps` flag to the `npm install` in the
build-storybook step
- Added a step to build the storybook to the `test-website` workflow.
- Updated the `test-website` workflow to run when the workflow file is
changed.
