For #25507
A bump to the latest version to the github `cache` action to 4.2.0. our
current version (v2) was deprecated. more info for the deprecation can
be found here https://github.com/actions/cache/discussions/1510
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
User stories that are yet to be prioritized are clogging up the "New
requests" column on the drafting board. Only new feature requests should
show up in this column
# Changes
- orbit >= 1.38.0, when configured to connect to
https://tuf.fleetctl.com (existing fleetd deployments) will now connect
to https://updates.fleetdm.com and start using the metadata in path
`/opt/orbit/updates-metadata.json`.
- orbit >= 1.38.0, when configured to connect to some custom TUF (not
Fleet's TUFs) will copy `/opt/orbit/tuf-metadata.json` to
`/opt/orbit/updates-metadata.json` (if it doesn't exist) and start using
the latter.
- fleetctl `4.63.0` will now generate artifacts using
https://updates.fleetdm.com by default (or a custom TUF if
`--update-url` is set) and generate two (same file) metadata files
`/opt/orbit/updates-metadata.json` and the legacy one to support
downgrades `/opt/orbit/tuf-metadata.json`.
- fleetctl `4.62.0` when configured to use custom TUF (not Fleet's TUF)
will generate just the legacy metadata file
`/opt/orbit/tuf-metadata.json`.
## User stories
See "User stories" in
https://github.com/fleetdm/confidential/issues/8488.
- [x] Update `update.defaultRootMetadata` and `update.DefaultURL` when
the new repository is ready.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Orbit runs on macOS, Linux and Windows. Check if the orbit
feature/bugfix should only apply to one platform (`runtime.GOOS`).
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [X] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
See failed workflow run
[here](https://github.com/fleetdm/fleet/actions/runs/12555703803)
- Fix the powershell script that was broken by `.yml` auto-format
- Exclude github workflow `.yml` files from prettier autoformating,
since they often contain non-yaml code as part of job definitions
- [ ] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
Also updates said constant via this script to include 5.15.0. Idea for
this is that including pre-releases as they're published ensures that by
the time the corresponding Fleet release ships we have a current list,
without having to cherry-pick these updates.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
updated a few sections and added iOS/iPadOS tests
# Checklist for submitter
---------
Co-authored-by: Janis Watts <184028114+jmwatts@users.noreply.github.com>
for #23825
This PR fixes the previous implementation for attesting
fleet/fleetctl/orbit binaries, and adds attestation to the fleet desktop
and osqueryd artifacts.
* correct permissions are added to all jobs
* tag removed from `subject-name` when attesting docker image
* using `artifacts.json` rather than the `artifacts` step output from
goreleaser to determine image digest
I'd like to add a separate job verifying the attestations, working on
that now but since all attestation steps are marked as
`continue-on-error` it can be a follow-on if we don't get it in with
this PR.
This PR adds a new workflow called "Stress Test Go Test" (aka the
RandoKiller) that allows running one or more tests repeatedly up to a
set number of times, or until a test fails. This is useful for:
* Trying to diagnose and debug a flaky test
* Verifying that a proposed fix for a flaky test actually works.
To use:
1. Create a branch whose name ends with "-randokiller"
2. Modify the .github/workflows/config/randokiller.json file to your
specifications (choosing the packages and tests to run, the mysql
matrix, and the number of runs to do)
3. Push up the branch
Since the stress test is intended to run a branch that you'll never
merge, you should feel free to add whatever logs to your tests or code
that will help diagnose failures.
I used this to diagnose and fix
https://github.com/fleetdm/fleet/pull/24697!
for #19106
This PR adds a Slack notification when the GitOps run fails in the
dogfood-gitops workflow. Whenever the actual GitOps action fails, it
should notify #help-dogfooding with a link to the failed action. Note
that this will alert on both merges to main and scheduled runs, which I
think we want. Also note that this is [currently failing on
main](https://github.com/fleetdm/fleet/actions/runs/12154006118) so this
alert will start going off daily until the issue is fixed 😶
### > Note: this will need a new Slack incoming webhook for sending
messages to #help-dogfooding, and a new
`SLACK_G_HELP_DOGFOODING_WEBHOOK_URL` repo secret with the webhook URL.
I tested this on a personal private repo just to make sure I got all the
syntax right:
<img width="422" alt="image"
src="https://github.com/user-attachments/assets/74d188eb-5c03-471b-a5db-9f578a56e2ab">
I beleive we don't need this step anymore, since `fleetctl gitops` will
replace it with real value and send to the server. This should be done
in #17309.
[#8489](https://github.com/fleetdm/confidential/issues/8489)
We had the timestamp check.
Robert added the root check recently.
Am now duplicating the check for `snapshot` and `targets` metadata
files.
PS: Please review with whitespace changes disabled.
#23905
- Update with upstream nanomdm changes up to
825f2979a2
- Removed PostgeSQL folder from our nanomdm
- Added nanomdm MySQL test job to our CI
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
From discussions with @jahzielv.
QAing ADE flows:
1. New version of fleetd is pushed to `edge`
2. QA folks can trigger this new workflow and download the generated
`fleetd-base.pkg` and `fleetd-base-manifest.plist`.
3. Host the downloaded files (in `foobar/`) in their ngroks URLs (using
e.g. `go tools ./tools/file-server 8085 foobar/`)
4. Use Fleet's `FLEET_DEV_DOWNLOAD_FLEETDM_URL` to point the Fleet
server to their ngrok URL.
Closes: https://github.com/fleetdm/confidential/issues/8351
Changes:
- Deleted the "Deploy app to bulk operations dashboard pipeline on
Heroku" workflow. This dashboard is now hosted in Render, and deploys
are triggered manually via the Render dashboard.
Closes: #22931
Changes:
- Updated the deploy workflows for the Fleet website and the
vulnerability dashboard to run on Ubuntu 22.04 to prevent issues we've
been seeing with the Heroku deploy action and the latest version of
Ubuntu.
#22206
This was discussed in the backend weekly.
Currently the test-packaging.yml is extremely unreliable (it has more
failures than successes), because of issues with Docker and colima on
Github macOS runners (we tried docker then colima but both have issues,
timeouts, etc.).
This only removes testing of MSI package generation from macOS. IMO this
is low risk as almost all Fleet devs generate MSI packages from their
macOS workstations.
This fix applies to cases (e.g.
00ec402f18) where order of files being
added is e.g.:
1. Migration A
2. Migration B
3. Test for migration A
This also reorders workflow steps so the ones that don't require setting
up Go + compiling happen first, so if we have a migration issue it gets
reported sooner.
# Checklist for submitter
- [x] Manual QA for all new/changed functionality
Also mention that we test with 8.4.2 in a few more places.
Note that while I'm editing release articles, this isn't retconning
minimum requirements; we mention in 4.55.0 release notes further down
that we expect 8.0.36.
- Add reminder to think about new activity items during the drafting
process
- For all checkboxes, we want to be intentional when we decided to make
"No changes" (instead of removing checkbox)
Updating the feature request template to note that the Fleet requestor
should provide a Gong snippet where a customer or prospect discussed a
feature when available
Done as part of oncall improvements.
`vars.GO_VERSION` can only be changed by admins and it's not public
(Fleet devs don't know the current value of the variable), this approach
uses the version specified in our `go.mod` file.
- "CLI changes" are for new fleetctl commands, options etc. fleetctl
wireframes go in Figma.
- "YAML changes" are for changes to Fleet's YAML used in GitOps
workflows. YAML wireframes are made as a draft PR to the YAML reference
[here](https://fleetdm.com/docs/configuration/yaml-files).
- `.github/workflows/push-osquery-perf-to-ecr.yml` has 0 workflow runs
(added but never used)
- `Dockerfile.osquery-perf` is only used by
`.github/workflows/push-osquery-perf-to-ecr.yml`.
Related to: #20296
Changes:
- Added `ee/bulk-operations-dashboard`, a Sails.js app that lets users
manage configuration profiles and scripts across multiple teams on a
Fleet instance.
- Added a Github workflow to deploy the app to Heroku
- Added a Github workflow to test changes to the bulk operations
dashboard.
#20571
## Summary of changes
We have a few moving parts in fleetctl land (`fleetdm/wix` is used to
build `msi`s and `fleetdm/bomutils` is used to build `pkg`s, and
`fleetdm/fleetctl` can be used to build packages using docker, no need
for fleetctl executable):
```mermaid
graph LR
fleetctl_exec[fleetctl<br>executable];
wix_image[fleetdm/wix<br>docker image];
bomutils_image[fleetdm/bomutils<br>docker image];
fleetctl_image[fleetdm/fleetctl<br>docker image];
fleetctl_exec -- uses --> wix_image;
fleetctl_image -- COPY dependencies<br>FROM --> wix_image;
fleetctl_exec -- uses --> bomutils_image;
fleetctl_image -- COPY dependencies<br>FROM --> bomutils_image;
```
So, we'll need to update the three images: `fleetdm/bomutils`,
`fleetdm/wix` & `fleetdm/fleetctl`.
- `tools/bomutils-docker/Dockerfile`, `tools/wix-docker/Dockerfile` and
`tools/fleetctl-docker/Dockerfile`: Updating the base image to fix the
CRITICAL vulnerabilities.
- Modified existing+unused
`.github/workflows/build-and-check-fleetctl-docker-and-deps.yml` to run
every day to check for CRITICAL vulnerabilities in `fleetdm/wix`,
`fleetdm/bomutils` and `fleetdm/fleetctl`.
- `.github/workflows/goreleaser-fleetctl-docker-deps.yaml`:
`fleetdm/bomutils` and `fleetdm/wix` were pushed manually a few years
ago (most likely by Zach), so I've added a new action to release them
when we have changes to release (like now). It will basically release
`fleetctl/bomutils` and `fleetdm/wix` when pushing a tag of the form
`fleetctl-docker-deps-*` (we'll need to protect such tag prefix).
- Changes in `.github/workflows/test-native-tooling-packaging.yml` to
build `fleetdm/bomutils` and `fleetdm/wix` for `fleetdm/fleetctl` to use
them instead of the ones in docker hub.
--
Build before upgrading `debian:stable-slim`:
https://github.com/fleetdm/fleet/actions/runs/10255391418/job/28372231837
Build after upgrading `debian:stable-slim`:
https://github.com/fleetdm/fleet/actions/runs/10255550034
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
This ensures `#help-engineering` is notified when a dogfood deploy is in
progress. It helps set people's expectations about what's going on while
the server is temporarily down.
#6085
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [x] Manual QA for all new/changed functionality
for #18297
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Manual QA for all new/changed functionality
#19182
Releasing new fleetd-base flow
- The flow has been QA'd
- The flow puts the generated files into new directories (`stable` and
`archive`), so risk is low
Oncall project to keep auto-generated documentation up-to-date.
- Auto-generated documentation from Golang source code.
- Auto-generated table JSON schema from yaml files in `schema/tables/`
(we currently remind the developer to run the sails.js commands in the
PR or ask Eric to do it).
Sample of the failure if the developer forgot to run `make generate-doc`
(similar to `make test-db-schema`):
Add note about manually requesting reviews on API design draft PRs,
because reviews aren't automatically requested until PRs are marked
ready for review.
cc @marko-lisica @noahtalerman
A few days ago, a new major version of goreleaser was published, which
is currently breaking our workflows:
```
⨯ command failed error=unknown flag: --rm-dist
```
This locks the version to a max satisfying semver under 1 until we have
time to update to the new major.
#18811
* Fixed bug where fleetd-chrome sent multiple read requests to Fleet
server at the same time.
* Improved console log output messages when Fleet server is down.
The `macos-latest` runner is using `macos-14` + ARM now, which was
causing the Docker install to fail.
I switched to `macos-13` since seems to be a cheap x86_64 alternative
and figured what was the problem with Colima so we don't have to deal
with Docker anymore.
Handbook changes for moving product discussion to
#g-mdm/#g-endpoint-ops: fleetdm/confidential#6056
- Replace #help-product design w/ #help-design or #g-mdm or #g-endpoint
ops
- Cut "Maintain current versions". It's documented in product rituals
The Wine developer does have an Apple Develeoper certificate but the
"Wine Stable" app bundle is not code-signed or notarized post-install &
disables Gatekeeper for the install. This adds a warning to the script
user about the app not being signed. post-install
---------
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
Changes:
- Updated the deploy-vulnerability-dashboard workflow to use the correct
variables for the Heroku steps.
- Added GitHub maintainers to `website/config/custom.js` for the GitHub
workflows related to the vulnerability dashboard.
Emojis are back on Dogfood team names. Need to rename the teams in UI
before merging. Otherwise, GitOps will simply create new teams.
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Closes: https://github.com/fleetdm/confidential/issues/4057
Changes:
- Added the contents of the fleet-vulnerability-dashboard repo to
ee/vulnerability-dashboard
- Added a github workflow to deploy the vulnerability dashboard on
Heroku
- Added a github workflow to test changes to the vulnerability-dashboard
- Updated the website's custom configuration to enable
auto-approvals/review requests to files in the
ee/vulnerability-dashboard folder
Adds a minimum supported node and yarn version to the project.
Currently if you are on an unsupported version of node or yarn, there is
no messaging telling you that is the issue. The build just fails, and
you are left to figure out it's because of your node version. With this
change, it will be much clearer why any of the node required commands
(e.g. make deps, make generate-dev, make lint-js, make test-js) are not
working, and it will tell you exactly which minimum version of node or
yarn you need.
**After the console error is clear about using an unsupported node
version**
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Manual QA for all new/changed functionality
- Permissions changes will either be a draft PR to manage access doc
page or explicitly mention that there's no change to the doc page
- Anyone on product team can assign API changes to engineering team
Moving mdm_profiles to it-and-security/lib/mdm_profiles so that they are
together with other gitops config files.
---------
Co-authored-by: Noah Talerman <noahtal@umich.edu>
for #16954
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Add "Product designer" section so that contributors know who to
contact with questions about UI, CLI, or API design
- Move entire "Context" section higher up so that it's easier to find
Reasons:
- Smaller artifacts on
https://github.com/fleetdm/fleet/actions/workflows/goreleaser-orbit.yaml
(used when releasing fleetd).
- Less error prone (human performing the release has to be careful to
not pick the macOS amd64 or arm64 version of orbit, and pick the
universal one)
- Moves a small step forward to #16131
- Clarify that the Figma link should take folks to the "ℹ️ Cover" page
- This way, everyone can see the status of the story: Work in progress,
Settled, Released
- This way, it's hard to accidentally link to the scratchpad file which
is not ready for dev
Should tackle #14026.
This will run a daily Github action and create a PR if there's a new
update in our TUF on `edge` or `stable`.
E.g. somebody releases 1.22.0 fleetd to `stable` on our TUF and the next
day this automation runs and will create a PR that updates the versions
in `orbit/TUF.md` (or they can run the workflow manually).
Am happy to amend the shape of `orbit/TUF.md` (or we can iterate later).
New flow for `fleetctl --package --type=msi` on macOS using arm64
processor (M1, M2, etc.)
- wine must be installed locally. See
./orbit/tools/build/install-wine-macos.sh and
https://wiki.winehq.org/MacOS for reference.
- --local-wix-dir can be used to point to a local Wix3 installation
(using this switch requires a current Fleet EE subscription)
#15463
PR for docs: https://github.com/fleetdm/fleet/pull/16459
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
#15881
This PR adds a script to test DB migrations with Percona XtraDB 5.7.25.
PS: To run this test before we merge this PR to `main` you will need to
change step 2 (`Make sure to be on latest main`), instead of `main` use
this branch `15881-test-migrations-with-percona`.
Add GitHub Actions for releasing fleetd-chrome beta and production. See
the included README updates for details.
This was tested with an `on: pull-request` trigger for the beta workflow
which is now removed for merging into the repo.
Closes: #14246
Changes:
- Added a new key to the rituals YAML configuration: `autoIssue.repo`.
This value should be a string that is the name of the GH repo that
issues for the ritual should be created in.
- Updated ritual validation in `build-static-content`.
- Added support for the "monthly" ritual frequency for rituals with an
`autoIssue` value.
- Updated the `create-issues-for-todays-rituals` script to create GitHub
issues for rituals.
---------
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
Co-authored-by: Sam Pfluger <108141731+Sampfluger88@users.noreply.github.com>
Related to: https://github.com/fleetdm/fleet/issues/15089
Changes:
- Replaced the expired osquery Slack invitation with a link to the Fleet
website's `/slack` redirect.
---------
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
This new `:incoming` label is used by engineers to filter down to _new_
bugs on their sprint board during each standup. They will remove the
label, indicating they have triaged the issue.
QA removes `:reproduce`, EM removes `:incoming`.
- Move "Scalability testing" to Engineering section. Engineering team
will have a better idea if the story needs load testing
---------
Co-authored-by: Luke Heath <luke@fleetdm.com>
#15563
- [X] Manual QA for all new/changed functionality
Tested by running the following:
If the changes haven't been merged to `main`:
```sh
fleetctl preview --preview-config 15563-move-external-dep-osquery-in-a-box-to-monorepo
fleetctl preview stop
fleetctl preview reset
```
If the changes were already merged to `main`:
```sh
fleetctl preview
fleetctl preview stop
fleetctl preview reset
```
Implementing a safety measure to prevent issues like #15910 in
production.
Setting the macOS version explicitly avoids unexpected changes in the
builder runtime, ensuring the Fleet Desktop executable remains
compatible.
As of this commit, 'macos-latest' refers to 'macos-12'. We're aligning
the worker to this version, although building on macOS 13.x (presently
in GitHub workers' beta) should also be viable.
Per discussion with @noahtalerman and @marko-lisica today: we're going
to aim to always add redirects in `/website/config/routes.js` for any
docs/external pages we link to in the Fleet UI & CLI, to reduce surface
areas of PRs when doc headings change or things are moved around...
---------
Co-authored-by: Eric <eashaw@sailsjs.com>
for https://github.com/fleetdm/fleet/issues/15584
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
Related to: #10209
Changes:
- Updated the go test in the Makefile to have the `--codepkg` flag.
- Added a newline to the `test-go` GH workflow to trigger a run for this
PR
> Note: I'm creating this as a draft PR to see the results of the "Test
Go" workflow
Update name to "Apply latest configuration profiles and macOS updates"
Because it used to say update MDM (workflow is for more than MDM), and
keeps it in line with workstation group.
Since a lot of bugs end up needing additional product design work, I
propose adding a (commented-out by default) section to this template to
standardize where to add design changes, once settled.
Reasoning: in estimation sessions, it can sometimes be hard to find this
information: sometimes it's in the comments, sometimes it's been added
to the description... either way, its not always obvious to spot. I
think it will help us move quicker if there's a consistent heading to
look for.
(Also, open to suggestions for other ways of wording that heading! This
is just the way I've been adding it to issue descriptions lately.)
