Release notes: https://tip.golang.org/doc/go1.24
> Go modules can now track executable dependencies using tool directives
in go.mod. This removes the need for the previous workaround of adding
tools as blank imports to a file conventionally named “tools.go”. The go
tool command can now run these tools in addition to tools shipped with
the Go distribution. For more information see [the
documentation](https://tip.golang.org/doc/modules/managing-dependencies#tools).
The new -tool flag for go get causes a tool directive to be added to the
current module for named packages in addition to adding require
directives.
I ran:
```
go get -tool github.com/fleetdm/fleet/v4/server/goose
go get -tool github.com/kevinburke/go-bindata
go get -tool github.com/quasilyte/go-ruleguard/dsl
go rm tools.go
go mod tidy
```
`make deps-go` was failing in CI because of the removal of `tools.go`
(my guess is that `go get .` was a nop because there was nothing in `.`
to download).
So, taking the chance of removing `deps-go` because `go` will download
packages during the build process. AFAICS there's no need to download
everything beforehand.
For #27267.
Below is what's shown immediately after selecting an EXE:
<img width="1254" alt="image"
src="https://github.com/user-attachments/assets/a28d8565-de88-448a-bdbc-92aefc34ad55"
/>
TODO:
* Tests
* GitOps requirements changes
* Disabling add button/adding errors when required scripts aren't
specified
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Luke Heath <luke@fleetdm.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: RachelElysia <rachel@fleetdm.com>
#22206
This was discussed in the backend weekly.
Currently the test-packaging.yml is extremely unreliable (it has more
failures than successes), because of issues with Docker and colima on
Github macOS runners (we tried docker then colima but both have issues,
timeouts, etc.).
This only removes testing of MSI package generation from macOS. IMO this
is low risk as almost all Fleet devs generate MSI packages from their
macOS workstations.
Done as part of oncall improvements.
`vars.GO_VERSION` can only be changed by admins and it's not public
(Fleet devs don't know the current value of the variable), this approach
uses the version specified in our `go.mod` file.
#20571
## Summary of changes
We have a few moving parts in fleetctl land (`fleetdm/wix` is used to
build `msi`s and `fleetdm/bomutils` is used to build `pkg`s, and
`fleetdm/fleetctl` can be used to build packages using docker, no need
for fleetctl executable):
```mermaid
graph LR
fleetctl_exec[fleetctl<br>executable];
wix_image[fleetdm/wix<br>docker image];
bomutils_image[fleetdm/bomutils<br>docker image];
fleetctl_image[fleetdm/fleetctl<br>docker image];
fleetctl_exec -- uses --> wix_image;
fleetctl_image -- COPY dependencies<br>FROM --> wix_image;
fleetctl_exec -- uses --> bomutils_image;
fleetctl_image -- COPY dependencies<br>FROM --> bomutils_image;
```
So, we'll need to update the three images: `fleetdm/bomutils`,
`fleetdm/wix` & `fleetdm/fleetctl`.
- `tools/bomutils-docker/Dockerfile`, `tools/wix-docker/Dockerfile` and
`tools/fleetctl-docker/Dockerfile`: Updating the base image to fix the
CRITICAL vulnerabilities.
- Modified existing+unused
`.github/workflows/build-and-check-fleetctl-docker-and-deps.yml` to run
every day to check for CRITICAL vulnerabilities in `fleetdm/wix`,
`fleetdm/bomutils` and `fleetdm/fleetctl`.
- `.github/workflows/goreleaser-fleetctl-docker-deps.yaml`:
`fleetdm/bomutils` and `fleetdm/wix` were pushed manually a few years
ago (most likely by Zach), so I've added a new action to release them
when we have changes to release (like now). It will basically release
`fleetctl/bomutils` and `fleetdm/wix` when pushing a tag of the form
`fleetctl-docker-deps-*` (we'll need to protect such tag prefix).
- Changes in `.github/workflows/test-native-tooling-packaging.yml` to
build `fleetdm/bomutils` and `fleetdm/wix` for `fleetdm/fleetctl` to use
them instead of the ones in docker hub.
--
Build before upgrading `debian:stable-slim`:
https://github.com/fleetdm/fleet/actions/runs/10255391418/job/28372231837
Build after upgrading `debian:stable-slim`:
https://github.com/fleetdm/fleet/actions/runs/10255550034
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
The `macos-latest` runner is using `macos-14` + ARM now, which was
causing the Docker install to fail.
I switched to `macos-13` since seems to be a cheap x86_64 alternative
and figured what was the problem with Colima so we don't have to deal
with Docker anymore.
The Wine developer does have an Apple Develeoper certificate but the
"Wine Stable" app bundle is not code-signed or notarized post-install &
disables Gatekeeper for the install. This adds a warning to the script
user about the app not being signed. post-install
---------
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
New flow for `fleetctl --package --type=msi` on macOS using arm64
processor (M1, M2, etc.)
- wine must be installed locally. See
./orbit/tools/build/install-wine-macos.sh and
https://wiki.winehq.org/MacOS for reference.
- --local-wix-dir can be used to point to a local Wix3 installation
(using this switch requires a current Fleet EE subscription)
#15463
PR for docs: https://github.com/fleetdm/fleet/pull/16459
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
For #13715, this:
- Upgrades the Go version to `1.21.1`, infrastructure changes are
addressed separately at https://github.com/fleetdm/fleet/pull/13878
- Upgrades the linter version, as the current version doesn't work well
after the Go upgrade
- Fixes new linting errors (we now get errors for memory aliasing in
loops! 🎉 )
After this is merged people will need to:
1. Update their Go version. I use `gvm` and I did it like:
```
$ gvm install go1.21.1
$ gvm use go1.21.1 --default
```
2. Update the local version of `golangci-lint`:
```
$ go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.54.2
```
3. (optional) depending on your setup, you might need to re-install some
packages, for example:
```
# goimports to automatically import libraries
$ go install golang.org/x/tools/cmd/goimports@latest
# gopls for the language server
$ go install golang.org/x/tools/gopls@latest
# etc...
```
At the moment, in Github Actions, when a job has `uses:
actions/setup-go` it uses a specific commit from that repo.
In that commit, it used `set-output` somewhere, which is now deprecated
and will be disabled within the next month or so.
See here for more information:
https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
This PR changes every instance where `actions/setup-go@...` was used and
replaces it with release `v2.1.3`. [From the release
notes](https://github.com/actions/setup-go/releases/tag/v2.1.3):
> Updated communication with runner to use environment files rather then
workflow commands
Which is what the above Github blog recommends doing.
---
Addationally, the latest version of this Github Action is
[`v4.0.0`](https://github.com/actions/setup-go/releases/tag/v4.0.0),
which you may want to update to in the future.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
* Update go to 1.19.4
* Comment out failing package test
* Comment out ALL the packaging tests for windows for the moment
* Update go to 1.19.4
* Comment out failing package test
* Comment out ALL the packaging tests for windows for the moment
* Update changelog
* Bump versions
* Update changelog to reflect this being a security release
* Bump go to 1.19.1
* Bump remaining go-version to the 1.19.1
* Add extra paths for test-go
* Oops, putting the right path in the right place
* gofmt file
* gofmt ALL THE THINGS
* Moar changes
* Actually, go.mod doesn't like minor versions
The macOS runners installing Docker are having problems initializing the new Docker version (4.11.0) which effectively blocks PRs with Go code.
This locks the Docker version we install to 4.10.0, which works until we figure out a solution or a new Docker version goes out.
Related to #6364 and #6363, this:
- Adds a new Docker image, `fleetdm/fleetctl` equipped with all necessary dependencies to build Fleet-osquery binaries for all platforms
- Modifies the package generation logic to special case this scenario via an environment variable `FLEETCTL_NATIVE_TOOLING`
- Adds a new GitHub workflow to test this
There are more details in the README, but part of the special-casing logic is in place to output the binaries to a folder named `build` when they are run with `FLEETCTL_NATIVE_TOOLING`, this is so we can persist the binary generated by the docker container via a bind mount:
```bash
docker run -v "$(pwd):/build" fleetdm/fleetctl package --type=msi
```
To test this changeset, I have generated packages for all platforms, both via the new Docker image and via the classic `fleetctl package`.
* Do not use golangci action for better reproducibility
* Add fix to trigger build
* Fix all reported issues
* fix more lint errors
* Add missing import
* Remove unused method
* Remove change not necessary
* Adding permissions to docs.yml and integration.yml
* Update codeql-analysis.yml
Adding top level read permissions to codeql workflow
* Update codeql-analysis.yml
Adding manual dispatch to codeql - to be able to test it easier
* Update deploy-fleet-website.yml
Adding top level read permission + write in the job so it can push the website
* Update test-website.yml
test-website should only need read permissions on content.
* Update fleet-and-orbit.yml
Testing Fleet and Orbit should be fine with top level read access
* Update fleetctl-preview.yml
fleetctl-preview should be fine with just read access at top level
* Update push-osquery-perf-to-ecr.yml
ECR is out of github so read permissions should be enough
* Update semgrep-analysis.yml
semgrep should only need read
* Update test-packaging.yml
Should only need read permission - setting on top
* Update test.yml
Should not need any write access - setting to READ on top.
* Update deploy-fleet-website.yml
Removing git write permission - since this pushes to Heroku not GitHub
* Tweaked as per Zach's comments
Removed some useless restrictions (contents none on a public repo for example)
* Removed meaningless permissions
contents: none - this does not have any security advantage on a public repo
- Fix Windows MSI generation by changing permissions (#2655).
- Refactor temp directory initialization.
- Use root user for Wine in WiX Docker container.
- Support .pkg packaging on Linux without dependencies (besides Docker)
