#15557
Following the precedent that Lucas used for other similar PRs, the best
way to review is probably by commits.
* The first one simply copies over the files from the fork to the
monorepo
* Second one adjusts all import paths
* Third one tidies up the `go.mod` files
* Last one fixes the linter issues in the nanomdm package
# Checklist for submitter
- ~~Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~~ (not a user-visible change)
- [x] Manual QA for all new/changed functionality (ran test suite,
re-generated mocks)
I also verified that our Go test suite did run the newly moved `nanomdm`
package steps:
```
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/cryptoutil 0.003s coverage: 0.0% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/mdm 0.005s coverage: 46.2% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/service/certauth 1.320s coverage: 20.7% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/storage/file 0.007s coverage: 24.1% of statements in github.com/fleetdm/fleet/v4/...
```
> 📜 Related issue: #15635
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Tests were added in the scep repo:
https://github.com/fleetdm/scep/pull/1
#15556
We will need to pay attention when releasing fleet (the github actions
were modified to use the local file now).
Should be reviewed by commits (first commit is the actual adding of the
`version.go` file)
- [X] Manual QA for all new/changed functionality
Manually tested the following:
- `Settings -> My account` on the UI and checked the `/version` endpoint
response. (Or also visiting https://localhost:8080/version on a
browser).
- Ran `make fleetctl fleet`, `./build/fleetctl --version` and
`./build/fleet version`.
#15555
Probably the best way to review this is commit by commit:
- First commit does the actual moving.
- Second commit fixes golangci-lint issues (in the least effort way to
avoid refactoring or rearrangement of some of the code).
- Third commit moves a printf to before the migration step is executed.
In the past some customers hitting migration issues (like migration
steps hanging or taking long to execute) and wanted to know which one
was it. The only way to know was to look at the repository and looking
for the next migration after the last one logged.
Checks:
- [X] Manual QA for all new/changed functionality
Manual tests:
- `make fleet && make db-reset`.
- Adding a new migration via `make migration name=Foobar` and then
running `./build/fleet prepare db`.
- Enrolling a new device to Fleet (smoke test).
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from
0.14.0 to 0.17.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="9d2ee975ef"><code>9d2ee97</code></a>
ssh: implement strict KEX protocol changes</li>
<li><a
href="4e5a26183e"><code>4e5a261</code></a>
ssh: close net.Conn on all NewServerConn errors</li>
<li><a
href="152cdb1503"><code>152cdb1</code></a>
x509roots/fallback: update bundle</li>
<li><a
href="fdfe1f8531"><code>fdfe1f8</code></a>
ssh: defer channel window adjustment</li>
<li><a
href="b8ffc16e10"><code>b8ffc16</code></a>
blake2b: drop Go 1.6, Go 1.8 compatibility</li>
<li><a
href="7e6fbd82c8"><code>7e6fbd8</code></a>
ssh: wrap errors from client handshake</li>
<li><a
href="bda2f3f5cf"><code>bda2f3f</code></a>
argon2: avoid clobbering BP</li>
<li><a
href="325b735346"><code>325b735</code></a>
ssh/test: skip TestSSHCLIAuth on Windows</li>
<li><a
href="1eadac50a5"><code>1eadac5</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="b2d7c26edb"><code>b2d7c26</code></a>
ssh: add (*Client).DialContext method</li>
<li>Additional commits viewable in <a
href="https://github.com/golang/crypto/compare/v0.14.0...v0.17.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts page](https://github.com/fleetdm/fleet/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#14888
@getvictor This is ready for review, but keeping as draft as there are
probably many tests that need amending.
I used the new version of the `./tools/nvd/nvdvuln/nvdvuln.go` to
compare the current vulnerabilities found in our dogfood environment
with the vulnerabilities found by the code in this PR and both results
match:
```
go run -race -tags fts5 ./tools/nvd/nvdvuln/nvdvuln.go --debug --db_dir ./local --software_from_url <dogfood URL> --software_from_api_token <API_TOKEN> --sync 2>&1 | tee out.txt
[...]
CVEs found and expected matched!
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: Victor Lyuboslavsky <victor@fleetdm.com>
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
Loom explaining changes (hit 5 min limit):
https://www.loom.com/share/e59b63bf638e4d9cad7984ef589b878d?sid=111fff75-115a-4a44-ae4f-6f25fede0d51#14887
- [x] Need to merge fleetdm/nvd PR
https://github.com/fleetdm/nvd/pull/25 before this one.
# Checklist for submitter
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Manually tested (with corresponding fleetdm/fleet changes) in my
personal fork: https://github.com/getvictor/nvd/releases
# QA Plan (must be done before merging this PR, and after merging the
nvd PR)
- [ ] Fork https://github.com/fleetdm/nvd and point `generate.yml` to
this branch.
[example](9d8e54930b/.github/workflows/generate.yml (L26))
- [ ] Add NVD_API_KEY to nvd secrets, and run the the nvd generate
GitHub action. Get key:
https://nvd.nist.gov/developers/request-an-api-key
- [ ] Compare the generated `cpe-###.sqlite.gz` to the previous one. One
way is to open it up with sqlite3 and `select * from cpe_2 order by
cpe23;` and dump results to a CSV file. Known differences are:
- New file has ~2,500 more records
- Backslashes are handled differently for `Backpack\CRUD` and `Philips
In.Sight B120\37` products -- not a new issue since we do not support
those products right now
- `cpe:2.3🅰️moodle:moodle:4.2.0:*:*:*:*:*:*:*` -- this appears OK.
Also, it is a PHP plugin, and we don't support these currently.
- [ ] Record the existing vulnerabilities of current hosts.
- [ ] Stop any running fleet server. Delete `/tmp/vulndbs/cpe.sqlite`.
Can also delete other files there, or not delete this file -- it should
be overwritten by the new file. Also delete all rows in software_cpe and
software_cve DB tables. (Or can just spin up a fresh fleet server with
fresh DB, and re-enroll hosts (after setting the new env variable
below))
- [ ] Find the path to the generated `cpe-###.sqlite.gz` file
- [ ] Set `FLEET_VULNERABILITIES_CPE_DATABASE_URL` environment variable
to the above path, and start fleet server.
- [ ] After server's vulnerabilities cron job runs, the new
vulnerabilities should match the previous vulnerabilities
#15022
The issue in the package is being fixed here
https://github.com/osquery/osquery-go/pull/117
But to not block on that we will downgrade the osquery-go version we
use.
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [ ] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
#14543
Python's certifi package is ingested with the following version string:
`2023.7.22`.
The NVD dataset uses the following versioning: `2023.07.22`.
This PR updates the nvdtools package.
This is the fix in nvdtools that fixes this exact false positive:
c0d18738cd
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
For #13715, this:
- Upgrades the Go version to `1.21.1`, infrastructure changes are
addressed separately at https://github.com/fleetdm/fleet/pull/13878
- Upgrades the linter version, as the current version doesn't work well
after the Go upgrade
- Fixes new linting errors (we now get errors for memory aliasing in
loops! 🎉 )
After this is merged people will need to:
1. Update their Go version. I use `gvm` and I did it like:
```
$ gvm install go1.21.1
$ gvm use go1.21.1 --default
```
2. Update the local version of `golangci-lint`:
```
$ go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.54.2
```
3. (optional) depending on your setup, you might need to re-install some
packages, for example:
```
# goimports to automatically import libraries
$ go install golang.org/x/tools/cmd/goimports@latest
# gopls for the language server
$ go install golang.org/x/tools/gopls@latest
# etc...
```
changelog for the version bump can be found here:
https://github.com/macadmins/osquery-extension/releases/tag/v0.0.15
related to #13158 as this fixes a bug in the extension causing `profiles
show --type enrollment` to be run almost every time we queried something
from the `mdm` table.
I couldn't find any new tables. Other than that, some dependencies were
updated on their repo as well, most notably `osquery/osquery-go`
Issue #12261
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go)
from 1.43.16 to 1.44.288.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's
releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.288 (2023-06-22)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/chime-sdk-identity</code>: Updates service API and
documentation</li>
<li><code>service/chime-sdk-messaging</code>: Updates service API and
documentation</li>
<li><code>service/kendra</code>: Updates service API and documentation
<ul>
<li>Introducing Amazon Kendra Retrieve API that can be used to retrieve
relevant passages or text excerpts given an input query.</li>
</ul>
</li>
<li><code>service/states</code>: Updates service API and documentation
<ul>
<li>Adds support for Versions and Aliases. Adds 8 operations:
PublishStateMachineVersion, DeleteStateMachineVersion,
ListStateMachineVersions, CreateStateMachineAlias,
DescribeStateMachineAlias, UpdateStateMachineAlias,
DeleteStateMachineAlias, ListStateMachineAliases</li>
</ul>
</li>
</ul>
<h1>Release v1.44.287 (2023-06-21)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/dynamodb</code>: Updates service API, documentation,
waiters, paginators, and examples
<ul>
<li>Documentation updates for DynamoDB</li>
</ul>
</li>
<li><code>service/elasticmapreduce</code>: Updates service API,
documentation, and paginators
<ul>
<li>This release introduces a new Amazon EMR EPI called
ListSupportedInstanceTypes that returns a list of all instance types
supported by a given EMR release.</li>
</ul>
</li>
<li><code>service/inspector2</code>: Updates service API and
documentation</li>
<li><code>service/mediaconvert</code>: Updates service API and
documentation
<ul>
<li>This release introduces the bandwidth reduction filter for the HEVC
encoder, increases the limits of outputs per job, and updates support
for the Nagra SDK to version 1.14.7.</li>
</ul>
</li>
<li><code>service/mq</code>: Updates service API and documentation
<ul>
<li>The Cross Region Disaster Recovery feature allows to replicate a
brokers state from one region to another in order to provide customers
with multi-region resiliency in the event of a regional outage.</li>
</ul>
</li>
<li><code>service/sagemaker</code>: Updates service API and
documentation
<ul>
<li>This release provides support in SageMaker for output files in
training jobs to be uploaded without compression and enable customer to
deploy uncompressed model from S3 to real-time inference Endpoints. In
addition, ml.trn1n.32xlarge is added to supported instance type list in
training job.</li>
</ul>
</li>
<li><code>service/transfer</code>: Updates service API and documentation
<ul>
<li>This release adds a new parameter StructuredLogDestinations to
CreateServer, UpdateServer APIs.</li>
</ul>
</li>
</ul>
<h1>Release v1.44.286 (2023-06-20)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/appflow</code>: Updates service API and
documentation</li>
<li><code>service/config</code>: Updates service API</li>
<li><code>service/ec2</code>: Updates service API and documentation
<ul>
<li>Adds support for targeting Dedicated Host allocations by assetIds in
AWS Outposts</li>
</ul>
</li>
<li><code>service/lambda</code>: Updates service API and documentation
<ul>
<li>This release adds RecursiveInvocationException to the Invoke API and
InvokeWithResponseStream API.</li>
</ul>
</li>
<li><code>service/redshift</code>: Updates service API, documentation,
and paginators
<ul>
<li>Added support for custom domain names for Redshift Provisioned
clusters. This feature enables customers to create a custom domain name
and use ACM to generate fully secure connections to it.</li>
</ul>
</li>
</ul>
<h1>Release v1.44.285 (2023-06-19)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/cloudformation</code>: Updates service API and
documentation
<ul>
<li>Specify desired CloudFormation behavior in the event of ChangeSet
execution failure using the CreateChangeSet OnStackFailure
parameter</li>
</ul>
</li>
<li><code>service/ec2</code>: Updates service API, documentation, and
examples
<ul>
<li>API changes to AWS Verified Access to include data from trust
providers in logs</li>
</ul>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="4db4de06bc"><code>4db4de0</code></a>
Release v1.44.288 (2023-06-22) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4892">#4892</a>)</li>
<li><a
href="ac3d1b0ef2"><code>ac3d1b0</code></a>
Release v1.44.287 (2023-06-21) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4891">#4891</a>)</li>
<li><a
href="e85309cb11"><code>e85309c</code></a>
Release v1.44.286 (2023-06-20) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4890">#4890</a>)</li>
<li><a
href="d3b5bbea5d"><code>d3b5bbe</code></a>
Release v1.44.285 (2023-06-19) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4889">#4889</a>)</li>
<li><a
href="87a671cef0"><code>87a671c</code></a>
Release v1.44.284 (2023-06-16) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4887">#4887</a>)</li>
<li><a
href="efb91b58d6"><code>efb91b5</code></a>
Release v1.44.283 (2023-06-15) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4884">#4884</a>)</li>
<li><a
href="8962904930"><code>8962904</code></a>
Release v1.44.282 (2023-06-13) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4882">#4882</a>)</li>
<li><a
href="dd4970433c"><code>dd49704</code></a>
Release v1.44.281 (2023-06-12) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4881">#4881</a>)</li>
<li><a
href="12e9c10fa1"><code>12e9c10</code></a>
Release v1.44.280 (2023-06-09) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4880">#4880</a>)</li>
<li><a
href="a5bef15429"><code>a5bef15</code></a>
Release v1.44.279 (2023-06-08) (<a
href="https://redirect.github.com/aws/aws-sdk-go/issues/4878">#4878</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/aws/aws-sdk-go/compare/v1.43.16...v1.44.288">compare
view</a></li>
</ul>
</details>
<br />
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
In #10957, a customer attempted to run a live query on ~60k devices. We
saw that Fleet receives all results from the devices, but stops sending
them to the browser. We believe this _might_ be a deadlock (we couldn't
confirm this because of lack of access to the Fleet instance itself).
We haven't fully confirmed this to be the reason of the
hang/missing-results, but it's worth updating because it [fixes a
possible deadlock in the
package](https://github.com/igm/sockjs-go/releases/tag/v3.0.2).
The remaining changes are the result of running `go mod tidy`.
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.~
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
---------
Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
```
go get -u go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux go.opentelemetry.io/otel go.opentelemetry.io/otel/exporters/otlp/otlptrace go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc go.opentelemetry.io/otel/sdk; go mod tidy
```
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
A recent dependabot PR (#9792) upgraded the Docker library, resulting in
errors running some tests:
```
FLEET_VERSION_A=v4.28.0 FLEET_VERSION_B=main go test ./test/upgrade *[main]
go: downloading github.com/stretchr/testify v1.8.2
--- FAIL: TestUpgradeAToB (34.73s)
fleet_test.go:71: start fleet: Error response from daemon: client version 1.42 is too new. Maximum supported API version is 1.41
```
Even after upgrading Docker Desktop to the latest, this error still
persisted. It seems that Docker released a version of the client that is
newer than the latest version of the server.
Reverting to a supported client version.
related to #10441, inspired by the prior work done in
https://github.com/kolide/fleet/pull/1360, this PR:
1. Adds a migration to use `utf8mb4_general_ci` as the default collation
for the database and all the tables. From [MySQL's documentation][1]:
> To change the table default character set and all character columns
> (CHAR, VARCHAR, TEXT) to a new character set, use a statement like
> this:
>
> ```
> ALTER TABLE tbl_name CONVERT TO CHARACTER SET charset_name;
> ```
> The statement also changes the collation of all character columns. If
> you specify no COLLATE clause to indicate which collation to use, the
> statement uses default collation for the character set.
2. Changes the connection settings to use `utf8mb4_general_ci` as the
default collation, from the [driver docs][2]:
> Sets the collation used for client-server interaction on
connection. In contrast to charset, collation does not issue additional
queries. If the specified collation is unavailable on the target server,
the connection will fail.
[1]: https://dev.mysql.com/doc/refman/5.7/en/alter-table.html
[2]: https://github.com/go-sql-driver/mysql
**TODO:** discuss how we can enforce this, is setting the database
default collation enough? should we add some kind of custom lint rule to
all migrations?
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Bumps
[github.com/kevinburke/go-bindata](https://github.com/kevinburke/go-bindata)
from 3.22.0+incompatible to 3.24.0+incompatible.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/kevinburke/go-bindata/releases">github.com/kevinburke/go-bindata's
releases</a>.</em></p>
<blockquote>
<p>v3.24.0</p>
<p>v3.23.0</p>
<p>test</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/kevinburke/go-bindata/blob/master/CHANGELOG.md">github.com/kevinburke/go-bindata's
changelog</a>.</em></p>
<blockquote>
<h2>3.24.0</h2>
<p>Remove uses of io/ioutil; you must use Go 1.18 or higher with this
version of
go-bindata and its generated asset files.</p>
<p>Update generated doc comments for compatibility with Go's updated doc
comment
guidelines.</p>
<h2>3.21.0</h2>
<p>Replace "Debug" with "AssetDebug" to reduce the
likelihood of conflicts.</p>
<h2>3.20.0</h2>
<p>Add the "Debug" constant if assets have been generated
using the <code>--debug</code> flag
at the command line.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="1b67e2de03"><code>1b67e2d</code></a>
v3.24.0</li>
<li><a
href="2708ad2791"><code>2708ad2</code></a>
CHANGELOG.md: prep for v3.24 release</li>
<li><a
href="5708eee1c2"><code>5708eee</code></a>
.github: add Go 1.18, 1.19</li>
<li><a
href="4a992e31ef"><code>4a992e3</code></a>
all: replace use of io/ioutil</li>
<li><a
href="adf7cd2dcf"><code>adf7cd2</code></a>
v3.23.0</li>
<li><a
href="54fe8e9bc4"><code>54fe8e9</code></a>
Release binaries for arm64</li>
<li><a
href="7ea0201288"><code>7ea0201</code></a>
.github: use latest Go versions</li>
<li><a
href="12dca65da1"><code>12dca65</code></a>
benchmark should write to tmp dir, not ./testdata. Fixes <a
href="https://github-redirect.dependabot.com/kevinburke/go-bindata/issues/42">#42</a>.</li>
<li><a
href="0d7fe269ac"><code>0d7fe26</code></a>
go-bindata: don't panic if given invalid regex</li>
<li><a
href="930726ace6"><code>930726a</code></a>
release: remove underscores from function names</li>
<li>Additional commits viewable in <a
href="https://github.com/kevinburke/go-bindata/compare/v3.22.0...v3.24.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.5.0 to
0.7.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="8e2b117aee"><code>8e2b117</code></a>
http2/hpack: avoid quadratic complexity in hpack decoding</li>
<li><a
href="547e7edf38"><code>547e7ed</code></a>
http2: avoid referencing ResponseWrite.Write parameter after
returning</li>
<li><a
href="39940adcaa"><code>39940ad</code></a>
html: parse comments per HTML spec</li>
<li><a
href="87ce33ecb4"><code>87ce33e</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="415cb6d518"><code>415cb6d</code></a>
all: fix some comments</li>
<li><a
href="7e3c19ca52"><code>7e3c19c</code></a>
all: correct typos in comments</li>
<li><a
href="296f09aa38"><code>296f09a</code></a>
http2: case insensitive handling for 100-continue</li>
<li><a
href="f8411da775"><code>f8411da</code></a>
nettest: fix tests on dragonfly and js/wasm</li>
<li>See full diff in <a
href="https://github.com/golang/net/compare/v0.5.0...v0.7.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts page](https://github.com/fleetdm/fleet/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps
[github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go)
from 0.12.0 to 0.18.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/getsentry/sentry-go/releases">github.com/getsentry/sentry-go's
releases</a>.</em></p>
<blockquote>
<h2>0.18.0</h2>
<p>The Sentry SDK team is happy to announce the immediate availability
of Sentry Go SDK v0.18.0.
This release contains initial support for <a
href="https://opentelemetry.io/">OpenTelemetry</a> and various other bug
fixes and improvements.</p>
<p><strong>Note</strong>: This is the last release supporting Go
1.17.</p>
<h3>Features</h3>
<ul>
<li>
<p>Initial support for <a
href="https://opentelemetry.io/">OpenTelemetry</a>.
You can now send all your OpenTelemetry spans to Sentry.</p>
<p>Install the <code>otel</code> module</p>
<pre lang="bash"><code>go get github.com/getsentry/sentry-go \
github.com/getsentry/sentry-go/otel
</code></pre>
<p>Configure the Sentry and OpenTelemetry SDKs</p>
<pre lang="go"><code>import (
"go.opentelemetry.io/otel"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"github.com/getsentry/sentry-go"
"github.com/getsentry/sentry-go/otel"
// ...
)
<p>// Initlaize the Sentry SDK
sentry.Init(sentry.ClientOptions{
Dsn: "<strong>DSN</strong>",
EnableTracing: true,
TracesSampleRate: 1.0,
})</p>
<p>// Set up the Sentry span processor
tp := sdktrace.NewTracerProvider(
sdktrace.WithSpanProcessor(sentryotel.NewSentrySpanProcessor()),
// ...
)
otel.SetTracerProvider(tp)</p>
<p>// Set up the Sentry propagator
otel.SetTextMapPropagator(sentryotel.NewSentryPropagator())
</code></pre></p>
<p>You can read more about using OpenTelemetry with Sentry in our <a
href="https://docs.sentry.io/platforms/go/performance/instrumentation/opentelemetry/">docs</a>.</p>
</li>
</ul>
<h3>Bug Fixes</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md">github.com/getsentry/sentry-go's
changelog</a>.</em></p>
<blockquote>
<h2>0.18.0</h2>
<p>The Sentry SDK team is happy to announce the immediate availability
of Sentry Go SDK v0.18.0.
This release contains initial support for <a
href="https://opentelemetry.io/">OpenTelemetry</a> and various other bug
fixes and improvements.</p>
<p><strong>Note</strong>: This is the last release supporting Go
1.17.</p>
<h3>Features</h3>
<ul>
<li>
<p>Initial support for <a
href="https://opentelemetry.io/">OpenTelemetry</a>.
You can now send all your OpenTelemetry spans to Sentry.</p>
<p>Install the <code>otel</code> module</p>
<pre lang="bash"><code>go get github.com/getsentry/sentry-go \
github.com/getsentry/sentry-go/otel
</code></pre>
<p>Configure the Sentry and OpenTelemetry SDKs</p>
<pre lang="go"><code>import (
"go.opentelemetry.io/otel"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"github.com/getsentry/sentry-go"
"github.com/getsentry/sentry-go/otel"
// ...
)
<p>// Initlaize the Sentry SDK
sentry.Init(sentry.ClientOptions{
Dsn: "<strong>DSN</strong>",
EnableTracing: true,
TracesSampleRate: 1.0,
})</p>
<p>// Set up the Sentry span processor
tp := sdktrace.NewTracerProvider(
sdktrace.WithSpanProcessor(sentryotel.NewSentrySpanProcessor()),
// ...
)
otel.SetTracerProvider(tp)</p>
<p>// Set up the Sentry propagator
otel.SetTextMapPropagator(sentryotel.NewSentryPropagator())
</code></pre></p>
<p>You can read more about using OpenTelemetry with Sentry in our <a
href="https://docs.sentry.io/platforms/go/performance/instrumentation/opentelemetry/">docs</a>.</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="4b97c8e661"><code>4b97c8e</code></a>
release: 0.18.0</li>
<li><a
href="79724631e3"><code>7972463</code></a>
fix(release): Bump root module version in submodules (<a
href="https://github-redirect.dependabot.com/getsentry/sentry-go/issues/576">#576</a>)</li>
<li><a
href="612c167049"><code>612c167</code></a>
fix(build): Tidy otel submodule (<a
href="https://github-redirect.dependabot.com/getsentry/sentry-go/issues/575">#575</a>)</li>
<li><a
href="3ec3943355"><code>3ec3943</code></a>
doc: Prepare 0.18.0 (<a
href="https://github-redirect.dependabot.com/getsentry/sentry-go/issues/572">#572</a>)</li>
<li><a
href="bb6e2bdbf0"><code>bb6e2bd</code></a>
fix(otel): Use vendored bagggage implementation in propagator (<a
href="https://github-redirect.dependabot.com/getsentry/sentry-go/issues/573">#573</a>)</li>
<li><a
href="3964ecefd0"><code>3964ece</code></a>
fix(baggage): Update baggage parsing and encoding in vendored otel
package (#...</li>
<li><a
href="ef3a838eef"><code>ef3a838</code></a>
test: Add Codecov configuration file (<a
href="https://github-redirect.dependabot.com/getsentry/sentry-go/issues/569">#569</a>)</li>
<li><a
href="de904d6e97"><code>de904d6</code></a>
fix(otel): Require otel 1.11.0 and higher (<a
href="https://github-redirect.dependabot.com/getsentry/sentry-go/issues/567">#567</a>)</li>
<li><a
href="d0a2aa3ebd"><code>d0a2aa3</code></a>
fix: Create a frozen DSC when calling ToBaggage (<a
href="https://github-redirect.dependabot.com/getsentry/sentry-go/issues/566">#566</a>)</li>
<li><a
href="fb7618028b"><code>fb76180</code></a>
release: Update .craft.yml to be ready for otel submodule release (<a
href="https://github-redirect.dependabot.com/getsentry/sentry-go/issues/565">#565</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/getsentry/sentry-go/compare/v0.12.0...v0.18.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Issue #9599
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Two minor things are happening here:
1. A new direct dependency was added in #9489, this runs `go mod tidy`
to update `go.mod` and `go.sum`
2. `"go.uber.org/atomic"` was automatically imported instead of
`"sys/atomic"` in #9494
This may be needed for CIS 2.3.2.2 check:
```
Correct date and time settings are required for authentication protocols, file creation,
modification dates and log entries. Ensure that time on the computer is within
acceptable limits. Truly accurate time is measured within milliseconds. For this audit, a
drift under four and a half minutes passes the control check. Since Kerberos is one of
the important features of macOS integration into Directory systems, the guidance here
is to warn you before there could be an impact to operations. From the perspective of
accurate time, this check is not strict, so it may be too great for your organization. Your
organization can adjust to a smaller offset value as needed.
```
#9239
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
This PR provides the tooling that fleetdm.com will use to sign MDM Push
CSRs.
This was tested against the output of #8812, and the resulting requests
were validated to be accepted by Apple.
Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
this upgrades the following packages by running:
```
go get golang.org/x/text@v0.3.8
go get golang.org/x/net@v0.0.0-20220906165146-f3363e06e74c
go get github.com/docker/distribution@v2.8.0
```
`golang.org/x/text` and `golang.org/x/net` are safe to upgrade as they
only contain the security fix.
`github.com/docker/distribution` is only used by `test/upgrade` and I
have verified that everything works properly by running a test.
* Update go-tuf to v0.5.0
This was triggered by the security advisory
[GHSA-3633-5h82-39pq](https://github.com/theupdateframework/go-tuf/security/advisories/GHSA-3633-5h82-39pq).
Fleet's use of go-tuf is not vulnerable to this issue due to not using
key thresholds greater than 1.
There were some API changes that necessitate changing the initialization
code for the TUF client. See
https://github.com/theupdateframework/go-tuf/issues/379 for further
discussion.
* Add changes file
* Update default root metadata
* Add review changes to update-go-tuf branch
* Update tests
* Add more checks to roots output
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* WIP
* Adding DEP functionality to Fleet
* Better organize additional MDM code
* Add cmdr.py and amend API paths
* Fix lint
* Add demo file
* Fix demo.md
* go mod tidy
* Add munki setup to Fleet
* Add diagram to demo.md
* Add fixes
* Update TODOs and demo.md
* Fix cmdr.py and add TODO
* Add endpoints to demo.md
* Add more Munki PoC/demo stuff
* WIP
* Remove proposals from PoC
* Replace prepare commands with fleetctl commands
* Update demo.md with current state
* Remove config field
* Amend demo
* Remove Munki setup from MVP-Dogfood
* Update demo.md
* Add apple mdm commands (#7769)
* fleetctl enqueue mdm command
* fix deps
* Fix build
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
* Add command to upload installers
* go mod tidy
* fix subcommands help
There is a bug in urfave/cli where help text is not generated properly when subcommands
are nested too deep.
* Add support for installing apps
* Add a way to list enrolled devices
* Add dep listing
* Rearrange endpoints
* Move DEP routine to schedule
* Define paths globally
* Add a way to list enrollments and installers
* Parse device-ids as comma-separated string
* Remove unused types
* Add simple commands and nest under enqueue-command
* Fix simple commands
* Add help to enqueue-command
* merge apple_mdm database
* Fix commands
* update nanomdm
* Split nanomdm and nanodep schemas
* Set 512 MB in memory for upload
* Remove empty file
* Amend profile
* Add sample commands
* Add delete installers and fix bug in DEP profile assigning
* Add dogfood.md deployment guide
* Update schema.sql
* Dump schema with MySQL 5
* Set default value for authenticate_at
* add tokens to enrollment profiles
When a device downloads an MDM enrollment profile, verify the token passed
as a query parameter. This ensures untrusted devices don't enroll with
our MDM server.
- Rename enrollments to enrollment profiles. Enrollments is used by nano
to refer to devices that are enrolled with MDM
- Rename endpoint /api/<version>/fleet/mdm/apple/enrollments to ../enrollmentprofiles
- Generate a token for authentication when creating an enrollment profile
- Return unauthorized if token is invalid when downloading an enrollment profile from /api/mdm/apple/enroll?token=
* remove mdm apple server url
* update docs
* make dump-test-schema
* Update nanomdm with missing prefix table
* Add docs and simplify changes
* Add changes file
* Add method docs
* Fix compile and revert prepare.go changes
* Revert migration status check change
* Amend comments
* Add more docs
* Clarify storage of installers
* Remove TODO
* Remove unused
* update dogfood.md
* remove cmdr.py
* Add authorization tests
* Add TODO comment
* use kitlog for nano logging
* Add yaml tags
* Remove unused flag
* Remove changes file
* Only run DEP routine if MDM is enabled
* Add docs to all new exported types
* Add docs
* more nano logging changes
* Fix unintentional removal
* more nano logging changes
* Fix compile test
* Use string for configs and fix config test
* Add docs and amend changes
* revert changes to basicAuthHandler
* remove exported BasicAuthHandler
* rename rego authz type
* Add more information to dep list
* add db tag
* update deps
* Fix schema
* Remove unimplemented
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* improve error handling in vulnerabilities cron
* fix tests
* Use errHandler and go mod tidy
* Add dep that got removed by mod tidy
* add dsl to tools
* Add changes file
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Create Bulk Users
* WIP: Adding a test for bulk user import
* adding a user bulk create test
* Fixing description, removing password required, and adding more test cases
* Fixing description, removing password required, and adding more test cases
* Fixed all comments and added Random Password Generator
* returning an error in generateRandomPassword
* Using 2 loops to create user list and then create the actual users
* Adding a bulk user delete
* fixing a mistake in temp csv
* fixed lints and removed yamlFlag
* Do not use golangci action for better reproducibility
* Add fix to trigger build
* Fix all reported issues
* fix more lint errors
* Add missing import
* Remove unused method
* Remove change not necessary
Feature: Improve our capability to detect vulnerable software on Ubuntu hosts
To improve the capability of detecting vulnerable software on Ubuntu, we are now using OVAL definitions to detect vulnerable software on Ubuntu hosts. If data sync is enabled (disable_data_sync=false) OVAL definitions are automatically kept up to date (they are 'refreshed' once per day) - there's also the option to manually download the OVAL definitions using the 'fleetctl vulnerability-data-stream' command. Downloaded definitions are then parsed into an intermediary format and then used to identify vulnerable software on Ubuntu hosts. Finally, any 'recent' detected vulnerabilities are sent to any third-party integrations.
This solves #5679 , and also implements #5515, #5509 and lays the ground for #5516
With the introduction of Wrap, Is and As in the standard library, we've now got built-in support for wrapping.
On top of that, a common pattern in the community is to define errors tailored to the context of each project while still conforming to the error and Unwrap interfaces (see Upspin, Chromium)
The output now includes stack traces and additional info
* Add logs to troubleshoot orbit
* Run journalctl on a different step
* Add legacy orbit support to opt version of fleetctl
* Fix macos logs permission error
* Checkout repository
* Compile fleetctl from branch
Install orbit to /opt instead of /var/lib. When installing to /var/lib,
the default selinux context of var_lib_t gets applied, which results in
an AVC error when running via systemd.
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
Allows identification of which Orbit versions are in use from the update
server.
Refactored the build information into a separate `package build` to
support importing it from multiple places.
* Orbit: Add Fleet Desktop support to Windows
* Rename workflow, fix linux build
* Do not compile systray on linux
* nolint on unused
* Fix lint properly
* nolint both checkers
* Fix monitor logic in desktopRunner
* Fix interrupt and execute order
* Upgrade and replace kolide/osquery-go with osquery/osquery-go
* Upgrade macadmins/osquery-extension to v0.0.7
* Upgrade kolide/launcher to latest
* go mod tidy
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Fleet Desktop Packaging WIP
* Ignore gosec warning
* Trigger on PR too
* Install Go in workflow
* Pass url parameter to desktop app
* Fix fleetctl package
* Final set of changes for v1 of Fleet Desktop
* Add changes
* PR fixes
* Fix CI build
* add larger menu bar icon
* Add transparency item
* Delete host_device_auth entry on host deletion
* Add SetTargetChannel
* Update white logo and add desktop to update runner
* Add fleet-desktop monitoring to orbit
* Define fleet-desktop app exec name
* Fix update runner creation
* Add API test before enabling the My device menu item
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* geoip wip
* return nil if ip is empty string or if ParseIP returns nil
* add ui component to render geolocation if available, address PR feedback
* render public ip if available
* add changes file, document geoip in deployment guide
* update rest-api docs
* Add disable-updates flag to fleetctl and orbit
* Fix ruleguard execution error on make lint-go
* Introduce dev-mode for ease of development of orbit
* Add changes file
* Add CentOS parsing and post-processing in fleet
* Add tests and amend SyncCPEDatabase
* Add test for centosPostProcessing
* Changes from PR comments
* Amend software test
* Fix sync test
* Add index to source and vendor
* Use os.MkdirTemp
* Rearrange migrations
* Regenerate test schema
* Add support for testing migrations (#4112)
* Add support for testing migrations
* Rename migration in tests
* Changes suggested in PR
* Go mod tidy
Resolves the warning described in #3699 by updating to the latest
version of the dependency with the warning fixed.
The warning should go away on all clients after new metadata is
generated with these changes.
* Add sentry
* Fix gosum
* More gosum fixes
* Add missing def for config
* Enrich sentry scope a bit
* Add changes file
* Add goroutine safe scope to errors
* Encapsulate sentry logic
* Add documentation for new flag
* Add sentry capturing to crons and other background tasks
* Only send to sentry when enabled
* Add software count API
* Fix makefile
* Fine no mock generating at this point
* Actually, one last try
* Use go install instead
* Fix go sum/mod
* Improve documentation
* Try setting node to 14
* Do caching of app config per instance instead of across all of them in redis
* Add changes file
* Simplify code based on review comment
* Use go-cache instead of creating our own
* Dont export consts
* Copy app config before returning it
* Fix lint
* Update go sum
* Update go sum
* Fix races in go tests and run with -race on CI
* Fix race in pubsub
* Increase timeout to 15m for go tests
* CI takes forever, try disabling race
* Remove timeout from go tests
* Add safe mkdirall and open
* Use secure as much as possible and merge gomodules for orbit to fleet
* Improve openfile and mkdirall to check for permissiveness instead of equality
* Don't shift
* Fix links
* Address review comments
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Add first cve db generation
* WIP but with final strategy, preparring to merge main
* Fix merge conflicts
* WIP
* wip
* Insert CVEs to the db
* Remove unused code
* Use wg instead of counting
* Call cancelFunc to avoid ctx leak
* Fix logs for better readability
* Point code to fleetdm instead of my repo
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Address review comments
- Use goreleaser to automate release process.
- Add new dockerfiles for fleet (with fleetctl) and fleetctl (only).
- Add GitHub Action Workflow to run goreleaser on new tag.
- Update NPM to match new archive naming.
This should support Redis in both cluster and non-cluster modes.
Updates were made separately to github.com/throttled/throttled to support the slight changes in types.
Co-authored-by: Joseph Macaulay <joseph.macaulay@uber.com>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
1. use [staticcheck](https://staticcheck.io/) to check the code, and fix some issues.
2. use `go fmt` to format the code.
3. use `go mod tidy` clean the go mod.
- Maintain software inventory with detail queries.
- Associated database migrations.
- Feature flagged off by default (see documentation for details to turn on).
- Documentation.
- New test helper for slice element comparisons skipping ID.
- Fix issue with built-in labels showing multiple platforms when hosts
are reinstalled with new platform.
- Add Red Hat Linux built-in label.
- Display more labels by default in target selector.
Fixes#546, #553
Prevent abuse of these endpoints with rate limiting backed by Redis. The
limits assigned should be appropriate for almost any Fleet deployment.
Closes#530
This PR contains the initial implementation of the fleetctl updates commands, along with documentation on using this to self-host an agent update server.
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
- This seems to be the maintained, trusted (by Homebrew, etc.) version
of go-bindata.
- Add tools.go file to pin version with go modules.
- Use go run to run the binary, making easier configuration for new developers.
