This removes policy information from `GET /api/_version_/fleet/device/{token}` from non-premium Fleet instances.
Starting the server with `./build/fleet serve --dev --dev_license`
```bash
$ curl -s https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a | jq '.host.policies // "not present"'
[
{
"id": 3,
"name": "Antivirus healthy (Linux)",
"query": "SELECT score FROM (SELECT case when COUNT(*) = 2 then 1 ELSE 0 END AS score FROM processes WHERE (name = 'clamd') OR (name = 'freshclam')) WHERE score == 1;",
"description": "Checks that both ClamAV's daemon and its updater service (freshclam) are running.",
"author_id": 1,
"author_name": "Roberto",
"author_email": "test@example.com",
"team_id": null,
"resolution": "Ensure ClamAV and Freshclam are installed and running.",
"platform": "darwin,linux",
"created_at": "2022-05-23T20:53:36Z",
"updated_at": "2022-06-03T13:17:42Z",
"response": ""
}
]
```
Starting the server with `./build/fleet serve --dev`
```bash
$ curl -s https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a | jq '.host.policies // "not present"'
"not present"
```
* fix old root dir in orbit
* add changes
* Add automation for orbit shell (with TUF)
* Fix workflow syntax
* Add logging to latest fleetctl preview action
* Add changes to fix workflow
* Use macOS host for TUF server and package generation
* Remove copy/paste if clause
* Fix orbit logs on macOS, Ubuntu
* Simplify TUF and generation of packages
* Set enroll secret instead of getting it
* Increase timeouts
* Add step id
* Fixes to the upload/download of artifacts
* Rearrange steps to not lose the downloads
* Fix copy/paste
* Add fleetctl login step
* Add missing config set
* Fix quotes on Windows
* Increase timeout
* Fix job termination
* Disable FLEET_DESKTOP for now
* Checkout repository on macOS
* Fix logs path
* Enable fleet desktop
* Use cancel, nitpick
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
This adds a new device authenticated endpoint, `/api/_version_/fleet/device/{token}/policies` to retrieve the device policies.
An example request / response looks like:
```bash
curl https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a/policies
```
```json
{
"policies": [
{
"id": 3,
"name": "Antivirus healthy (Linux)",
"query": "SELECT score FROM (SELECT case when COUNT(*) = 2 then 1 ELSE 0 END AS score FROM processes WHERE (name = 'clamd') OR (name = 'freshclam')) WHERE score == 1;",
"description": "Checks that both ClamAV's daemon and its updater service (freshclam) are running.",
"author_id": 1,
"author_name": "Admin",
"author_email": "admin@example.com",
"team_id": null,
"resolution": "Ensure ClamAV and Freshclam are installed and running.",
"platform": "darwin,windows,linux",
"created_at": "2022-05-23T20:53:36Z",
"updated_at": "2022-05-23T20:53:36Z",
"response": "fail"
}
]
}
```
Related to [#5685](https://github.com/fleetdm/fleet/issues/5685), in another changeset I will be adding "client" endpoints so we can consume this endpoint from Fleet Desktop
Related to #5776, this ensures that when a host is re-enrolled on a different team we cleanup existing policy memberships in the same way we do when a host is assigned a team through the API.
This adds documentation about our debugging endpoints and a brief excerpt about the fleetctl debug command with instructions to generate the archive so we have a place with instructions to which we can direct people.
It also adds the .prof file extension to profiling files, which hopefully clarifies that they are meant to be used by go tool pprof.
This adds compatibility in `fleetctl preview` to work with docker compose (version 2). Since this version was released this April, we are still keeping backwards compatibility and using docker-compose as a fallback.
As v2 is now the recommended version and v1 is deprecated, this also rewords all prompts and help messages to say "docker compose".
Rel: #5746
this change prevents errors from being automatically cleared once they are read. A new flag `-flush` is introduced to flush errors on read if necessary.
Related to https://github.com/fleetdm/fleet/issues/5504, this change attempts to improve the output of the `fleetctl debug errors` command by:
- Adding a warning message to redact sensitive data
- Adding a `json` extension to the output file
- Allowing to stream the output to stdout via the `-stdout` flag or the `STDOUT` env var
The output after this changes is:
```
~/projects/fleet $ ./build/fleetctl debug errors
################################################################################
# WARNING:
# The generated file may contain sensitive data.
# Please review the file before sharing.
#
# Output written to: fleet-errors-2022-05-05T12:46:42-03:00.json
################################################################################
```
It also modifies the output of `fleetctl debug archive`
```
################################################################################
# WARNING:
# The files in the generated archive may contain sensitive data.
# Please review them before sharing.
#
# Archive written to: fleet-profiles-archive-2022-05-05T12:46:59-03:00.tar.gz
################################################################################
```
* Add (beta) support for Fleet Desktop to linux
* Add dependency for linux desktop
* Amend makefile uname check
* Clarify env vars used for linux in execuser
* Add final set of fixes
* Remove -it from docker run
* Add desktop to the update runner for Linux
* Re-arrange tag.gz and fix upgrade check for linux desktop
Install orbit to /opt instead of /var/lib. When installing to /var/lib,
the default selinux context of var_lib_t gets applied, which results in
an AVC error when running via systemd.
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
To add support for #400, we're using the macadmins/osquery-extension to gather Google Chrome profiles from hosts.
Under the hood, the extension looks and parses a json file in which Chrome stores a bunch of data. Given that emails are not required to create Chrome profiles, some of the profiles stored in this file and returned by the query we're using contain empty emails.
The idea after this change is to prevent empty emails from being ingested in the first place instead of filtering them after the fact. I have also included a migration to clean the rows with empty email columns.
Fixes#4780
* Bug 5066: Format config durations
Change duration values returned by 'fleetctl get config --include-server-config' from nanoseconds to a human readable format.
* Fix access control issues with users
* Fix access control issues with packs
* Fix access control issues with software
* Changes suggested by Martin
* All users can access the global schedule
* Restrict access to activities
* Add explicit test for team admin escalation vuln
* All global users should be able to read all software
* Handbook editor pass - Security - GitHub Security (#5108)
* Update security.md
All edits are recorded by line:
395 replaced “open-source” with “open source”
411 replaced “open-source” with “open source”
439 added “the” before “comment”; replaced “repositories,” with “repositories”
445 deleted “being” before “located”
458 added “and” after “PR”
489 replaced “on” with “in”
493 replaced “open-source” with “open source”; Replaced “privileges,” with “privileges”
* Update security.md
line 479
* Update security.md
added (static analysis tools used to identify problems in code) to line 479
* Fix UI
* Fix UI
* revert api v1 to latest in documentation (#5149)
* revert api v1 to latest in documentation
* Update fleetctl doc page
Co-authored-by: Noah Talerman <noahtal@umich.edu>
* Add team admin team policy automation; fix e2e
* Update to company page of the handbook (#5164)
Updated "Why do we use a wireframe-first approach?" section of company.md
* removed extra data on smaller screens (#5154)
* Update for team automations; e2e
* Jira Integration: Cypress e2e tests only (#5055)
* Update company.md (#5170)
This is to update the formatting under "empathy" and to fix the spelling of "help text."
This was done as per @mikermcneil .
This is related to #https://github.com/fleetdm/fleet/pull/4941 and https://github.com/fleetdm/fleet/issues/4902
* fix update updated_at for aggregated_stats (#5112)
Update the updated_at column when using ON DUPLICATE UPDATE so that
the counts_updated_at is up to date
* basic sql formatting in code ie whitespace around operators
* Fix e2e test
* Fix tests in server/authz
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Desmi-Dizney <99777687+Desmi-Dizney@users.noreply.github.com>
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Update the updated_at column when using ON DUPLICATE UPDATE so that
the counts_updated_at is up to date
* basic sql formatting in code ie whitespace around operators
* Remove required password reset flag when creating new API-only user
* Add test for 'fleetctl user create' command
* Set NewUserFuncInvoked to false in between tests
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
* added teams webhook calls
* split global admin and team admin tests; added policy automation
* changes file
* fixed type errors
* fixed e2e test
* fixed typo
* fixed admin test syntax
* fixed test logic for team maintainer
* lint fixes
* more admin e2e test fixes
* fixed team policy test
* removed duplicate test
This optimizes the query in particular for Domain Controllers, where
there are a number of users with no local accounts and don't need to be
searched for software.
See #4261
not set on the INSERT.
- OUT: Only sets the ID on the passed session and returns it. (`CreatedAt`, `AccessedAt`, are not set.)
New version:
```go
func (ds *Datastore) NewSession(ctx context.Context, userID uint, sessionKey string) (*fleet.Session, error) {
sqlStatement := `
INSERT INTO sessions (
user_id,
` + "`key`" + `
)
VALUES(?,?)
`
result, err := ds.writer.ExecContext(ctx, sqlStatement, userID, sessionKey)
if err != nil {
return nil, ctxerr.Wrap(ctx, err, "inserting session")
}
id, _ := result.LastInsertId() // cannot fail with the mysql driver
return ds.sessionByID(ctx, ds.writer, uint(id))
}
```
- IN: Define arguments that are truly used when creating a session.
- OUT: Load and return the fleet.Session struct with all values set (using the `ds.writer` to support read replicas correctly).
PS: The new `NewSession` version mimics what we already do with other entities, like policies (`Datastore.NewGlobalPolicy`).
* Orbit: Add Fleet Desktop support to Windows
* Rename workflow, fix linux build
* Do not compile systray on linux
* nolint on unused
* Fix lint properly
* nolint both checkers
* Fix monitor logic in desktopRunner
* Fix interrupt and execute order
Renaming a temporary file to the final destination does not always work.
Specifically, if the source and destination paths are on different file
systems, you will get the following error
invalid cross-device link
Instead, create temporary file in the destination directory.
- Update CHANGELOG
- Bump versioning
- Tweak documentation
- Default `session_duration` to `5d`
- Add extra `#` to "Team policies" section so it doesn't show up in top level nav for docs
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Fleet Desktop Packaging WIP
* Ignore gosec warning
* Trigger on PR too
* Install Go in workflow
* Pass url parameter to desktop app
* Fix fleetctl package
* Final set of changes for v1 of Fleet Desktop
* Add changes
* PR fixes
* Fix CI build
* add larger menu bar icon
* Add transparency item
* Delete host_device_auth entry on host deletion
* Add SetTargetChannel
* Update white logo and add desktop to update runner
* Add fleet-desktop monitoring to orbit
* Define fleet-desktop app exec name
* Fix update runner creation
* Add API test before enabling the My device menu item
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
Add new usePlatformSelector custom hook
Add new usePlatformCompatibility custom hook
Add new PlatformSelector global component
Refactor PlatformCompatibility as global component
Refactor sql_tools to TypeScript
Improve type definitions for context/policy
Align PolicyPage and QueryPage with platform compatibility changes
* geoip wip
* return nil if ip is empty string or if ParseIP returns nil
* add ui component to render geolocation if available, address PR feedback
* render public ip if available
* add changes file, document geoip in deployment guide
* update rest-api docs
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Trigger workflow on PR
* Fixes to ease handling of artifact
* Fix, do not use target name as dir
* Remove workaround
* Error early if email already exists
* Update changes/issue-4361-mail-change-should-error
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
* Fix test
* Lint fixes
* Fix another test
* Also check for invites
* Improve error checks
* Update comment
* Update tests
* Fix test
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
* Add disable-updates flag to fleetctl and orbit
* Fix ruleguard execution error on make lint-go
* Introduce dev-mode for ease of development of orbit
* Add changes file
* Add CentOS parsing and post-processing in fleet
* Add tests and amend SyncCPEDatabase
* Add test for centosPostProcessing
* Changes from PR comments
* Amend software test
* Fix sync test
* Add index to source and vendor
* Use os.MkdirTemp
* Rearrange migrations
* Regenerate test schema
* Add support for testing migrations (#4112)
* Add support for testing migrations
* Rename migration in tests
* Changes suggested in PR
* Go mod tidy
* Fix OSVersion to work properly when distribution does not follow symver
Certain distros such as ClearLinux or ArchLinux do not use symver or any
dotted versioning scheme for their releases. Archlinux uses the static
string "Rolling" and ClearLinux uses a single build number such as 35550
for their versions.
In Fleet console, this shows up as a string like "Archlinux 0.0.0.0"
which makes very little sense to the user. This change makes it so that
if OSQuery cannot generate a dotted version number, we should instead
use the build id as an opaque string.
* Add /changes
* Add tests for os_version
* fixup
* fixup
* Add platform filters for MDM/Munki/Chrome queries
This should help quiet warnings that users/customers have reported when
these queries try to run on platforms without the macadmins extension
tables.
For #4123
* Improve documentation
* add changes file
* revert doc formatting
* Update tests
* Yet another test fix
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Add remaining policy templates
* Removeempty resolution field
* Update naming for standard policies
* Add enabled word to SIP policy
* Use full SIP
* Also change in constants
* Update windows disk encryption
* Add changes file
* Tweak windows disk encryption policy
* Address lint errors
* Make requested changes
* Reflect changes in policy tempaltes
* Make sure that standard policies and policy templates are the same
* Edit automatic login disabled description
* Also edit in constants
* Change webhooks db lock duration to one hour
* Reload interval and check time left
* Change interval to one hour to match the db lock
* Add missing continue in case of failure
* Fix lint-go
* Allow to search with target_sw with a period
* Remove tests for a different thign
* Add a test for programs as a source as well
* Use MATCH again and add proper tests
- Add new "Software" tab to host details page
- Add dropdown to filter vulnerable software
- Extend DataTable client-side filtering to accommodate filter inputs that are controlled by parent components
- Refactor host details software into separate component
* Add new software page
* Add hosts counts and last updated to software card on homepage
* Extend InfoCard component to all children to set dynamic title information and hyperlinks
* Add new TeamsDropdownHeader component
* Refactor TableContainer to move server-side pagination buttons to DataTable
* Extend DataTable to include footer element
* Better jitter (#3716) (#3744)
* Better jitter (#3716)
* Better jitter
* Fix lint
* Use milliseconds
* Make duration milliseconds
* Update based on Lucas' suggestion
* Add changes file
* Panic on error
* Fix compilation error
* Use time.after instead of time.tick to not leak
* Include browser extensions in software inventory (#3733) (#3757)
Use appropriate JOINs against users table to include all results.
For #3557
* Build Docker images on push to any branch (#3756) (#3758)
Allows for testing release branches and any others.
* Optimize users detail query (#3754) (#3770)
@Smjert reported instances of Windows Domain Controllers having massive
resource utilization and being killed by the watchdog when running this
query. In his test environment, this new query performs much better.
* Refactor async host processing to avoid redis SCAN keys (for policies) (#3657) (#3755)
* Issue 3707 clean targets on delete (#3739) (#3776)
* wip
* Delete targets when deleting teams, hosts, and labels
* Add changes file
* Fix error message
* Remove unused teamsTable
* Cleanup new pack
* Clean new packs at end of test
* Update operating system policy (#3779)
- Update policy's query to check for operating system versions greater than or equal to
* Improve loading of manage policies page (#3695) (#3781)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Fix logo (#3765)
* fixed duplicate query modal not closing (#3787)
* fixed style for connection line on activity feed (#3789)
* Improve loading state for query platform compatibility (#3752) (#3783)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Disable pagination for teams page (#3794)
* Shorten "simple" query API period to 25s (#3775) (#3804)
This helps the period stay under the default request timeouts for most
load balancers.
Some default timeouts:
* AWS ALB - 60s
* Nginx - 60s
* GCP LB - 30s
* Clear user checkbox selections after successfully removing selected row from table UI #3798 (#3801)
* Alias fleetctl sandbox to fleetctl preview (#3803) (#3806)
Preparing for some work the product team is planning to rebrand preview
to sandbox.
* Remove flash of spinner to no spinner (#3799)
* Revert "Better jitter (#3716) (#3744)" (#3807)
This reverts commit c857b52d2b.
* Fix teams dropdown sizing (#3759) (#3810)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* fixed link for expiration message (#3788)
* Fixed empty string for compatible platforms (#3812)
* fixed empty string for compatible platforms
* default checked for platforms on policy query
* Block autocomplete for username and password in SMTP settings (#3732) (#3816)
* No hosts for software/policy renders better message (#3701) (#3821)
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
* Get failing policies webhook data on page load (#3824)
* Fix reset password flow (#3826)
* Prepare for 4.9.0 (#3820)
- Add CHANGELOG
- Bump versioning
- Remove changes files
- Spelling fix for operating system policy's description
* Remove merge artifact
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Luke Heath <luke@fleetdm.com>
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
* Add sentry
* Fix gosum
* More gosum fixes
* Add missing def for config
* Enrich sentry scope a bit
* Add changes file
* Add goroutine safe scope to errors
* Encapsulate sentry logic
* Add documentation for new flag
* Add sentry capturing to crons and other background tasks
* Only send to sentry when enabled
This helps the period stay under the default request timeouts for most
load balancers.
Some default timeouts:
* AWS ALB - 60s
* Nginx - 60s
* GCP LB - 30s
* Allow sort by more than one key
* more accurate message for no hosts on filter
* added changes file
* lint fix
* more accurate message for no hosts on filter
* added changes file
* lint fix
* small logical change
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* wip
* Delete targets when deleting teams, hosts, and labels
* Add changes file
* Fix error message
* Remove unused teamsTable
* Cleanup new pack
* Clean new packs at end of test
@Smjert reported instances of Windows Domain Controllers having massive
resource utilization and being killed by the watchdog when running this
query. In his test environment, this new query performs much better.
* WIP
* Amend tests
* Do not load aggregated stats for packs
* Add option to host lite
* Fix remaining TODOs
* Fix osquery_utils tests
* Fix SQL
* Fix SQL (bis)
* Restore AuthenticateHost to load once
* Code improvements and re-add deferred host save
* More fixes to the PR
* Wrap users table update on tx
* Add caching to ListPacksForHost and ListScheduledQueriesInPack
* Remove SaveHostSoftware (replaced by UpdateHostSoftware)
* Add unit tests for new functionality
* Add changes file
* Fix scheduled queries test
