This adds documentation about our debugging endpoints and a brief excerpt about the fleetctl debug command with instructions to generate the archive so we have a place with instructions to which we can direct people.
It also adds the .prof file extension to profiling files, which hopefully clarifies that they are meant to be used by go tool pprof.
This adds compatibility in `fleetctl preview` to work with docker compose (version 2). Since this version was released this April, we are still keeping backwards compatibility and using docker-compose as a fallback.
As v2 is now the recommended version and v1 is deprecated, this also rewords all prompts and help messages to say "docker compose".
Rel: #5746
this change prevents errors from being automatically cleared once they are read. A new flag `-flush` is introduced to flush errors on read if necessary.
Related to https://github.com/fleetdm/fleet/issues/5504, this change attempts to improve the output of the `fleetctl debug errors` command by:
- Adding a warning message to redact sensitive data
- Adding a `json` extension to the output file
- Allowing to stream the output to stdout via the `-stdout` flag or the `STDOUT` env var
The output after this changes is:
```
~/projects/fleet $ ./build/fleetctl debug errors
################################################################################
# WARNING:
# The generated file may contain sensitive data.
# Please review the file before sharing.
#
# Output written to: fleet-errors-2022-05-05T12:46:42-03:00.json
################################################################################
```
It also modifies the output of `fleetctl debug archive`
```
################################################################################
# WARNING:
# The files in the generated archive may contain sensitive data.
# Please review them before sharing.
#
# Archive written to: fleet-profiles-archive-2022-05-05T12:46:59-03:00.tar.gz
################################################################################
```
* Add (beta) support for Fleet Desktop to linux
* Add dependency for linux desktop
* Amend makefile uname check
* Clarify env vars used for linux in execuser
* Add final set of fixes
* Remove -it from docker run
* Add desktop to the update runner for Linux
* Re-arrange tag.gz and fix upgrade check for linux desktop
Install orbit to /opt instead of /var/lib. When installing to /var/lib,
the default selinux context of var_lib_t gets applied, which results in
an AVC error when running via systemd.
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
To add support for #400, we're using the macadmins/osquery-extension to gather Google Chrome profiles from hosts.
Under the hood, the extension looks and parses a json file in which Chrome stores a bunch of data. Given that emails are not required to create Chrome profiles, some of the profiles stored in this file and returned by the query we're using contain empty emails.
The idea after this change is to prevent empty emails from being ingested in the first place instead of filtering them after the fact. I have also included a migration to clean the rows with empty email columns.
Fixes#4780
* Bug 5066: Format config durations
Change duration values returned by 'fleetctl get config --include-server-config' from nanoseconds to a human readable format.
* Fix access control issues with users
* Fix access control issues with packs
* Fix access control issues with software
* Changes suggested by Martin
* All users can access the global schedule
* Restrict access to activities
* Add explicit test for team admin escalation vuln
* All global users should be able to read all software
* Handbook editor pass - Security - GitHub Security (#5108)
* Update security.md
All edits are recorded by line:
395 replaced “open-source” with “open source”
411 replaced “open-source” with “open source”
439 added “the” before “comment”; replaced “repositories,” with “repositories”
445 deleted “being” before “located”
458 added “and” after “PR”
489 replaced “on” with “in”
493 replaced “open-source” with “open source”; Replaced “privileges,” with “privileges”
* Update security.md
line 479
* Update security.md
added (static analysis tools used to identify problems in code) to line 479
* Fix UI
* Fix UI
* revert api v1 to latest in documentation (#5149)
* revert api v1 to latest in documentation
* Update fleetctl doc page
Co-authored-by: Noah Talerman <noahtal@umich.edu>
* Add team admin team policy automation; fix e2e
* Update to company page of the handbook (#5164)
Updated "Why do we use a wireframe-first approach?" section of company.md
* removed extra data on smaller screens (#5154)
* Update for team automations; e2e
* Jira Integration: Cypress e2e tests only (#5055)
* Update company.md (#5170)
This is to update the formatting under "empathy" and to fix the spelling of "help text."
This was done as per @mikermcneil .
This is related to #https://github.com/fleetdm/fleet/pull/4941 and https://github.com/fleetdm/fleet/issues/4902
* fix update updated_at for aggregated_stats (#5112)
Update the updated_at column when using ON DUPLICATE UPDATE so that
the counts_updated_at is up to date
* basic sql formatting in code ie whitespace around operators
* Fix e2e test
* Fix tests in server/authz
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Desmi-Dizney <99777687+Desmi-Dizney@users.noreply.github.com>
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Update the updated_at column when using ON DUPLICATE UPDATE so that
the counts_updated_at is up to date
* basic sql formatting in code ie whitespace around operators
* Remove required password reset flag when creating new API-only user
* Add test for 'fleetctl user create' command
* Set NewUserFuncInvoked to false in between tests
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
* added teams webhook calls
* split global admin and team admin tests; added policy automation
* changes file
* fixed type errors
* fixed e2e test
* fixed typo
* fixed admin test syntax
* fixed test logic for team maintainer
* lint fixes
* more admin e2e test fixes
* fixed team policy test
* removed duplicate test
This optimizes the query in particular for Domain Controllers, where
there are a number of users with no local accounts and don't need to be
searched for software.
See #4261
not set on the INSERT.
- OUT: Only sets the ID on the passed session and returns it. (`CreatedAt`, `AccessedAt`, are not set.)
New version:
```go
func (ds *Datastore) NewSession(ctx context.Context, userID uint, sessionKey string) (*fleet.Session, error) {
sqlStatement := `
INSERT INTO sessions (
user_id,
` + "`key`" + `
)
VALUES(?,?)
`
result, err := ds.writer.ExecContext(ctx, sqlStatement, userID, sessionKey)
if err != nil {
return nil, ctxerr.Wrap(ctx, err, "inserting session")
}
id, _ := result.LastInsertId() // cannot fail with the mysql driver
return ds.sessionByID(ctx, ds.writer, uint(id))
}
```
- IN: Define arguments that are truly used when creating a session.
- OUT: Load and return the fleet.Session struct with all values set (using the `ds.writer` to support read replicas correctly).
PS: The new `NewSession` version mimics what we already do with other entities, like policies (`Datastore.NewGlobalPolicy`).
* Orbit: Add Fleet Desktop support to Windows
* Rename workflow, fix linux build
* Do not compile systray on linux
* nolint on unused
* Fix lint properly
* nolint both checkers
* Fix monitor logic in desktopRunner
* Fix interrupt and execute order
Renaming a temporary file to the final destination does not always work.
Specifically, if the source and destination paths are on different file
systems, you will get the following error
invalid cross-device link
Instead, create temporary file in the destination directory.
- Update CHANGELOG
- Bump versioning
- Tweak documentation
- Default `session_duration` to `5d`
- Add extra `#` to "Team policies" section so it doesn't show up in top level nav for docs
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Fleet Desktop Packaging WIP
* Ignore gosec warning
* Trigger on PR too
* Install Go in workflow
* Pass url parameter to desktop app
* Fix fleetctl package
* Final set of changes for v1 of Fleet Desktop
* Add changes
* PR fixes
* Fix CI build
* add larger menu bar icon
* Add transparency item
* Delete host_device_auth entry on host deletion
* Add SetTargetChannel
* Update white logo and add desktop to update runner
* Add fleet-desktop monitoring to orbit
* Define fleet-desktop app exec name
* Fix update runner creation
* Add API test before enabling the My device menu item
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
Add new usePlatformSelector custom hook
Add new usePlatformCompatibility custom hook
Add new PlatformSelector global component
Refactor PlatformCompatibility as global component
Refactor sql_tools to TypeScript
Improve type definitions for context/policy
Align PolicyPage and QueryPage with platform compatibility changes
* geoip wip
* return nil if ip is empty string or if ParseIP returns nil
* add ui component to render geolocation if available, address PR feedback
* render public ip if available
* add changes file, document geoip in deployment guide
* update rest-api docs
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Trigger workflow on PR
* Fixes to ease handling of artifact
* Fix, do not use target name as dir
* Remove workaround
* Error early if email already exists
* Update changes/issue-4361-mail-change-should-error
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
* Fix test
* Lint fixes
* Fix another test
* Also check for invites
* Improve error checks
* Update comment
* Update tests
* Fix test
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
* Add disable-updates flag to fleetctl and orbit
* Fix ruleguard execution error on make lint-go
* Introduce dev-mode for ease of development of orbit
* Add changes file
* Add CentOS parsing and post-processing in fleet
* Add tests and amend SyncCPEDatabase
* Add test for centosPostProcessing
* Changes from PR comments
* Amend software test
* Fix sync test
* Add index to source and vendor
* Use os.MkdirTemp
* Rearrange migrations
* Regenerate test schema
* Add support for testing migrations (#4112)
* Add support for testing migrations
* Rename migration in tests
* Changes suggested in PR
* Go mod tidy
* Fix OSVersion to work properly when distribution does not follow symver
Certain distros such as ClearLinux or ArchLinux do not use symver or any
dotted versioning scheme for their releases. Archlinux uses the static
string "Rolling" and ClearLinux uses a single build number such as 35550
for their versions.
In Fleet console, this shows up as a string like "Archlinux 0.0.0.0"
which makes very little sense to the user. This change makes it so that
if OSQuery cannot generate a dotted version number, we should instead
use the build id as an opaque string.
* Add /changes
* Add tests for os_version
* fixup
* fixup
* Add platform filters for MDM/Munki/Chrome queries
This should help quiet warnings that users/customers have reported when
these queries try to run on platforms without the macadmins extension
tables.
For #4123
* Improve documentation
* add changes file
* revert doc formatting
* Update tests
* Yet another test fix
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Add remaining policy templates
* Removeempty resolution field
* Update naming for standard policies
* Add enabled word to SIP policy
* Use full SIP
* Also change in constants
* Update windows disk encryption
* Add changes file
* Tweak windows disk encryption policy
* Address lint errors
* Make requested changes
* Reflect changes in policy tempaltes
* Make sure that standard policies and policy templates are the same
* Edit automatic login disabled description
* Also edit in constants
* Change webhooks db lock duration to one hour
* Reload interval and check time left
* Change interval to one hour to match the db lock
* Add missing continue in case of failure
* Fix lint-go
* Allow to search with target_sw with a period
* Remove tests for a different thign
* Add a test for programs as a source as well
* Use MATCH again and add proper tests
- Add new "Software" tab to host details page
- Add dropdown to filter vulnerable software
- Extend DataTable client-side filtering to accommodate filter inputs that are controlled by parent components
- Refactor host details software into separate component
* Add new software page
* Add hosts counts and last updated to software card on homepage
* Extend InfoCard component to all children to set dynamic title information and hyperlinks
* Add new TeamsDropdownHeader component
* Refactor TableContainer to move server-side pagination buttons to DataTable
* Extend DataTable to include footer element
* Better jitter (#3716) (#3744)
* Better jitter (#3716)
* Better jitter
* Fix lint
* Use milliseconds
* Make duration milliseconds
* Update based on Lucas' suggestion
* Add changes file
* Panic on error
* Fix compilation error
* Use time.after instead of time.tick to not leak
* Include browser extensions in software inventory (#3733) (#3757)
Use appropriate JOINs against users table to include all results.
For #3557
* Build Docker images on push to any branch (#3756) (#3758)
Allows for testing release branches and any others.
* Optimize users detail query (#3754) (#3770)
@Smjert reported instances of Windows Domain Controllers having massive
resource utilization and being killed by the watchdog when running this
query. In his test environment, this new query performs much better.
* Refactor async host processing to avoid redis SCAN keys (for policies) (#3657) (#3755)
* Issue 3707 clean targets on delete (#3739) (#3776)
* wip
* Delete targets when deleting teams, hosts, and labels
* Add changes file
* Fix error message
* Remove unused teamsTable
* Cleanup new pack
* Clean new packs at end of test
* Update operating system policy (#3779)
- Update policy's query to check for operating system versions greater than or equal to
* Improve loading of manage policies page (#3695) (#3781)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Fix logo (#3765)
* fixed duplicate query modal not closing (#3787)
* fixed style for connection line on activity feed (#3789)
* Improve loading state for query platform compatibility (#3752) (#3783)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Disable pagination for teams page (#3794)
* Shorten "simple" query API period to 25s (#3775) (#3804)
This helps the period stay under the default request timeouts for most
load balancers.
Some default timeouts:
* AWS ALB - 60s
* Nginx - 60s
* GCP LB - 30s
* Clear user checkbox selections after successfully removing selected row from table UI #3798 (#3801)
* Alias fleetctl sandbox to fleetctl preview (#3803) (#3806)
Preparing for some work the product team is planning to rebrand preview
to sandbox.
* Remove flash of spinner to no spinner (#3799)
* Revert "Better jitter (#3716) (#3744)" (#3807)
This reverts commit c857b52d2b.
* Fix teams dropdown sizing (#3759) (#3810)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* fixed link for expiration message (#3788)
* Fixed empty string for compatible platforms (#3812)
* fixed empty string for compatible platforms
* default checked for platforms on policy query
* Block autocomplete for username and password in SMTP settings (#3732) (#3816)
* No hosts for software/policy renders better message (#3701) (#3821)
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
* Get failing policies webhook data on page load (#3824)
* Fix reset password flow (#3826)
* Prepare for 4.9.0 (#3820)
- Add CHANGELOG
- Bump versioning
- Remove changes files
- Spelling fix for operating system policy's description
* Remove merge artifact
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Luke Heath <luke@fleetdm.com>
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
* Add sentry
* Fix gosum
* More gosum fixes
* Add missing def for config
* Enrich sentry scope a bit
* Add changes file
* Add goroutine safe scope to errors
* Encapsulate sentry logic
* Add documentation for new flag
* Add sentry capturing to crons and other background tasks
* Only send to sentry when enabled
This helps the period stay under the default request timeouts for most
load balancers.
Some default timeouts:
* AWS ALB - 60s
* Nginx - 60s
* GCP LB - 30s
* Allow sort by more than one key
* more accurate message for no hosts on filter
* added changes file
* lint fix
* more accurate message for no hosts on filter
* added changes file
* lint fix
* small logical change
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* wip
* Delete targets when deleting teams, hosts, and labels
* Add changes file
* Fix error message
* Remove unused teamsTable
* Cleanup new pack
* Clean new packs at end of test
@Smjert reported instances of Windows Domain Controllers having massive
resource utilization and being killed by the watchdog when running this
query. In his test environment, this new query performs much better.
* WIP
* Amend tests
* Do not load aggregated stats for packs
* Add option to host lite
* Fix remaining TODOs
* Fix osquery_utils tests
* Fix SQL
* Fix SQL (bis)
* Restore AuthenticateHost to load once
* Code improvements and re-add deferred host save
* More fixes to the PR
* Wrap users table update on tx
* Add caching to ListPacksForHost and ListScheduledQueriesInPack
* Remove SaveHostSoftware (replaced by UpdateHostSoftware)
* Add unit tests for new functionality
* Add changes file
* Fix scheduled queries test
* Remove cleanups and fk for host ids
* Readd missing things to the schema
* Remove unused
* Add changes file and fix some error messages
* Fix test
* Use tx instead of plain writer
* Other fixes
* More not found test fixes
* Go back to getcontext
* Add fixes for running tests with mysql:8
* Add getServer function
* Test github matrix
* Add changes file for the user facing fix
* Remove unused mysql8 docker-compose
* Fix orbit and osqueryd logging on Windows
* Add functionality to test the change and use systemprofile app data
* Add centos syslog to README.md
* Add wait on orbit for osquery extension socket to exist (#3571)
* Wait for osquery extension socket to exist
* Amend changes
* Fix lint
* Restore timeout
* Add webhook to app config
* Add redis failing policies set and webhook
* Add basic webhook test
* Store hostname in redis
* Global policy deletion to remove policy ID from set and config
* Also process new passing policies
* Fix unit test
* Sort hosts
* Add more tests
* Add ListSets to the failing policies interface
* Fix server URL and garbage collect on the triggering side
* Do not use Redis SCAN
* Fix Redis operation order
* Add API changes to doc
* Add comments
* Add more tests
* Fix tests
* Add tests for config update upon deletion of policies
* Run make dump-test-schema
* Ignore policies that failed to run
* Add proper unit tests to trigger logic
* Fix comments
* WIP
* Add tests to service_osquerty_test.go
* Use SSCAN for listing hosts instead of SMEMBERS
* Add failing policies to docs/01-Using-Fleet/configuration-files/README.md
* Remove skip
* Fix PR comments
* Finish first draft of API versions
* wip
* Finalize tests
* Revert change in handler
* Remove made up version
* Update versioning with aliases
* Add changes file
* Address review comments
* Revert overupdated routes
* Expand life time of deprecated APIs
* Fix test
* Comment out problematic part of test
* Revert bad path changes
* Display count is now correct, bug with 0 being falsy
* Empty component renders for client side search even if data table 'disappears'
* Hide display count if display count is 0 includes for client side search
* Hide search and dropdown if no queries have been created
* Remove tx from SaveHost
Also change REPLACE for software for an insert.
* Remove bundle identifier from the get since it's not part of the key
* Revert unneeded change
* Add software count API
* Fix makefile
* Fine no mock generating at this point
* Actually, one last try
* Use go install instead
* Fix go sum/mod
* Improve documentation
* Try setting node to 14
* Do caching of app config per instance instead of across all of them in redis
* Add changes file
* Simplify code based on review comment
* Use go-cache instead of creating our own
* Dont export consts
* Copy app config before returning it
* Fix lint
* Update go sum
* Update go sum
* Add fleetctl command to get missing migrations
* Fix copy paste and lint
* Detect migrations applied out of order
* Add extra bullet to changes
* Trigger creation of migration status tables
* Fix unit tests
* PR fixes
* PR comment fixes
- Orbit automatically loads the flagfile when it exists in the orbit
root.
- Add packaging support to include flagfile with package.
- Fix a panic when osquery fails to start up.
* Add host count to software API
* Update docs
* Update fleetctl tests to account for host counts
* Update docs to mention host_count special case
* Update func comment
* Ignore admin forced password reset if sso is enabled
* Add tests for authenticatedUser
* If SSOEnabled is true, then set AdminForcedPasswordReset to false
* Update tests to cover new password reset and sso flow
* Add changes file
* Serialize hosts writes per instance
* Write hosts asynchronously
* Dont make the save in a goroutine
* Revert "Dont make the save in a goroutine"
This reverts commit 4a890c5271.
* Make all savehosts async
* Address review comments and make this approach configurable
* Address review comments
* Disable bulk seen time marking for a test
* Move host seen times to a new table
* Remove unused
* Add seen_time to list hosts
* Add some jitter to seen time flushing
* Remove unused
* Add timeout to deferred save host
* Add tests for serialSaveHost
* Update hosts in labels and policy executions in a serial way
* Address review comments and remove fk constraints in host software
* Make errCh buffered
* Add changes file
* Readd key
* Make preview work when run from scratch (no orbit running)
* Make vulnerability processing less RAM hungry
* Add changes file
* Only get the cpe list once
* Remove cache
* Try killing osquery as well as orbit and clear their db dir
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
- Fix Windows MSI generation by changing permissions (#2655).
- Refactor temp directory initialization.
- Use root user for Wine in WiX Docker container.
- Support .pkg packaging on Linux without dependencies (besides Docker)
- Add functionality to parse sql for platform compatibility based on osquery tables
- Surface query-platform compatibility in create/edit query form
- Surface query-platform compatibility as new column in manage queries table
- Add filter-by-platform feature to manage queries table
- Refactor manage queries page to useQuery and useContext patterns
* Make migrations compatible with GTID replication
Fixes an issue some deployments encountered when migrations used a
statement that is unsupported in GTID replication mode (#2462).
Local dev MySQL now enforces this consistency, so it should be easier to
maintain compatibility going forward.
* Update docker-compose formatting
* if exists
* wip
* wip
* wip
* Add performance stats
* Add docs and other self review fixes
* lint
* Update based on review comments
* Add quick cleanup first and then reset to 1hr
* Reduce the load in the test
* added global teams dropdown
* added teams dropdown
* created tabs wrapper component
* comment out software table for now
* lint fixes
* added changes file
* removed test data
* fixed permissions
* fixed scroll behavior for test with sticky header
* lint fixes
* Add new UI feature: filter hosts by software version
* Move specific CVE information to appear alongside aggregate hosts filtered by software version
rather than on individual host details page
* Relocate users table to below software inventory on host details page
* Update Context API to contain enrollSecret
* Update e2e tests with generate installer modal
* Edge case with empty Team showing Generate Installer CTA
* fixed flow for resetting current session
* ignoring downloads for windows because paths
* using correct shell for windows
* using correct shell for windows
* ignoring exception; we can't fake react context
* changes file
* must wait for flash to show
* lint fixes
* re-added secure package
* Start a fleetctl preview test
* Add tests for fleetctl preview
* Fix setting of fleetctl auth token in test
* Add fleet instance vulnerabilities config to response of GetAppConfig
* Add checks that fleetctl preview enables vulnerability detection
* Adjust doc for get config API response
* Add the include-server-config flag to fleetctl get config
* Update test now that some of the PRs have been merged
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
- Use `TRUNCATE TABLE` rather than `DELETE FROM` for improved performance.
- Move DDL statement after truncate to avoid issues with retries (due to
column already being created).
#2360
* Ignore empty host users or software inventory
* Only store additional if it's not nil
* Update label/policy updated at when we record the executions and skip saving host
* Update changes file
* Add new feature: team policies
* Continue work on team policies
* Continue work on team policies
* Continue team policies
* Revert accidental deletion
* Rename variables
* code refactored; working on runtime errors
* updated front end docs
* Update URLs from team to teams, add tests for policy auth
* Fix test
* Continue work on team policies
* Add permission checks
* mange hosts functional and cleaned up; typing
* improved label logic
* added try catch to awaits
* lint fixes
* frontend unit tests don't work for functional components
* test fix
* revert
* Address errors related to refetch on window focus
* Add loading error check
* Fix typos in loading error checks
* Guard against invariant condition in useEffect
* Update links and routes for team policies
* lint fixes
* Update frontend/pages/hosts/ManageHostsPage/helpers.ts
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Change inherited policies button, tooltip
* lint fixes
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
Add tooltip to surface bundle_identifier in software inventory table
Add new column to software inventory table to surface "last used" information
Add link from software inventory table to manage hosts page filtered by software id
Replace software vulnerabilities table with info banner directing users to manage hosts page filtered by software id where software-specific vulnerabilities will be displayed
Refactor SoftwareVulnerabilities.jsx using TypeScript
Add utility function for sorting string representations of dates and refactor semantics of existing sort functions
* Replace content editables with textboxes
* Change log
* removed lib
* fixed bugs; no error handling for empty name
* lint fixes
* fixed query test
* lint fixes
* corrected query test
* bug fix; save modal didn't open for new query
Co-authored-by: Martavis Parker <martavis@auraticdevelopment.com>
* refactored state mgmt for query console
* changes files
* lint fixes
* teams e2e test were flaky
* lint fixes
* fixed bug for observer running query, run again button
* fixed inconsistent query body change
* disabled refetch on window focus
* added ability to delete selected sql text
* refactored state mgmt for query console
* changes files
* lint fixes
* teams e2e test were flaky
* lint fixes
* fixed bug for observer running query, run again button
* fixed inconsistent query body change
* disabled refetch on window focus
* Cache app config in redis
* Add changes files
* Replace string with constant
* Revert some test refactorign and duplicate a bit of test code
* Add test for AppConfig with redis failing
* Fix lint
* Use Doer so it works better in clusters
* Skip unmarshalling if we already did
* Allow to cache hosts if configured
* Omit the setting if empty
* Remove hashing, too much CPU
* Revert caching of host auth... needs a more thought through approach
* Remove config
* Remove old config
* Remove locker interface
* Fix test and address review comments
* Add max jitter percent config
* Fix jitter calc
* Remove comment
* Reduce test jitter to make tests less flaky
* Remove jitter entirely
* Document new config
* Fix doc link
* fixed style for sidebar
* changes file
* Reduce minimum width of body-wrap to prevent right sidebar from overflowing min app width of 1200px
Co-authored-by: Noah Talerman <noahtal@umich.edu>
* Add team policies
* Add team policy documentation
* Add changes file
* Update titles
* Fix lint
* Rewrite TeamAuthorize for more clarify
* Explicitly use two slices for clarity
* Simplify switch
* Implement fleetctl get software and the underlying API
* Add documentation
* Simplify list software implementation
* Lint fixes
* Make team name unique
* Address review comments
* Fix lint
* Fix tests
* Step 1 for improving query experience (#1591)
* fake change to create draft PR
* temp routes to work and not modify old query page
* created new API abstraction for query
* refactored App.jsx to prepare react-query
* fixed flow of redirects after page refresh; functional component added
* setup for getting data on edit
* implementing functions for query page
* Old form showing on new setup
* improving and breaking up query form
* no need for the helpers anymore; clean up
* added type for button component variant
* step toward new save modal; have to switch gears to #1619
* creating new query works
* clean up
* linting cleanup
* added default value for new query
* will address dynamic save disabled in edit step
* Step 2 for improving query experience (select targets) (#1732)
* fake change to create draft PR
* temp routes to work and not modify old query page
* created new API abstraction for query
* refactored App.jsx to prepare react-query
* fixed flow of redirects after page refresh; functional component added
* setup for getting data on edit
* implementing functions for query page
* Old form showing on new setup
* improving and breaking up query form
* no need for the helpers anymore; clean up
* added type for button component variant
* step toward new save modal; have to switch gears to #1619
* creating new query works
* clean up
* linting cleanup
* added default value for new query
* split steps into separate files for readability
* components laid out
* new targets picker
* function clean up
* styling tables
* fixing logic
* fixed logic to keep getting related hosts
* formatting targets for API
* fixed default query
* clean up
* styled target selectors; fixed target input styles
* began total count
* forgot to remove debugging code
* lint fixes
* added target count from API
* clean up
* able to remove selected host targets from table
* lint fixes
* Improving query experience - Step 3 (query results) (#1766)
* fake change to create draft PR
* temp routes to work and not modify old query page
* created new API abstraction for query
* refactored App.jsx to prepare react-query
* fixed flow of redirects after page refresh; functional component added
* setup for getting data on edit
* implementing functions for query page
* Old form showing on new setup
* improving and breaking up query form
* no need for the helpers anymore; clean up
* added type for button component variant
* step toward new save modal; have to switch gears to #1619
* creating new query works
* clean up
* linting cleanup
* added default value for new query
* split steps into separate files for readability
* components laid out
* new targets picker
* function clean up
* styling tables
* fixing logic
* fixed logic to keep getting related hosts
* formatting targets for API
* fixed default query
* clean up
* styled target selectors; fixed target input styles
* began total count
* forgot to remove debugging code
* lint fixes
* added target count from API
* clean up
* able to remove selected host targets from table
* lint fixes
* connected run query with modern React/JS; clean up
* linting fixes
* fixed logic to retrieve results from live query
* linting fixes
* created new, simpler query progress
* populating results and errors tables as expected
* syntax fixes
* fixing styles for query results
* more styling for query results
* manual merge from main
* Rename core->free and basic->premium
* Fix lint js
* Comment out portion of test that seems to timeout
* Rename tier to premium if basic is still loaded
* go sum
* Query Experience Cleanup Tasks (#1807)
* fixes to get merged main branch to build and work
* moved screens for query pages; clean up
* updated and typed react ace for query form; clean up
* using console error instead
* added real types instead of `any` except for errors
* query side panel ts and functional. prep for close task.
* ability to hide, show query table sidebar
* improved live query status warning
* added loading and error state for targets search
* error screen for targets; improved loading display
* now using API-created label for all linux
* missed some files on previous commit
* able to edit query
* clean up
* lint fixes
* query results showing as they come
* remove unused code
* removed old query page. major file cleanup.
* removed selectedTargets redux implementation
* removed unused redux actions and reducers
* removed unused keys in initial state
* selectedOsqueryTable is now using context API
* removed all querypages redux code
* set up context for app and user
* fixed auth with temp fix for wrapper
* completed redux removal from query page
* fixed var names coming from main branch
* fixed var name changes coming from issue 1501
* fixed save popup bug; clean up
* added permissions
* fixed login redirect
* removed unused props
* linting fix
* clean up
* removed unused component, refactor, and clean up
* fixed styles for step 1 as admin
* fixed styles for step 1 as observer
* fixed percentage of online hosts
* added loading progress to query stop button
* reset query status on run again
* added download icon to export button text
* fixed error reset on name input; fixed styles
* fixed bug where query value wasn't saving
* fixed query value when blank
* fixed bug - default query was running every time
* auto adding host from url to targets
* fixed flows for repeating run and save steps
* fleet ace is now TS and functional
* fixed a couple of tests
* fixed issues with query value text inconsistencies
* fixed query side panel not showing
* hiding error count if not > 0
* fixed showing editor for different roles
* using integer for targets
* go sum
* fixed targets param
* catching all errors while running query
* fixed hover state for title and description
* ignore unit test for now; lint fixes
* locking react-ace version
* ignoring tests breaking in github actions
* brought tests back
* fixing file name
* fixing file name again
* fixed e2e test
* have to ignore tests for now
* ignore certain premium tests for now
* one last test to revamp
* another test
* fixed teamflow test
* fixed observer query 403
* lint fixes
* fixed maintainer test
* added changes file
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Skeleton UI
* Rebase to main
* Work towards config API request modification
* Nest and unnest when formatting for server and frontend
* Changelog
* Add validation to UI, fix ? vertical spacing
* Rebase e2e
* 1 of 2 passing unit tests for config
* Update REST-API.md to include webhook_settings
* Destructure / flatten config webhook in unit test
* Merge advance options e2e conflict
* x and y example not x and x
* Fix observer e2e
* Add new data to read only example request
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Martavis Parker <martavis@auraticdevelopment.com>
* Add extra debug logging for hosts
* Add changes file
* Ignore if appconfig is nil
* Use slice of uints instead of a string
* Debug response request for enabled hosts
* Add host-id to request/response
* Lint fixes
* Add missing AppConfigFuncs
* added package command from orbit as fleetctl command
* update deployment docs
* add changes file
* added tests for package command, run go mod tidy & go mod verify
* validate that package files exist
* comment out msi packaging test until we can investigate github runner permission issues
* Rename core->free and basic->premium
* Fix lint js
* Comment out portion of test that seems to timeout
* Rename tier to premium if basic is still loaded
* fix get hosts command to properly output json/yaml based on command line flag
* add changes file
* added tests for get hosts when specifiying host
* added additional hosts to be returned in test cases
* go fmt
* Create vuln path if possible
* Make sure we skip creation if static instance is selected
* Document behavior
* Fix return in crons and test without sleeps
* wip
* Add tests and finish implementation
* Add proper default for periodicity, changes file, and documentation
* Fix tests and add defaults also to new installs
* EnableHostUsers should be true if undefined as well
* In some cases, periodicity can be zero because of the migrations
* Apply defaults when migrating appconfig
* Fix lint
* lint
* Address review comments
* Add global policies
* Update documentation and add extra parameter to config
* Fix failing tests
* Store historic policy records
* Address review comments
And also remove other inmem references I saw by chance
* Add documentation for get by id request
* Add parameter doc
* Move schema generation to a cmd instead of a test
Otherwise it messes up running all tests sometimes depending on how parallel it does
* Remove brain dump for another task
* Make migration tests a separate beast
* Make schema generation idempotent and move dbutils cmd to tools
* Allow all filters and add counts to Policy
* Add test for Policy
* Make team schedule names more user friendly and hide them from host pack stats
* Delete test for a different bug and add migration
* Update name template
* Use GROUP_CONCAT instead of JSON_ARRAYAGG
* Update server/datastore/mysql/software.go
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
- Uses Context API to track state
- resetSelectedRows default to false and only momentarily becomes true upon successful notification
Several other ideas were explored and ruled out including: asyncDebounce (impossible approach), applying state throughout the app (not concise nor maintainable), modifications directly to DataTable only (react-table did not have the right key combinations to reset selected rows manually when we needed).
Closes#1540
Co-authored by: @martavis
* Add All Linux label
* Change name to Linux instead of All Linux to see if e2e likes it better
* Revert "Change name to Linux instead of All Linux to see if e2e likes it better"
This reverts commit 26b79f214e.
* Fix all linux label insert
* Style all settings side panels
* Add builtin label icons
* Update tests aligning jest userStub and adminUserStub
* Update tests adding cypress checks for user teams/roles
* Make receive calls to redis conn thread safe
Also removes REDIS_TEST env var. Redis is lightweight and fast, no need
to skip these tests.
* No need to increase the wait
* add team_id filter to fleetctl via get hosts --team flag & api via api/v1/fleet/hosts and api/v1/fleet/labels/id/hosts
* update tests & add changes file
* Dont delete pack stats before inserting new ones to prevent deadlocks
* Remove fk for scheduled_query_stats
* Remove fk removal
* Fix tests
* Remove unneeded comment
* modify packs api to filter non-empty pack_type from response
* change list packs store method to allow filtering for "system-level" packs, by default the api filters these packs from being returned
* add changes file
* don't allow modifications to global or team packs via apply spec
* refactor to use PackListOptions
* #1372 created dropdown for status
* #1372 fixed default state for dropdown
* #1372 added help text and styling
* clean up
* fixed linting
* created changes log
* fixed e2e test
* created new header
* clean up
* added logic to edit and delete label using icons
* reworked selectedFilter to support status & label
* fixed multiple params in url
* comment clean up
* fixed tests
* linting fixes
* fixed height of status dropdown
* bug fix for selecting status 1st, label 2nd
* fixed e2e test
* minor style fix for side panel label scroll
* fixed label e2e test
* removed SQL editor for label selection
* removed edit and delete for platform labels
* fixed bugs loading hosts for every label click
* fixed linting
* fixed basic e2e test
* fixed observer basic e2e test
* modified changes file
* fixed bug with label replacement logic for url
* Expose vulnerabilities data in host software data
* Gather cves and software in one query
* Expand the test to cover all cases
* Make test less flaky
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Add first cve db generation
* WIP but with final strategy, preparring to merge main
* Fix merge conflicts
* WIP
* wip
* Insert CVEs to the db
* Remove unused code
* Use wg instead of counting
* Call cancelFunc to avoid ctx leak
* Fix logs for better readability
* Point code to fleetdm instead of my repo
* Don't return errors in distributed query ingestion, just log them
* Allow for multiple errors in the logging context
* Update check when loading host
* Log multiple errors and add tests for other changes
* Add missing host func
* Add another missing host func
* Add changes file
* Add basic idea
* Implement the new logging strategy everywhere
* Remove unused const
* Add tests and fix error cases
* Fix logging in osquery service
* If there are extras, log info unless force debug
* Change to info
* Fix test
* Make logging context more chainable and force info for sessions
* fleetctl preview downloads standard query library and populates console
* add change log file, use specGroupFromBytes to parse standard query library file
- Remove changes files that were collected during the 4.1.0 release cycle
- Add removing changes files step to release process documentation
- Update support process docs to reflect the change to discussing the last on-call rotation at 🍽️ Daily standup meeting
* update printHost to serialize HostResponse rather than just Host
* added change log file
* Update changes/issue-1373-add-status-to-fleetctl-get-command
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* WIP
* Send usage analytics
* Improve loggin of cron tasks and fix test
* Implement appconfig method now that we are checking that as well
* Address review comments
* Add host users
* Add changes file and test removing pull_request from the on test
* Remove users and store the removal timestamp
* Improve test yml to allow for PRs from forks
* Make roles for users mandatory
* Remove nop migration
* Add missing test for wrong role
* Properly validate global and team roles
* Address codacy issues
* Address codacy review
* No need to check for nil
- Update create/edit user modal to accurately reflect if SSO is enabled for the individual user (rather than if enabled for the organization as a whole)
- Update create/edit user modal tooltips and disabled fields based on state of organization settings (SSO and SMTP)
* First approach to diff
* Refactor things for better readability and testing
* Remove draft comment for algorithm
* Format things a bit better
* Remove unused and simplify code a bit
* Refactor for readability and testing
* Add changes file
* Implement new approach based on review comments
* Make sure to only delete from the current host
* Add single uninstall test and fix code
* Improve code based on review
* Refactor error handling for better extensibility and add more scaffolding for specific db errors
* Add integration tests to check errors from mysql are translated properly
* Address review comments
* Add changes file
