Commit graph

304 commits

Author SHA1 Message Date
Lucas Manuel Rodriguez
81672ee50e
Add dev sql interceptor for logging of queries (#3815)
* Add dev sql interceptor for logging of queries

* Remove extra spaces and tabs from queries

* Make regex global
2022-01-21 14:28:21 -03:00
Tomas Touceda
9d572309ae
Add sentry (#3669)
* Add sentry

* Fix gosum

* More gosum fixes

* Add missing def for config

* Enrich sentry scope a bit

* Add changes file

* Add goroutine safe scope to errors

* Encapsulate sentry logic

* Add documentation for new flag

* Add sentry capturing to crons and other background tasks

* Only send to sentry when enabled
2022-01-20 16:41:02 -03:00
Zach Wasserman
d55108fcbf
Alias fleetctl sandbox to fleetctl preview (#3803)
Preparing for some work the product team is planning to rebrand preview
to sandbox.
2022-01-19 22:12:28 -08:00
Tomas Touceda
b47cf3d2d4
Better jitter (#3716)
* Better jitter

* Fix lint

* Use milliseconds

* Make duration milliseconds

* Update based on Lucas' suggestion

* Add changes file

* Panic on error

* Fix compilation error
2022-01-18 11:29:57 -03:00
Martin Angers
1f185a7a8b
Refactor async host processing to avoid redis SCAN keys (for labels only) (#3639) 2022-01-17 14:53:59 -05:00
Lucas Manuel Rodriguez
61b21df79e
Add flags to fleetctl preview to set custom orbit and osqueryd update channels (#3683)
* Add channel options for orbit update on fleetctl preview

* Add changes

* Also set channels on orbit execution in preview
2022-01-14 18:25:26 -03:00
gillespi314
6952653e1b
Fix usage stats to send missing numLabels field (#3606) 2022-01-13 11:11:13 -06:00
Tomas Touceda
4bed4757fb
Remove cleanups and fk for host ids (#3607)
* Remove cleanups and fk for host ids

* Readd missing things to the schema

* Remove unused

* Add changes file and fix some error messages

* Fix test

* Use tx instead of plain writer

* Other fixes

* More not found test fixes

* Go back to getcontext
2022-01-12 14:07:51 -03:00
Martin Angers
f14f97156c
Add read and write timeout options for redis (#3624) 2022-01-11 17:08:39 -05:00
Tomas Touceda
c662cd2b53
Add policies yaml (#3464)
* Add policies yaml

* Add documentation and address review comments

* Amend documentation
2022-01-11 11:04:29 -03:00
Zach Wasserman
8929917ee9
Update development license (#3562) 2022-01-03 15:35:53 -03:00
Zach Wasserman
4a3de40a94
Add no-hosts option to fleetctl preview (#3544)
Intended mostly for testing, this is documented only in the Fleet
handbook.
2021-12-31 15:13:28 -08:00
Lucas Manuel Rodriguez
2f4ecb1b6b
fleetctl package command to check for PEM file (#3375)
#3374
2021-12-29 17:32:55 -08:00
Lucas Manuel Rodriguez
787944482b
Global policies automation webhooks (#3378)
* Add webhook to app config

* Add redis failing policies set and webhook

* Add basic webhook test

* Store hostname in redis

* Global policy deletion to remove policy ID from set and config

* Also process new passing policies

* Fix unit test

* Sort hosts

* Add more tests

* Add ListSets to the failing policies interface

* Fix server URL and garbage collect on the triggering side

* Do not use Redis SCAN

* Fix Redis operation order

* Add API changes to doc

* Add comments

* Add more tests

* Fix tests

* Add tests for config update upon deletion of policies

* Run make dump-test-schema

* Ignore policies that failed to run

* Add proper unit tests to trigger logic

* Fix comments

* WIP

* Add tests to service_osquerty_test.go

* Use SSCAN for listing hosts instead of SMEMBERS

* Add failing policies to docs/01-Using-Fleet/configuration-files/README.md

* Remove skip

* Fix PR comments
2021-12-23 18:26:55 -03:00
Lucas Manuel Rodriguez
5dc93a8c7a
Fix TestPackage version (#3484) 2021-12-23 15:13:38 -03:00
Zach Wasserman
51b19b445f
Prepare for Orbit 0.0.5 release (#3478) 2021-12-22 16:02:26 -08:00
Martin Angers
1e2059585a
Add support for google chrome profiles (#3423) 2021-12-21 15:36:19 -05:00
Tomas Touceda
39e71c6d77
Add mdm and munki (#3406)
* Draft for mdm and munki as direct ingest funcs

* Expose mdm/munki over API

* Add test for service and fix bug

* Update queries test

* Fix lint

* Address review comments
2021-12-21 09:37:58 -03:00
Martin Angers
e988d16eb3
Update the prometheus go client library (#3140) 2021-12-20 09:20:58 -05:00
Martin Angers
4143a37056
Fix redis scan keys issue for live queries (#3107) 2021-12-14 16:30:26 -05:00
Zach Wasserman
a35ddfd4d8
Increase session duration with --dev flag (#3339) 2021-12-12 17:31:07 -08:00
Lucas Manuel Rodriguez
4213ddb141
Rename missing policy platforms to platform (#3334) 2021-12-10 13:55:49 -03:00
Lucas Manuel Rodriguez
fac83f6b11
Add unique host users to osquery-perf (#3255)
* Add unique software option to osquery-perf

* Add unique host users

Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2021-12-09 18:24:48 -03:00
Lucas Manuel Rodriguez
b9c65abc6a
Add unique software option to osquery-perf (#3253)
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2021-12-09 18:05:32 -03:00
Tomas Touceda
4df7093395
Osquery perf add scheduled query stats (#3252)
* Add scheduled queries to osquery perf

* Remove log line

* Update parsing of packs to be more readable

* Remove log
2021-12-09 17:20:32 -03:00
Lucas Manuel Rodriguez
d0765cb9ee
Do not exit on serve/prepare if there are unknown migrations (#3262)
* Do not exit on serve/prepare if there are unknown migrations

* PR review changes
2021-12-08 19:50:00 -03:00
gillespi314
6ee6dba4a4
Update anonymous usage statistics to record Fleet feature adoption (#3056)
* Collect additional anonymous usage statistics

* Update fleetdm.com and docs for usage stats

* Fix lint-go errors

* Fix failing test

* update webhook inputs

* Update receive-usage-analytics.js

* fix failing test

* Update statistics_test.go

Co-authored-by: eashaw <eashaw@sailsjs.com>
2021-12-06 14:39:00 -06:00
Lucas Manuel Rodriguez
c5375263b6
Change policy platforms name to platform singular (for consistency) (#3221)
* Change policy platforms name to platform singular

* Fix fleetctl tests
2021-12-06 13:56:28 -03:00
Martin Angers
b84ef79bf4
Add the fleetctl debug {errors,db-locks} sub-commands (#3168) 2021-12-06 09:26:01 -05:00
Lucas Manuel Rodriguez
b9a408704e
Add platforms field to policies (#3181)
* Add platforms field to policies

* Fix fleetctl tests

* PR review changes

* Add missing tests

* Add changes for ListPoliciesForHost
2021-12-03 15:33:33 -03:00
Tomas Touceda
142006cbdd
Issue 2099 enable vulnerability processing new installs (#3104)
* Enable vulnerability processing by default

* Add changes file

* Remove wrong env var

* Fix import

* Fix tests

* Update log line

* Remove unneeded import
2021-12-03 14:44:57 -03:00
Tomas Touceda
a36e029d95
App config caching per instance (#3071)
* Do caching of app config per instance instead of across all of them in redis

* Add changes file

* Simplify code based on review comment

* Use go-cache instead of creating our own

* Dont export consts

* Copy app config before returning it

* Fix lint

* Update go sum

* Update go sum
2021-11-29 12:51:57 -03:00
Martin Angers
c997f853e5
Make creation of http.Client uniform across the codebase (#3097) 2021-11-24 15:56:54 -05:00
Lucas Manuel Rodriguez
964f85b174
Amend policy creation and spec (for proprietary query), and add update APIs (#2890)
* Amend policy creation (proprietary query), add update APIs

* Fix Datastore.SavePolicy bug (and add tests)

* Add integration tests for new policy APIs

* Add author email

* Add activities

* Push breaking changes for return policy fields

* WIP

* Add integration test for host policies

* Make more improvements to policy representation

* Improve upgrade code (from PR review comments)

* PR changes

* Revert activities for policies

* Use *uint instead of uint for queryID, use fleet.PolicyPayload

* Filter out other schemas

* New policy flow (#2922)

* created new policy flow -- no API connection

* added api props

* fixed prop name

* lint fixes

* removed unused modal; fixed style

* name, desc icons; created global components

* lint fixes

* ignoring certain files and lines for prettier

* Update frontend/pages/policies/PolicyPage/PolicyPage.tsx

* Make policy names unique across deployment

* Amend upgrade script

* Fix migration for unique names

* Do not deduplicate but instead rename policies

Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
2021-11-24 14:16:42 -03:00
Tomas Touceda
c65f8507fe
Remove host counts from software (#3082)
* Remove host counts from software

* Actually remove the host count from the struct

* Fix get test
2021-11-23 15:50:51 -03:00
Lucas Manuel Rodriguez
028c2643f7
Fleet serve to exit if migrations are missing (default) (#2803)
* Add option to fleet to exit if migrations are missing

* Reverse serve exit migrations logic

* Fix typo
2021-11-22 14:47:24 -03:00
Lucas Manuel Rodriguez
0642bfdb1d
Add fleetctl debug migrations command to get missing migrations (#2967)
* Add fleetctl command to get missing migrations

* Fix copy paste and lint

* Detect migrations applied out of order

* Add extra bullet to changes

* Trigger creation of migration status tables

* Fix unit tests

* PR fixes

* PR comment fixes
2021-11-22 14:06:12 -03:00
Tomas Touceda
4cb0060070
Prettify and compare strings instead of structs in fleetctl get tests (#3047)
* Prettify and compare strings instead of structs in fleetctl get tests

* Prettify JSON testdata

* Simplify scanner for multi json
2021-11-22 12:20:09 -03:00
Martin Angers
69a4985cac
Use new error handling approach in other packages (#2954) 2021-11-22 09:13:26 -05:00
Lucas Manuel Rodriguez
59e01fbe08
Make fleetctl apply -f fail with unknown kind: config fields (#3026)
* Make fleetctl apply fail with unknown fields

* Add unit test
2021-11-19 12:37:36 -03:00
Lucas Manuel Rodriguez
d1dcc0e78a
Add probability of policy pass flag to osquery-perf (#3014) 2021-11-19 08:50:25 -03:00
Zach Wasserman
b65c61cb12
Change Orbit osquery to Fleet osquery in generated packages (#3034)
Use "Fleet osquery" rather than "Orbit osquery" in packages generated
 by `fleetctl package`.
2021-11-18 17:43:52 -08:00
Zach Wasserman
3811a63daf
Add osquery flagfile support in Orbit (#3006)
- Orbit automatically loads the flagfile when it exists in the orbit
  root.
- Add packaging support to include flagfile with package.
- Fix a panic when osquery fails to start up.
2021-11-18 15:06:33 -08:00
Lucas Manuel Rodriguez
5c1edaf527
Kill existing instances of orbit and osqueryd in fleetctl preview (#2921)
* Kill existing instances of orbit and osqueryd in fleetctl preview

* Add changes file
2021-11-15 18:12:25 -03:00
Martin Angers
3ae57a0242
Create errors with ctxerr, add the call to store them in redis (#2786) 2021-11-15 09:11:38 -05:00
Martin Angers
348841c90b
Add explanation output to fleetctl package (#2864) 2021-11-15 08:40:58 -05:00
Tomas Touceda
b802af6f44
Add host count to software API (#2879)
* Add host count to software API

* Update docs

* Update fleetctl tests to account for host counts

* Update docs to mention host_count special case

* Update func comment
2021-11-11 08:49:17 -03:00
Tomas Touceda
a16b98a4a1
Add logging of interval for webhooks (#2853) 2021-11-09 11:59:16 -03:00
Tomas Touceda
7db6de7278
Serialize hosts writes per instance (#2753)
* Serialize hosts writes per instance

* Write hosts asynchronously

* Dont make the save in a goroutine

* Revert "Dont make the save in a goroutine"

This reverts commit 4a890c5271.

* Make all savehosts async

* Address review comments and make this approach configurable

* Address review comments

* Disable bulk seen time marking for a test

* Move host seen times to a new table

* Remove unused

* Add seen_time to list hosts

* Add some jitter to seen time flushing

* Remove unused

* Add timeout to deferred save host

* Add tests for serialSaveHost

* Update hosts in labels and policy executions in a serial way

* Address review comments and remove fk constraints in host software

* Make errCh buffered

* Add changes file

* Readd key
2021-11-08 11:42:37 -03:00
Tomas Touceda
2902da76ca
Issue 2027 better error visibility (#2069) 2021-11-02 13:35:57 -04:00
Lucas Manuel Rodriguez
43d1a8c9a0
Fix races in go tests and run with -race on CI (#2712)
* Fix races in go tests and run with -race on CI

* Fix race in pubsub

* Increase timeout to 15m for go tests

* CI takes forever, try disabling race

* Remove timeout from go tests
2021-11-01 15:31:01 -03:00
Lucas Manuel Rodriguez
8642bb785e
Add MVP support for software inventory to osquery-perf for load testing (#2751)
* Add MVP support for software inventory to osquery-perf for load testing

* Fix test compile
2021-11-01 15:23:31 -03:00
Martin Angers
a8735d55bb
Implement async processing of hosts for label queries (#2288) 2021-11-01 14:13:16 -04:00
Lucas Manuel Rodriguez
384ce0f13d
Amend WriteTimeout for live queries GET sync API (#2752) 2021-11-01 09:31:56 -03:00
Tomas Touceda
fcb5d5b392
Make vulnerability processing less RAM hungry (#2739)
* Make preview work when run from scratch (no orbit running)

* Make vulnerability processing less RAM hungry

* Add changes file

* Only get the cpe list once

* Remove cache

* Try killing osquery as well as orbit and clear their db dir

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
2021-10-29 11:27:12 -03:00
Zach Wasserman
37e4452274
Open browser in fleetctl preview (#2730)
Attempts to automatically open the browser and log in.
2021-10-27 21:50:32 -07:00
Zach Wasserman
8dc34be3a3
Updates and fixes for packaging (#2682)
- Fix Windows MSI generation by changing permissions (#2655).
- Refactor temp directory initialization.
- Use root user for Wine in WiX Docker container.
- Support .pkg packaging on Linux without dependencies (besides Docker)
2021-10-27 16:17:41 -07:00
Tomas Touceda
21153e9400
Add sleep to give orbit a chance to win the id 1 for the host (#2723)
* Add sleep to give orbit a chance to win the id 1 for the host

* Wait until we have the current host
2021-10-27 17:41:03 -03:00
Tomas Touceda
a37024e3b1
Issue 2699 preview reset stops orbit (#2720)
* Stop orbit when running preview reset as well

* Detect if orbit is already running and dont start it again
2021-10-27 15:57:24 -03:00
Tomas Touceda
3369436741
Remove policies yaml for now and apply policies in preview (#2644) 2021-10-25 09:17:34 -03:00
gillespi314
6c58159fb8
Change default org name for fleet preview (#2642) 2021-10-22 17:35:01 -05:00
Lucas Manuel Rodriguez
54805b5b72
Perform early session check on fleetctl commands (#2620)
* Perform early session check on fleetctl

* Add fleetctl test for the early session check
2021-10-22 15:41:17 -03:00
Luke Heath
99673eb148
Fix broken links to docs (#2561) 2021-10-22 12:39:20 -05:00
Tomas Touceda
c2696c1261
Issue 1892 query perf everywhere (#2580)
* wip

* wip

* wip

* Add performance stats

* Add docs and other self review fixes

* lint

* Update based on review comments

* Add quick cleanup first and then reset to 1hr

* Reduce the load in the test
2021-10-20 18:35:38 -03:00
Tomas Touceda
cbe931c0e9
Add vulnerable filter for software and also wire up the query search (#2604)
* Add vulnerable filter for software and also wire up the query search

* Add documentation

* Update to use software list options
2021-10-20 18:01:20 -03:00
Tomas Touceda
c3f7577bd8
Add query description and resolution to the list of policies in hosts (#2584)
* Add query description and resolution to the list of policies in hosts

* Add docs

* Fix get host tests
2021-10-20 12:07:16 -03:00
Martin Angers
07cf136d03
Add TLS support to Redis connections (#2568) 2021-10-20 10:09:18 -04:00
Tomas Touceda
fe5660e006
Reimplement host expiration to not need mysql events (#2552)
* Reimplement host expiration to not need mysql events

* Update mocks
2021-10-19 17:47:37 -03:00
Martin Angers
a7420140ce
Use a redis cluster-friendly store for rate limit (#2577) 2021-10-19 16:24:13 -04:00
Zach Wasserman
1de7b8c4ca
Update osquery-perf readme (#2573) 2021-10-19 11:54:45 -07:00
Martin Angers
057d4e8b2e
Add configuration and support for Redis to read from replicas (#2509) 2021-10-18 09:32:17 -04:00
Tomas Touceda
b11c6ffe31
Download and launch orbit for the current device when starting preview (#2437)
* Download and launch orbit for the current device when starting preview

* Add windows kill

* Fix imports

* Fix import

* Disable secure directory checks on windows for now

* Add missing import
2021-10-15 18:23:06 -03:00
Tomas Touceda
89c6d18ed2
Issue 2293 reveal failing policies list host (#2528)
* Add host issues

* Update docs

* Remove comment

* Update tests
2021-10-15 07:34:30 -03:00
Tomas Touceda
d3a0d62902
Issue 2456 policies yaml (#2512)
* wip

* Add policy specs support

* Add documentation

* Make policy apply idempotent

* Fold in code

* Improve tests and simplify auth checks

* Lint and fix test
2021-10-15 07:34:11 -03:00
Tomas Touceda
825939e3dc
Improve osquery perf (#2503)
* Improve osquery perf

* Update dependencies

* Embed template so this can be run from the root of the repo

* Fix lint

* Address review comments
2021-10-14 10:09:58 -03:00
Martin Angers
e1afb3547e
Update preview tests (#2487) 2021-10-12 11:49:25 -04:00
Martin Angers
dcf701ef2b
Support fleetctl preview from a branch, default to production (#2476) 2021-10-12 10:20:28 -04:00
Martin Angers
3aef96e15c
Add test for fleetctl preview (#2388)
* Start a fleetctl preview test

* Add tests for fleetctl preview

* Fix setting of fleetctl auth token in test

* Add fleet instance vulnerabilities config to response of GetAppConfig

* Add checks that fleetctl preview enables vulnerability detection

* Adjust doc for get config API response

* Add the include-server-config flag to fleetctl get config

* Update test now that some of the PRs have been merged

Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2021-10-07 10:19:10 -03:00
Martin Angers
9a36fce082
Enable vulnerability detection in fleetctl preview (#2376) 2021-10-07 09:02:49 -03:00
Martin Angers
e4dae5f3d1
Disable anonymous analytics collection for fleetctl preview (#2372)
* Disable anonymous analytics collection for fleetctl preview

* Force run checks
2021-10-07 09:01:51 -03:00
Tomas Touceda
9b809a2d35
Issue 1890 host details policy (#2410)
* wip

* Add policies to hosts
2021-10-07 08:11:10 -03:00
Tomas Touceda
28c3e64e41
Expose additional queries in fleetctl get hosts (#2349) 2021-10-04 15:18:02 -03:00
Tomas Touceda
435178e93c
Add bundle identifier to software when available (#2220)
* Add bundle identifier to software when available

* Update docs

* Delete unneeded test
2021-09-28 18:13:34 -03:00
Tomas Touceda
685245c2bd
Cache app config in redis (#2205)
* Cache app config in redis

* Add changes files

* Replace string with constant

* Revert some test refactorign and duplicate a bit of test code

* Add test for AppConfig with redis failing

* Fix lint

* Use Doer so it works better in clusters

* Skip unmarshalling if we already did

* Allow to cache hosts if configured

* Omit the setting if empty

* Remove hashing, too much CPU

* Revert caching of host auth... needs a more thought through approach

* Remove config

* Remove old config

* Remove locker interface

* Fix test and address review comments
2021-09-28 10:01:53 -03:00
Tomas Touceda
2033d8208c
Add policy updated at (#2246)
* wip

* Add policy updated at interval and update the UI to use that

* Update rest api

* Fix tests
2021-09-27 16:27:38 -03:00
Tomas Touceda
8600d71d35
Add osquery perf (#2190)
* Add osquery perf

* Update dockerfile and gh action
2021-09-22 17:18:55 -03:00
Zach Wasserman
d9b2f4a6fb
Add --tag flag to fleetctl preview (#2171)
Allows specifying a version of the Fleet image to run.
2021-09-21 17:08:58 -07:00
Mike McNeil
8faea43990
fix fleetctl preview after standard query library yml moved (#2175)
* fixes https://github.com/fleetdm/fleet/issues/2172

* also fixes contribute link on fleetdm.com/queries
2021-09-21 18:43:27 -05:00
Tomas Touceda
4650484960
Remove fk label membership (#2157)
* Remove fk from label membership

* Add changes file

* Fix tests

* No need to IGNORE anymore
2021-09-21 11:48:20 -03:00
Martin Angers
86dce785ae
Improve performance of the Go test suite (#2060)
Closes #1805
2021-09-20 14:09:38 -04:00
Tomas Touceda
e6368cc57f
Refactor integration tests (#1821)
* Refactor integration tests

* Remove nopCloser and use io.NopCloser

* Address review comments
2021-09-15 16:27:53 -03:00
Martin Angers
1fa5ce16b8
Add configurable Redis connection retries and following of cluster redirections (#2045)
Closes #1969
2021-09-15 08:50:32 -04:00
Martin Angers
d956ca1bec
Use the context in the Datastore layer. (#2030) 2021-09-14 10:44:02 -04:00
Tomas Touceda
176037cad2
Implement fleetctl get software and the underlying API (#1999)
* Implement fleetctl get software and the underlying API

* Add documentation

* Simplify list software implementation

* Lint fixes

* Make team name unique

* Address review comments

* Fix lint

* Fix tests
2021-09-14 10:58:48 -03:00
Tomas Touceda
429af72b69
Issue 1963 vulnerabilities no sync (#1976)
* wip

* Add tests for skip sync

* Add changes file

* Fix lint
2021-09-14 10:58:35 -03:00
Martin Angers
4f4185372d
Add support for context in datastore/mysql layer (#1962)
This is just to pass down the context to the datastore layer, it doesn't
use it just yet - this will be in a follow-up PR.
2021-09-14 08:11:07 -04:00
keepwn
b42d213689
Support close the websocket of LiveQueryResultsHandler (#1006)
* Support close `LiveQueryResultsHandler`

* Start adding test

* Make LiveQuery exit when the context is Done

* Fix lint and remove debug print

* Update server/service/client_live_query.go

Co-authored-by: Zach Wasserman <zach@fleetdm.com>

* Revert "Update server/service/client_live_query.go"

This reverts commit be67ca1512.

Co-authored-by: Tomas Touceda <chiiph@gmail.com>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
2021-09-10 16:26:39 -03:00
Benjamin Edwards
823775844e
added package command from orbit as fleetctl command (#1802)
* added package command from orbit as fleetctl command

* update deployment docs

* add changes file

* added tests for package command, run go mod tidy & go mod verify

* validate that package files exist

* comment out msi packaging test until we can investigate github runner permission issues
2021-09-09 01:34:12 -04:00
Tomas Touceda
b2efc9f51c
Make redis conn timeout and keep alive configurable (#1968)
* Make redis conn timeout and keep alive configurable

* Document new configs

* Correct config name
2021-09-08 17:55:12 -03:00
Tomas Touceda
6d2ae02efd
Rename core->free and basic->premium (#1870)
* Rename core->free and basic->premium

* Fix lint js

* Comment out portion of test that seems to timeout

* Rename tier  to premium if basic is still loaded
2021-09-03 13:05:23 -03:00