Commit graph

98 commits

Author SHA1 Message Date
Lucas Manuel Rodriguez
f82c01833b
Change webhooks db lock duration to one hour (#3589)
* Change webhooks db lock duration to one hour

* Reload interval and check time left

* Change interval to one hour to match the db lock

* Add missing continue in case of failure

* Fix lint-go
2022-02-07 17:14:23 -03:00
Martin Angers
6e2ba62744
Trigger webhooks for recently published vulnerabilities (#3941) 2022-02-02 16:34:37 -05:00
Tomas Touceda
66976080ae
Just log the error if vulnerabilities fail (#3963)
* Just log the error if vulnerabilities fail

* Add explicit return for more secure future iterations
2022-02-01 14:41:48 -03:00
Tomas Touceda
ffabf803a3
Aggregate munki and mdm data (#3886)
* Aggregate munki and mdm data

* Update doc

* Use reader to read

* Reader to read

* Address review comments
2022-01-26 17:55:07 -03:00
Martin Angers
9a0f749641
Add hosts_count field to "list software" endpoint (#3873) 2022-01-26 09:47:56 -05:00
Lucas Manuel Rodriguez
81672ee50e
Add dev sql interceptor for logging of queries (#3815)
* Add dev sql interceptor for logging of queries

* Remove extra spaces and tabs from queries

* Make regex global
2022-01-21 14:28:21 -03:00
Tomas Touceda
9d572309ae
Add sentry (#3669)
* Add sentry

* Fix gosum

* More gosum fixes

* Add missing def for config

* Enrich sentry scope a bit

* Add changes file

* Add goroutine safe scope to errors

* Encapsulate sentry logic

* Add documentation for new flag

* Add sentry capturing to crons and other background tasks

* Only send to sentry when enabled
2022-01-20 16:41:02 -03:00
Tomas Touceda
b47cf3d2d4
Better jitter (#3716)
* Better jitter

* Fix lint

* Use milliseconds

* Make duration milliseconds

* Update based on Lucas' suggestion

* Add changes file

* Panic on error

* Fix compilation error
2022-01-18 11:29:57 -03:00
Martin Angers
1f185a7a8b
Refactor async host processing to avoid redis SCAN keys (for labels only) (#3639) 2022-01-17 14:53:59 -05:00
gillespi314
6952653e1b
Fix usage stats to send missing numLabels field (#3606) 2022-01-13 11:11:13 -06:00
Tomas Touceda
4bed4757fb
Remove cleanups and fk for host ids (#3607)
* Remove cleanups and fk for host ids

* Readd missing things to the schema

* Remove unused

* Add changes file and fix some error messages

* Fix test

* Use tx instead of plain writer

* Other fixes

* More not found test fixes

* Go back to getcontext
2022-01-12 14:07:51 -03:00
Martin Angers
f14f97156c
Add read and write timeout options for redis (#3624) 2022-01-11 17:08:39 -05:00
Zach Wasserman
8929917ee9
Update development license (#3562) 2022-01-03 15:35:53 -03:00
Lucas Manuel Rodriguez
787944482b
Global policies automation webhooks (#3378)
* Add webhook to app config

* Add redis failing policies set and webhook

* Add basic webhook test

* Store hostname in redis

* Global policy deletion to remove policy ID from set and config

* Also process new passing policies

* Fix unit test

* Sort hosts

* Add more tests

* Add ListSets to the failing policies interface

* Fix server URL and garbage collect on the triggering side

* Do not use Redis SCAN

* Fix Redis operation order

* Add API changes to doc

* Add comments

* Add more tests

* Fix tests

* Add tests for config update upon deletion of policies

* Run make dump-test-schema

* Ignore policies that failed to run

* Add proper unit tests to trigger logic

* Fix comments

* WIP

* Add tests to service_osquerty_test.go

* Use SSCAN for listing hosts instead of SMEMBERS

* Add failing policies to docs/01-Using-Fleet/configuration-files/README.md

* Remove skip

* Fix PR comments
2021-12-23 18:26:55 -03:00
Martin Angers
e988d16eb3
Update the prometheus go client library (#3140) 2021-12-20 09:20:58 -05:00
Martin Angers
4143a37056
Fix redis scan keys issue for live queries (#3107) 2021-12-14 16:30:26 -05:00
Zach Wasserman
a35ddfd4d8
Increase session duration with --dev flag (#3339) 2021-12-12 17:31:07 -08:00
Lucas Manuel Rodriguez
d0765cb9ee
Do not exit on serve/prepare if there are unknown migrations (#3262)
* Do not exit on serve/prepare if there are unknown migrations

* PR review changes
2021-12-08 19:50:00 -03:00
gillespi314
6ee6dba4a4
Update anonymous usage statistics to record Fleet feature adoption (#3056)
* Collect additional anonymous usage statistics

* Update fleetdm.com and docs for usage stats

* Fix lint-go errors

* Fix failing test

* update webhook inputs

* Update receive-usage-analytics.js

* fix failing test

* Update statistics_test.go

Co-authored-by: eashaw <eashaw@sailsjs.com>
2021-12-06 14:39:00 -06:00
Tomas Touceda
142006cbdd
Issue 2099 enable vulnerability processing new installs (#3104)
* Enable vulnerability processing by default

* Add changes file

* Remove wrong env var

* Fix import

* Fix tests

* Update log line

* Remove unneeded import
2021-12-03 14:44:57 -03:00
Tomas Touceda
a36e029d95
App config caching per instance (#3071)
* Do caching of app config per instance instead of across all of them in redis

* Add changes file

* Simplify code based on review comment

* Use go-cache instead of creating our own

* Dont export consts

* Copy app config before returning it

* Fix lint

* Update go sum

* Update go sum
2021-11-29 12:51:57 -03:00
Lucas Manuel Rodriguez
028c2643f7
Fleet serve to exit if migrations are missing (default) (#2803)
* Add option to fleet to exit if migrations are missing

* Reverse serve exit migrations logic

* Fix typo
2021-11-22 14:47:24 -03:00
Lucas Manuel Rodriguez
0642bfdb1d
Add fleetctl debug migrations command to get missing migrations (#2967)
* Add fleetctl command to get missing migrations

* Fix copy paste and lint

* Detect migrations applied out of order

* Add extra bullet to changes

* Trigger creation of migration status tables

* Fix unit tests

* PR fixes

* PR comment fixes
2021-11-22 14:06:12 -03:00
Martin Angers
69a4985cac
Use new error handling approach in other packages (#2954) 2021-11-22 09:13:26 -05:00
Tomas Touceda
a16b98a4a1
Add logging of interval for webhooks (#2853) 2021-11-09 11:59:16 -03:00
Tomas Touceda
7db6de7278
Serialize hosts writes per instance (#2753)
* Serialize hosts writes per instance

* Write hosts asynchronously

* Dont make the save in a goroutine

* Revert "Dont make the save in a goroutine"

This reverts commit 4a890c5271.

* Make all savehosts async

* Address review comments and make this approach configurable

* Address review comments

* Disable bulk seen time marking for a test

* Move host seen times to a new table

* Remove unused

* Add seen_time to list hosts

* Add some jitter to seen time flushing

* Remove unused

* Add timeout to deferred save host

* Add tests for serialSaveHost

* Update hosts in labels and policy executions in a serial way

* Address review comments and remove fk constraints in host software

* Make errCh buffered

* Add changes file

* Readd key
2021-11-08 11:42:37 -03:00
Tomas Touceda
2902da76ca
Issue 2027 better error visibility (#2069) 2021-11-02 13:35:57 -04:00
Martin Angers
a8735d55bb
Implement async processing of hosts for label queries (#2288) 2021-11-01 14:13:16 -04:00
Lucas Manuel Rodriguez
384ce0f13d
Amend WriteTimeout for live queries GET sync API (#2752) 2021-11-01 09:31:56 -03:00
Tomas Touceda
c2696c1261
Issue 1892 query perf everywhere (#2580)
* wip

* wip

* wip

* Add performance stats

* Add docs and other self review fixes

* lint

* Update based on review comments

* Add quick cleanup first and then reset to 1hr

* Reduce the load in the test
2021-10-20 18:35:38 -03:00
Martin Angers
07cf136d03
Add TLS support to Redis connections (#2568) 2021-10-20 10:09:18 -04:00
Tomas Touceda
fe5660e006
Reimplement host expiration to not need mysql events (#2552)
* Reimplement host expiration to not need mysql events

* Update mocks
2021-10-19 17:47:37 -03:00
Martin Angers
a7420140ce
Use a redis cluster-friendly store for rate limit (#2577) 2021-10-19 16:24:13 -04:00
Martin Angers
057d4e8b2e
Add configuration and support for Redis to read from replicas (#2509) 2021-10-18 09:32:17 -04:00
Tomas Touceda
685245c2bd
Cache app config in redis (#2205)
* Cache app config in redis

* Add changes files

* Replace string with constant

* Revert some test refactorign and duplicate a bit of test code

* Add test for AppConfig with redis failing

* Fix lint

* Use Doer so it works better in clusters

* Skip unmarshalling if we already did

* Allow to cache hosts if configured

* Omit the setting if empty

* Remove hashing, too much CPU

* Revert caching of host auth... needs a more thought through approach

* Remove config

* Remove old config

* Remove locker interface

* Fix test and address review comments
2021-09-28 10:01:53 -03:00
Tomas Touceda
4650484960
Remove fk label membership (#2157)
* Remove fk from label membership

* Add changes file

* Fix tests

* No need to IGNORE anymore
2021-09-21 11:48:20 -03:00
Martin Angers
1fa5ce16b8
Add configurable Redis connection retries and following of cluster redirections (#2045)
Closes #1969
2021-09-15 08:50:32 -04:00
Martin Angers
d956ca1bec
Use the context in the Datastore layer. (#2030) 2021-09-14 10:44:02 -04:00
Tomas Touceda
429af72b69
Issue 1963 vulnerabilities no sync (#1976)
* wip

* Add tests for skip sync

* Add changes file

* Fix lint
2021-09-14 10:58:35 -03:00
Martin Angers
4f4185372d
Add support for context in datastore/mysql layer (#1962)
This is just to pass down the context to the datastore layer, it doesn't
use it just yet - this will be in a follow-up PR.
2021-09-14 08:11:07 -04:00
Tomas Touceda
b2efc9f51c
Make redis conn timeout and keep alive configurable (#1968)
* Make redis conn timeout and keep alive configurable

* Document new configs

* Correct config name
2021-09-08 17:55:12 -03:00
Tomas Touceda
6d2ae02efd
Rename core->free and basic->premium (#1870)
* Rename core->free and basic->premium

* Fix lint js

* Comment out portion of test that seems to timeout

* Rename tier  to premium if basic is still loaded
2021-09-03 13:05:23 -03:00
Martin Angers
9a0871a2f1
Address issues related to Redis Cluster support (#1885)
Closes #1847 .
2021-09-01 16:32:57 -04:00
Martin Angers
3755a58070
Add support for MySQL read replicas (#1786)
Adds configuration options to use a read-only MySQL replica,
and uses it instead of the primary for reads.
2021-09-01 15:50:52 -04:00
Tomas Touceda
15b7f5dd80
Create vuln path if dynamic (#1830)
* Create vuln path if possible

* Make sure we skip creation if static instance is selected

* Document behavior

* Fix return in crons and test without sleeps
2021-08-30 12:29:05 -03:00
Tomas Touceda
8b908f6506
Issue 1599 offline webhook (#1777)
* wip

* Add tests and finish implementation

* Add proper default for periodicity, changes file, and documentation

* Fix tests and add defaults also to new installs

* EnableHostUsers should be true if undefined as well

* In some cases, periodicity can be zero because of the migrations

* Apply defaults when migrating appconfig

* Fix lint

* lint

* Address review comments
2021-08-27 11:15:36 -03:00
Tomas Touceda
d6d53acda5
Issue 1600 fleetctl license expiration (#1800)
* Show banner in fleet and fleetctl if license expired

* Ignore if license is nil or tier is core

* Address review comments
2021-08-26 10:28:53 -03:00
Tomas Touceda
c6c63ab12a
Refactor app config (POC, for now) (#1685) 2021-08-20 12:27:41 -03:00
Tomas Touceda
33791dbee8
Remove fk in scheduled_query_stats table and clean up orphan rows for it (#1720)
* Remove fk in scheduled_query_stats table and clean up orphan rows for it

* Improve test and fix bug with the cleanup
2021-08-18 18:30:48 -03:00
Tomas Touceda
70f728fe84
Make vulnerability processing more configurable (#1718)
* Make vulnerability processing more configurable

* Simplify leader selection
2021-08-18 17:25:14 -03:00