Commit graph

558 commits

Author SHA1 Message Date
gillespi314
d172128183
Add label information to host summary response (#5573) 2022-05-10 10:32:55 -05:00
Martin Angers
1fa7bb7a19
Support async saving of hosts' last seen time (#5640) 2022-05-10 11:29:17 -04:00
Michal Nicpon
a005b7e7b7
fix schema dump (#5562) 2022-05-10 09:03:04 -06:00
gillespi314
998471f716
Omit empty device mapping from host responses where optional device_mapping param is missing (#5541) 2022-05-05 13:13:53 -05:00
gillespi314
991c7ccadc
Add device_mapping to GET /hosts response (#5383) 2022-05-02 16:34:14 -05:00
gillespi314
1dabf52834
Add Zendesk external service integration for vulnerability automations (#5372) 2022-05-02 15:58:34 -05:00
Roberto Dip
a4725518ac
filter out Google Chrome profiles without an associated email before ingesting (#5440)
To add support for #400, we're using the macadmins/osquery-extension to gather Google Chrome profiles from hosts.

Under the hood, the extension looks and parses a json file in which Chrome stores a bunch of data. Given that emails are not required to create Chrome profiles, some of the profiles stored in this file and returned by the query we're using contain empty emails.

The idea after this change is to prevent empty emails from being ingested in the first place instead of filtering them after the fact. I have also included a migration to clean the rows with empty email columns.

Fixes #4780
2022-05-02 09:55:40 -03:00
Roberto Dip
2c23cb5f24
fix mysql tests by with a regexp assertion of possible error messages (#5444)
this fixes #5377 by asserting that the error message starts with "tsl" or "x509" as either of the strings represent the same error depending on the mysql version.

we should refactor the code to avoid checking on the error messages and probably use different error types instead.
2022-04-29 09:31:58 -03:00
Martin Angers
0d0c17e0c6
Add osquery.min_software_last_opened_at_diff configuration option (#5394) 2022-04-27 09:47:09 -04:00
Lucas Manuel Rodriguez
407d8b527e
Remove unused Datastore.SaveHost and unused Modified fields (#5245) 2022-04-26 18:33:29 -03:00
Martin Angers
253baee657
Collect last_opened_at for macOS software, and return it in host details payload (#5376) 2022-04-26 14:16:59 -04:00
Michal Nicpon
a64982f96c
Remove use of JSON_ARRAYAGG (#5350)
Not supported in MySQL < 5.7.22
2022-04-25 14:43:08 -06:00
Lucas Manuel Rodriguez
42f53b63b2
Remove recursion and return doRetryErr on getOrGenerateSoftwareIdDB (#5239)
* Remove recursion and return doRetryErr on getOrGenerateSoftwareIdDB

* Use GetContext and LIMIT 1
2022-04-20 11:02:35 -03:00
Lucas Manuel Rodriguez
29a902960e
Fix typos and amend PR template (#5227) 2022-04-19 18:09:55 -03:00
Juan Fernandez
2479d58262
Using '@' in target search causes 422 error response (#5148)
* Bug 4852:

When searching for Labels, breakdown query strings containing '@' in multiple search terms to
avoid issues with MySQL FTS.
2022-04-19 09:28:49 -04:00
gillespi314
70a61b86f1
Update os version aggregated stats for all teams (#5083) 2022-04-18 16:19:58 -05:00
Michal Nicpon
2dbf1d7975
fix update updated_at for aggregated_stats (#5112)
Update the updated_at column when using ON DUPLICATE UPDATE so that
the counts_updated_at is up to date

* basic sql formatting in code ie whitespace around operators
2022-04-15 14:09:47 -06:00
Martin Angers
7187f1adac
Queue jobs for Jira integration when enabled and new vulnerabilities are found. (#4975) 2022-04-11 16:42:16 -04:00
Martin Angers
5483adc26b
Detect the NOPERM error to mean redis cluster is disabled (#5058) 2022-04-11 16:17:30 -04:00
Ricky Grassmuck
eddc2e2bda
Replace uses of ansi quotes (") in SQL statements (#4726)
To ensure the product works properly when using MySQL with `ANSI_QUOTES` mode enabled,
replace all uses of `""` for values inside SQL statements with `''`

Co-authored-by: Ricky Grassmuck <r.grassmuck@cpanel.net>
2022-04-11 11:47:50 -07:00
Martin Angers
f28dc10a51
Merge jira-integration branch to main (#4949) 2022-04-05 12:56:15 -04:00
Martin Angers
90b15071a4
Introduce API version 2022-04, deprecate use of /global in paths (#4731) 2022-04-05 11:35:53 -04:00
Lucas Manuel Rodriguez
57816592ba
Add read replica testing helpers and fix non-sso login bug (#4908)
not set on the INSERT.
- OUT: Only sets the ID on the passed session and returns it. (`CreatedAt`, `AccessedAt`, are not set.)

New version:

```go
func (ds *Datastore) NewSession(ctx context.Context, userID uint, sessionKey string) (*fleet.Session, error) {
	sqlStatement := `
		INSERT INTO sessions (
			user_id,
			` + "`key`" + `
		)
		VALUES(?,?)
	`
	result, err := ds.writer.ExecContext(ctx, sqlStatement, userID, sessionKey)
	if err != nil {
		return nil, ctxerr.Wrap(ctx, err, "inserting session")
	}

	id, _ := result.LastInsertId() // cannot fail with the mysql driver
	return ds.sessionByID(ctx, ds.writer, uint(id))
}
```

- IN: Define arguments that are truly used when creating a session.
- OUT: Load and return the fleet.Session struct with all values set (using the `ds.writer` to support read replicas correctly).

PS: The new `NewSession` version mimics what we already do with other entities, like policies (`Datastore.NewGlobalPolicy`).
2022-04-04 16:52:05 -07:00
Martin Angers
71f2690ada
Add Jira integrations config support (#4863) 2022-03-30 09:10:02 -04:00
Michal Nicpon
a6902cc083
Add os versions endpoint (#4749) 2022-03-28 09:15:45 -06:00
Martin Angers
c4946335ff
Add migration to cleanup host-related tables (#4778)
#4448
2022-03-23 15:22:10 -07:00
Lucas Manuel Rodriguez
8f9ed3ef7b
Fix Datastore.DeleteHosts (#4745) 2022-03-23 12:15:05 -03:00
Michal Nicpon
7b671ac2a3
Add team failing policies webhook (#4633)
* add config to teams
* update api docs
* update tests
2022-03-21 13:16:47 -06:00
Lucas Manuel Rodriguez
ecdfd627b6
Fleet Desktop MVP (#4530)
* WIP

* WIP2

* Fix orbit and fleetctl tests

* Amend macos-app default

* Add some fixes

* Use fleetctl updates roots command

* Add more fixes to Updater

* Fixes to app publishing and downloading

* Add more changes to support fleetctl cross generation

* Amend comment

* Add pkg generation to ease testing

* Make more fixes

* Add changes entry

* Add legacy targets (until our TUF system exposes the new app)

* Fix fleetctl preview

* Fix bool flag

* Fix orbit logic for disabled-updates and dev-mode

* Fix TestPreview

* Remove constant and fix zip-slip attack (codeql)

* Return unknown error

* Fix updater's checkExec

* Add support for executable signing in init_tuf.sh

* Try only signing orbit

* Fix init_tuf.sh targets, macos-app only for osqueryd

* Specify GOARCH to support M1s

* Add workflow to generate osqueryd.app.tar.gz

* Use 5.2.2 on init_tuf.sh

* Add unit test for tar.gz target

* Use artifacts instead of releases

* Remove copy paste residue

* Fleet Desktop Packaging WIP

* Ignore gosec warning

* Trigger on PR too

* Install Go in workflow

* Pass url parameter to desktop app

* Fix fleetctl package

* Final set of changes for v1 of Fleet Desktop

* Add changes

* PR fixes

* Fix CI build

* add larger menu bar icon

* Add transparency item

* Delete host_device_auth entry on host deletion

* Add SetTargetChannel

* Update white logo and add desktop to update runner

* Add fleet-desktop monitoring to orbit

* Define fleet-desktop app exec name

* Fix update runner creation

* Add API test before enabling the My device menu item

Co-authored-by: Zach Wasserman <zach@fleetdm.com>
2022-03-21 14:53:53 -03:00
Benjamin Edwards
74bb559645
Add public ip to hosts & derive geolocation when rendering host (#4652)
* geoip wip
* return nil if ip is empty string or if ParseIP returns nil
* add ui component to render geolocation if available, address PR feedback
* render public ip if available
* add changes file, document geoip in deployment guide
* update rest-api docs
2022-03-21 12:29:52 -04:00
Zach Wasserman
efbc2b92bb
Fix race condition in updates test (#4661)
Copy the DefaultOptions in order to prevent a data race on the Targets
map. This race should only have effected testing.

Race detector output:

```
WARNING: DATA RACE
Read at 0x00c0000908d0 by goroutine 15:
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.18.0/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/fleetdm/fleet/v4/orbit/pkg/update.TestMakeRepoPath.func1()
      /home/runner/work/fleet/fleet/orbit/pkg/update/update_test.go:58 +0xb6
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x47

Previous write at 0x00c0000908d0 by goroutine 12:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.18.0/x64/src/runtime/map_faststr.go:203 +0x0
  github.com/fleetdm/fleet/v4/orbit/pkg/update.TestMakeRepoPath.func1()
      /home/runner/work/fleet/fleet/orbit/pkg/update/update_test.go:62 +0x1cb
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x47

Goroutine 15 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x724
  github.com/fleetdm/fleet/v4/orbit/pkg/update.TestMakeRepoPath()
      /home/runner/work/fleet/fleet/orbit/pkg/update/update_test.go:53 +0x1a4
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x47

Goroutine 12 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x724
  github.com/fleetdm/fleet/v4/orbit/pkg/update.TestMakeRepoPath()
      /home/runner/work/fleet/fleet/orbit/pkg/update/update_test.go:53 +0x1a4
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486
      +0x47
 ```
2022-03-18 09:30:45 -07:00
Martin Angers
c0591dac5d
Ignore pagination params for the download csv endpoint (#4608) 2022-03-16 09:01:52 -04:00
Lucas Manuel Rodriguez
d2ba34c8fc
Add distributed discovery query support for detail queries, add orbit… (#4597)
* Add distributed discovery query support for detail queries, add orbit_info ingestion

* Amend changes file
2022-03-15 16:51:00 -03:00
Martin Angers
a1c67547b3
Add new endpoints to retrieve device information by orbit identifier (#4531) 2022-03-09 16:13:56 -05:00
Martin Angers
c8bc026d6f
Migrate special-case endpoints to new pattern (#4511) 2022-03-08 11:27:38 -05:00
Tomas Touceda
dd62695791
Add host device auth table (#4494) 2022-03-08 09:05:53 -03:00
Martin Angers
4930ca2d0e
Support listing software hosts count filtered by team (#4388) 2022-02-28 13:55:14 -05:00
Martin Angers
18e7c8e236
Implement cron cleanup job of policy membership when policy platform is updated (#4331) 2022-02-23 16:10:37 -05:00
Martin Angers
ad5146c491
Allow updating the policy platform (part 1 of the ticket) (#4311) 2022-02-22 13:42:03 -05:00
Tomas Touceda
4034a7ab59
Make SearchHosts behave the same as the filtering in ListHosts (#4295) 2022-02-22 10:19:51 -03:00
Martin Angers
2ab1b9ec85
Remove expiration of API-only user tokens (#4314) 2022-02-22 08:12:03 -05:00
Michal Nicpon
4ce23c7d1b
Add team targets to pack spec (#4272)
* skip flaky tests
2022-02-21 09:18:58 -07:00
Tomas Touceda
6d582cffa0
Don't depend on last inserted id to get the id for the new software (#4298)
* Don't depend on last inserted id to get the id for the new software

* Correct typo
2022-02-18 16:30:24 -03:00
Tomas Touceda
8e68ec3b96
Insert one CVE per CPE when there are multiple matches (#4297)
* Insert one CVE per CPE when there are multiple matches

* Remove comment

* No need to do sqlx.In
2022-02-18 15:25:26 -03:00
gillespi314
089f49cc5d
Add new activity types for policy-related activities (#4043)
* Add new activities for policy create, delete, and modify, and apply policy spec
* Add PoliciesByID ds method; refactor delete svcs
2022-02-16 09:33:56 -06:00
Tomas Touceda
a49f2a06ed
Allow removal of munki versions (#4189)
* Skip any versions that are empty in munki

* Return not found if munki is uninstalled

* Fix lint

* wip

* Add deleted at for host_munki_version
2022-02-15 16:29:14 -03:00
Michal Nicpon
0be26613b3
Cache team agent options (#4193)
* use raw literal for json
* wrap cache to clone all values
2022-02-15 12:07:51 -07:00
Zachary Winnerman
c5c72ed713
Add apm for testing apm (#4053)
* Add apm for testing apm

* Testing opentracing

* testing

* Testing

* go fmt

* Add config switch for tracing.

* fixup

* Update cmd/fleet/serve.go

Co-authored-by: Tomas Touceda <chiiph@gmail.com>

* Add support for both elasticapm and opentelemetry

* Fix driver stuff and config options

* Fixup

* fixup

* Add changes file

* Add config for sql driver

* fixup

* Add doc to exported field

* testing

* fixup

* fixup

* Testing again

* fixup

* testing

* Undo

Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2022-02-15 12:42:22 -05:00
Lucas Manuel Rodriguez
be72dc356c
Add CentOS parsing+post-processing to reduce false positives in vulnerability processing (#4037)
* Add CentOS parsing and post-processing in fleet

* Add tests and amend SyncCPEDatabase

* Add test for centosPostProcessing

* Changes from PR comments

* Amend software test

* Fix sync test

* Add index to source and vendor

* Use os.MkdirTemp

* Rearrange migrations

* Regenerate test schema

* Add support for testing migrations (#4112)

* Add support for testing migrations

* Rename migration in tests

* Changes suggested in PR

* Go mod tidy
2022-02-14 15:13:44 -03:00
Lucas Manuel Rodriguez
9c25ea1641
Prepare LoadHostByNodeKey query once (#4128)
* Prepare LoadHostByNodeKey query once

* Use a protected map for storing statements

* Add proposed test
2022-02-14 12:13:38 -03:00