Commit graph

1272 commits

Author SHA1 Message Date
Victor Lyuboslavsky
bb30ffb21f
Added live query stats to activity. (#15682)
📺 Loom:
https://www.loom.com/share/474ac220dd054c389ebed6ba13f943b1?sid=70c2d94e-9fd9-40f5-af73-13e6d2e4c90a
Final piece for #467 -- Added live query stats to activity feed.
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-18 09:31:07 -06:00
Victor Lyuboslavsky
b930f66229
Live query stats are cleared when query SQL is modified. (#15710)
Live query stats are cleared when query SQL is modified.

Also, when deleting one or more queries, the associated aggregated stats
are now deleted as well.

#15709

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-18 08:45:19 -06:00
Victor Lyuboslavsky
0e040cc7b0
fleetctl now runs saved queries (#15667)
📺 Looom:
https://www.loom.com/share/1aec4616fa4449e7abac579084aef0ba?sid=0884f742-feb3-48bb-82dc-b7834bc9a6e1

Fixed fleetctl issue where it was creating a new query when running a
query by name, as opposed to using the existing saved query.
#15630

API change will be in a separate PR:
https://github.com/fleetdm/fleet/pull/15673

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-15 12:55:39 -06:00
Lucas Manuel Rodriguez
024a20ac11
Allow enrolling fleetd using osquery's instance identifier (#15570)
#14879

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2023-12-15 15:26:32 -03:00
Jahziel Villasana-Espinoza
bcb66e8893
feat: add populate software query param (#15661)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-14 16:18:30 -05:00
Tim Lee
ced538c916
15597 Observer list query filter (#15653) 2023-12-14 13:25:42 -07:00
Marcos Oviedo
9812c10d1d
Fixing Syncml cmd nested command data (#15632)
This relates to #15107 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [X] Manual QA for all new/changed functionality
2023-12-14 15:05:00 -03:00
Tim Lee
27073720c5
15254 oval scan err (#15499) 2023-12-14 10:39:28 -07:00
Roberto Dip
82b7156952
ensure distributed queries can't be empty strings (#15641)
There's a bug somewhere (Fleet, osquery or both?)
that causes hosts to check-in in a loop if you send an empty query
string.

we previously fixed this for detail query overrides (see #14286, #14296)
but I'm also adding a general check as a safeguard for issues like
#15524
2023-12-14 12:38:54 -03:00
Sarah Gillespie
27bd63d8f0
Fix unreleased bug in parsing enroll reference from MDM server URL (#15634) 2023-12-13 22:15:24 -06:00
Victor Lyuboslavsky
835eedae42
Performance stats for live queries (#15440)
📺 Loom explaining the PR:
https://www.loom.com/share/8c6ec0c362014910931f183d68307525?sid=7de51e6f-a59d-4d8f-b06b-f703f1692f17

#467 
Live Queries now collect stats. Stats are collected for saved queries
present in the Queries tab.
- After running a live query, user will see updated stats in Queries
tab.
- Query stats on Host Details page will no longer be cleared after
host/agent reboots.
- Query stats are now deleted when query is deleted.

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-13 14:46:59 -06:00
Victor Lyuboslavsky
2520fc2179
Reduce server load for fleetd upgrade (#15606)
📺 Loom:
https://www.loom.com/share/9e17848963574af3aa10d426b450bcd0?sid=f8078293-c7e1-4864-a8a3-4cec996971f5

#15476 
#15540 
#15542

After upgrading fleetd, customer-blanco saw a spike in traffic and a
spike in DB connections. These fixes attempt to reduce the traffic and
DB load when fleetd is upgraded.

On the server, added fleet/device/{token}/ping endpoint to be used by
agents to check their token.

On the agent:
- Removed call to fleet/orbit/device_token unless token needs to be
updated.
- Changed call to fleet/device/{token}/desktop with a less resource
intensive call to fleet/device/{token}/ping
- Removed call to fleet/orbit/ping

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2023-12-13 14:31:48 -06:00
Roberto Dip
fbaa479d01
pre-calculate host counts for software titles (#15601)
found during the load test, this addresses a performance issue with
software titles.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-13 12:48:57 -03:00
Jacob Shandling
3a482f56b7 Merge branch 'main' into 14415 2023-12-12 10:37:41 -08:00
Jahziel Villasana-Espinoza
e1eb017249
fix: send back queries but ignore them on the FE (#15507)
> #15009 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-12 10:36:33 -08:00
Roberto Dip
2ed3026819
Add pagination meta to software versions endpoint (#15550) 2023-12-12 12:24:20 -06:00
Tim Lee
f1acd30bcf
15380 extend hosts api (#15421) 2023-12-12 08:40:57 -07:00
Tim Lee
e981b7be84
hotfix: fix return codes on enterprise tests (#15578) 2023-12-12 07:58:47 -07:00
Tim Lee
33999cddae
15381 host query report api (#15441) 2023-12-11 15:33:31 -07:00
Jacob Shandling
d2359675d4 Merge branch 'main' into 14415 2023-12-11 13:52:45 -08:00
Sharon Katz
da755ea2ea
Bug-15430 change timeout errror code to 408 (#15552) 2023-12-11 12:34:59 -05:00
Victor Lyuboslavsky
9236a19342
Changed query performance statistics to uint64 to match osquery reports. (#15505)
#15472

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-11 11:29:17 -06:00
Victor Lyuboslavsky
eb155d17d3
Reducing the number of fleetd calls to fleet/orbit/config endpoint. (#15545)
Reducing the number of fleetd calls to fleet/orbit/config endpoint by
caching the config for 3 seconds.
#15541

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2023-12-11 07:04:24 -06:00
Jahziel Villasana-Espinoza
66939494f2
fix: validate agent platform options (#15482)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-08 10:20:31 -05:00
Tim Lee
3ba81e1cf6
15146 filter query results (#15473) 2023-12-07 11:24:56 -08:00
Sarah Gillespie
88e8eee1fe
Add backend capability to look up hosts based on MDM IdP email (#15478)
Issue #13034 

TODO: Frontend requirements will be covered in a separate PR.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-07 12:36:32 -06:00
Martin Angers
8b7d1a7e47
Remove Windows MDM feature flag (#15167)
For #14959 

---------

Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
2023-12-07 14:24:32 -03:00
Lucas Manuel Rodriguez
fb3d5f85bd
Results for queries in 2017-packs should always log (#15486)
#15168

- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
2023-12-07 13:05:59 -03:00
Martin Angers
e3d225ade7
Update fleetctl get software to list titles and versions. (#15444) 2023-12-06 16:07:03 -05:00
Martin Angers
6b128dd455
Allow filtering hosts by software_version_id and software_title_id. (#15433) 2023-12-06 14:59:00 -05:00
Jahziel Villasana-Espinoza
5fd799ff2e
feat: device health endpoint (#15432)
> #14920 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-06 14:42:29 -05:00
Roberto Dip
aada28c1c1
Add list/detail endpoints for software titles (#15464)
related to #15228
2023-12-06 15:28:31 -03:00
Sarah Gillespie
f19dc8abe0
Add GET software/versions and GET software/versions/:id endpoints (#15450) 2023-12-06 08:30:49 -06:00
mostlikelee
5677d734de Merge branch 'main' into 14415 2023-12-04 14:38:28 -07:00
Lucas Manuel Rodriguez
9b40573f54
Fix installed_from_dep value in activity for DEP re-enrolling (#15419)
#12037 

![Screenshot 2023-12-04 at 17 03
03](https://github.com/fleetdm/fleet/assets/2073526/e44cee72-1f1a-4a10-be80-26f0e06c8c91)

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
2023-12-04 17:14:16 -03:00
Tim Lee
dc3fc5e6f5
15378 record empty data results (#15403) 2023-12-04 08:31:35 -07:00
Martin Angers
f5f66ebe63
Unify profiles database identifier to profile_uuid for macOS and Windows profiles. (#15297) 2023-12-04 10:04:06 -05:00
Lucas Manuel Rodriguez
ab0700b4e0
Fix osquery result logging when queries are configured outside of Fleet (#15393)
#15168

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests.
- [X] Manual QA for all new/changed functionality.

The issue can be reproduced by running `osqueryd` with
`--config_plugin=filesystem --config_path=/path/to/config.json`
This means the osquery config is fetched from a file rather than from
Fleet's agent settings.
The `/path/to/config.json` has the agent settings, e.g.:
```
{
  "decorators": {
    "load": [
      "SELECT uuid AS host_uuid FROM system_info;",
      "SELECT hostname AS hostname FROM system_info;"
    ]
  },
  "options": {
    "disable_distributed": false,
    "distributed_interval": 10,
    "distributed_plugin": "tls",
    "distributed_tls_max_attempts": 3,
    "logger_tls_endpoint": "/api/osquery/log",
    "logger_tls_period": 10,
    "pack_delimiter": "/"
  },
  "schedule": {
    "USB devices": {
      "query": "SELECT * FROM usb_devices;",
      "interval": 15
    },
    "OS version": {
      "query": "SELECT * FROM os_version;",
      "interval": 10
    }
  },
  "packs": {
    "Elsewhere": {
      "queries": {
        "Osquery Info": {
          "query": "SELECT * FROM osquery_info;",
          "interval": 30,
          "platform": "",
          "version": "",
          "snapshot": true
        }
      }
    }
  }
}
```

The three queries should be logged to Fleet's configured result logging
destination (default is `filesystem`).
2023-12-04 11:18:49 -03:00
Roberto Dip
a7be0be9e9
improve mdm_windows query to account for multiple registry entries (#15391)
for #15362, this adjusts the query we use to get MDM details for windows
to account for hosts that might have more than one matching value in the
registry for any of the items we query.
2023-12-04 11:03:05 -03:00
Victor Lyuboslavsky
a7ae828581
Adding extension_id and browser columns to software table. (#15285)
https://github.com/fleetdm/confidential/issues/2118
API doc changes are at: https://github.com/fleetdm/fleet/pull/15092

This change affects Chrome-based and Firefox browsers. I tried Safari,
but it appears that latest Safari extensions simply show up as regular
apps on macOS.

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
  - Tested detail query changes on macOS, Windows, Ubuntu, and ChromeOS
  - Tested Chrome, Edge, Brave, and Firefox browsers

Testing notes:
1. Install some browser extensions.
2. Update fleet to this change.
3. Refresh hosts, and trigger vulnerabilities job (which updates host
software).
4. Do something like `GET
https://localhost:8080/api/v1/fleet/software?query=Password%20Manager`
to see `extension_id` and `browser` for the hosts that have updated.
2023-11-30 19:06:17 -06:00
Sarah Gillespie
976eeba61d
Add validations to disallow custom MDM profiles that contain names reserved by Fleet (#15373) 2023-11-30 17:19:18 -06:00
Roberto Dip
b07fbdc1d7
implement profile verification for windows (#15348)
for #14426.

In order to prevent import cycles and be able to use some type
definitions/constants I followed the same pattern we did for macOS by
creating a new package named `syncml`. This makes the changelog look
bigger than it actually is, so I split it into two commits to make it
easier to review:

-
[d7c233d](d7c233d54c)
moves the relevant bits to this new package
-
[7531a07](7531a0742b)
implements profile verification
2023-11-30 09:17:07 -03:00
Martin Angers
2f927df4f0
Implement Windows OS Updates (feature branch). (#15359) 2023-11-29 11:07:24 -05:00
Martin Angers
0b5eedb801
Fix edge case of AppConfig changes getting lost in cached mysql. (#15352) 2023-11-29 10:09:37 -05:00
Sarah Gillespie
1535b72305
Add hardware serial to host by identifier search (#15353) 2023-11-29 08:39:35 -06:00
Roberto Dip
f439432fae
feature: windows profiles (#15349) 2023-11-29 09:32:42 -05:00
Roberto Dip
4d56d25f11
show FV banner if the disk is encrypted but we don't get a key (#15317)
for #15068

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-11-27 18:23:01 -03:00
Roberto Dip
9c7b740b03
disable windows profiles feature in the back-end (#15278)
per discussion in slack, reverting relevant user-facing changes for
#13281
2023-11-27 16:08:52 -03:00
Victor Lyuboslavsky
8cfe272091
filtering hosts with invalid team_id now returns 400 error. (#15266)
#15037 
For endpoint fleet/hosts, filtering hosts with invalid team_id now
returns 400 error.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-11-22 15:04:48 -06:00
Roberto Dip
5f313c8972
fix various bugs after testing windows MDM profiles (#15264)
for #13281
2023-11-22 10:48:28 -03:00