* Remove tx from SaveHost
Also change REPLACE for software for an insert.
* Remove bundle identifier from the get since it's not part of the key
* Revert unneeded change
* Add software count API
* Fix makefile
* Fine no mock generating at this point
* Actually, one last try
* Use go install instead
* Fix go sum/mod
* Improve documentation
* Try setting node to 14
* Do caching of app config per instance instead of across all of them in redis
* Add changes file
* Simplify code based on review comment
* Use go-cache instead of creating our own
* Dont export consts
* Copy app config before returning it
* Fix lint
* Update go sum
* Update go sum
* Add fleetctl command to get missing migrations
* Fix copy paste and lint
* Detect migrations applied out of order
* Add extra bullet to changes
* Trigger creation of migration status tables
* Fix unit tests
* PR fixes
* PR comment fixes
* Further optimize host select with policies
* Add missing row close
* Improve migration
* Skip migration if column already exists
* Add test for host with/without extras
* Fix duplicate schedules and platform matching on scheduled queries
* scheduled_queries.platform can be NULL
* Add unit tests
* Add rhel host and check zero stats
* Add host count to software API
* Update docs
* Update fleetctl tests to account for host counts
* Update docs to mention host_count special case
* Update func comment
* Ignore admin forced password reset if sso is enabled
* Add tests for authenticatedUser
* If SSOEnabled is true, then set AdminForcedPasswordReset to false
* Update tests to cover new password reset and sso flow
* Add changes file
* Serialize hosts writes per instance
* Write hosts asynchronously
* Dont make the save in a goroutine
* Revert "Dont make the save in a goroutine"
This reverts commit 4a890c5271.
* Make all savehosts async
* Address review comments and make this approach configurable
* Address review comments
* Disable bulk seen time marking for a test
* Move host seen times to a new table
* Remove unused
* Add seen_time to list hosts
* Add some jitter to seen time flushing
* Remove unused
* Add timeout to deferred save host
* Add tests for serialSaveHost
* Update hosts in labels and policy executions in a serial way
* Address review comments and remove fk constraints in host software
* Make errCh buffered
* Add changes file
* Readd key
* Make migrations compatible with GTID replication
Fixes an issue some deployments encountered when migrations used a
statement that is unsupported in GTID replication mode (#2462).
Local dev MySQL now enforces this consistency, so it should be easier to
maintain compatibility going forward.
* Update docker-compose formatting
* if exists
* wip
* wip
* wip
* Add performance stats
* Add docs and other self review fixes
* lint
* Update based on review comments
* Add quick cleanup first and then reset to 1hr
* Reduce the load in the test
- Use `TRUNCATE TABLE` rather than `DELETE FROM` for improved performance.
- Move DDL statement after truncate to avoid issues with retries (due to
column already being created).
#2360
* Ignore empty host users or software inventory
* Only store additional if it's not nil
* Update label/policy updated at when we record the executions and skip saving host
* Update changes file
* Cache app config in redis
* Add changes files
* Replace string with constant
* Revert some test refactorign and duplicate a bit of test code
* Add test for AppConfig with redis failing
* Fix lint
* Use Doer so it works better in clusters
* Skip unmarshalling if we already did
* Allow to cache hosts if configured
* Omit the setting if empty
* Remove hashing, too much CPU
* Revert caching of host auth... needs a more thought through approach
* Remove config
* Remove old config
* Remove locker interface
* Fix test and address review comments
* Add max jitter percent config
* Fix jitter calc
* Remove comment
* Reduce test jitter to make tests less flaky
* Remove jitter entirely
* Document new config
* Fix doc link
* Add team policies
* Add team policy documentation
* Add changes file
* Update titles
* Fix lint
* Rewrite TeamAuthorize for more clarify
* Explicitly use two slices for clarity
* Simplify switch
* Implement fleetctl get software and the underlying API
* Add documentation
* Simplify list software implementation
* Lint fixes
* Make team name unique
* Address review comments
* Fix lint
* Fix tests
* wip
* Add tests and finish implementation
* Add proper default for periodicity, changes file, and documentation
* Fix tests and add defaults also to new installs
* EnableHostUsers should be true if undefined as well
* In some cases, periodicity can be zero because of the migrations
* Apply defaults when migrating appconfig
* Fix lint
* lint
* Address review comments
* Add global policies
* Update documentation and add extra parameter to config
* Fix failing tests
* Store historic policy records
* Address review comments
And also remove other inmem references I saw by chance
* Add documentation for get by id request
* Add parameter doc
* Move schema generation to a cmd instead of a test
Otherwise it messes up running all tests sometimes depending on how parallel it does
* Remove brain dump for another task
* Make migration tests a separate beast
* Make schema generation idempotent and move dbutils cmd to tools
* Allow all filters and add counts to Policy
* Add test for Policy
Add a relatively minimal set of linters that raise safe and
mostly un-opinionated issues with the code. It runs
automatically on CI via a github action.
* Make team schedule names more user friendly and hide them from host pack stats
* Delete test for a different bug and add migration
* Update name template
* Use GROUP_CONCAT instead of JSON_ARRAYAGG
* Update server/datastore/mysql/software.go
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
* Add All Linux label
* Change name to Linux instead of All Linux to see if e2e likes it better
* Revert "Change name to Linux instead of All Linux to see if e2e likes it better"
This reverts commit 26b79f214e.
* Fix all linux label insert
* Fix build, add missing tests for cpe translations
Also dont fail alltogether if there's one issue translating CPEs, log it and continue
* Make it once every hour again
* Use MATCH but escape strings
* add team_id filter to fleetctl via get hosts --team flag & api via api/v1/fleet/hosts and api/v1/fleet/labels/id/hosts
* update tests & add changes file
* Dont delete pack stats before inserting new ones to prevent deadlocks
* Remove fk for scheduled_query_stats
* Remove fk removal
* Fix tests
* Remove unneeded comment
* modify packs api to filter non-empty pack_type from response
* change list packs store method to allow filtering for "system-level" packs, by default the api filters these packs from being returned
* add changes file
* don't allow modifications to global or team packs via apply spec
* refactor to use PackListOptions
* Expose vulnerabilities data in host software data
* Gather cves and software in one query
* Expand the test to cover all cases
* Make test less flaky
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Add first cve db generation
* WIP but with final strategy, preparring to merge main
* Fix merge conflicts
* WIP
* wip
* Insert CVEs to the db
* Remove unused code
* Use wg instead of counting
* Call cancelFunc to avoid ctx leak
* Fix logs for better readability
* Point code to fleetdm instead of my repo
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Address review comments
* WIP
* Send usage analytics
* Improve loggin of cron tasks and fix test
* Implement appconfig method now that we are checking that as well
* Address review comments
* Migrate all mysql tests to the new form
* Only dump sql if MYSQL_TEST is on
* Removing parallel until we get rid of this code
* Move TestMain to an actual _test file
* A little experiment with tmpfs to speed up the db
* Let's make sure the dump.sql file is also in ram
* WIP
* Add get user_roles and apply for a user_roles spec to fleetctl
* Uncomment other tests
* Update test to check output
* Update test with the new struct
* Mock token so that it doesn't pick up the one in the local machine
* Address review comments
* Fix printJSON and printYaml
* Fix merge conflict error
* WIP
* wip
* wip
* Finish implementation
* Address review comments
* Fix flaky test
* WIP
* Add get user_roles and apply for a user_roles spec to fleetctl
* Uncomment other tests
* Update test to check output
* Update test with the new struct
* Mock token so that it doesn't pick up the one in the local machine
* Address review comments
* Fix printJSON and printYaml
* Fix merge conflict error
* If both roles are specified, fail
* Fix test
* Switch arguments around
* Update test with the new rule
* Fix other tests that fell through the cracks
* Add host users
* Add changes file and test removing pull_request from the on test
* Remove users and store the removal timestamp
* Improve test yml to allow for PRs from forks
* Make roles for users mandatory
* Remove nop migration
* Add missing test for wrong role
* Properly validate global and team roles
* Address codacy issues
* Address codacy review
* No need to check for nil
* First approach to diff
* Refactor things for better readability and testing
* Remove draft comment for algorithm
* Format things a bit better
* Remove unused and simplify code a bit
* Refactor for readability and testing
* Add changes file
* Implement new approach based on review comments
* Make sure to only delete from the current host
* Add single uninstall test and fix code
* Improve code based on review
* Refactor error handling for better extensibility and add more scaffolding for specific db errors
* Add integration tests to check errors from mysql are translated properly
* Address review comments
* Add changes file
* Remove username from UI code
* Remove username from tests
* Remove username from database
* Modify server endpoints for removing username
* Implement backend aspects of removing username
* Update API docs
* Add name to fleetctl
- Add enable_analytics column to database.
- Allow enable_analytics to be set via API.
- Add messaging in fleetctl setup.
Note that this defaults to off for existing installations, and defaults
on for newly set up installs.
No collection or sending of analytics yet exists, we are strictly
storing the preference at this time.
Part of #454
This may help with deadlocks on the `label_membership` table. It is not
clear from MySQL documentation whether the order of the records is
significant for locking within a single query. If it is, this should
help the problem. If it is not, this should have no negative impact.
May fix#1146
* #511 refactored update options - new params & ts
* updated server to include agent_options for read and update
* added agent options form to org settings
* #511 finished connecting agent form to server
* #511 fixing api to save/read agent options
* #511 linted
* #511 fixed reading & updating agent options
* #511 api fixes to support agent options
* #511 removed log
* Fix json.RawMessage pointers in tests
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
Reorder migrations from the long-running `teams` branch to ensure that
they can run successfully for deployments upgrading from a pre-4.0
release.
All migrations from the `teams` branch are reordered to take place
_after_ all migrations from the `main` branch, using `20210601` as the
new date, after the latest released `main` branch migration on `20210526`.
Fixes#1058
- Add TeamFilter to relevant host and label methods.
- Pass appropriate filter in service methods.
The dashboard should now show the appropriate hosts for a user's team membership.
- Add `team_id` field to secrets.
- Remove secret `name` and `active` fields (migration deletes inactive secrets).
- Assign hosts to Team based on secret provided.
- Add API for retrieving secrets by Team.
- Accept Teams as a searchable target type for the target selection API.
- Accept Teams for targets in running live queries.
- Refactoring to support these changes.
- Update API documentation.
- Move host `additional` into a separate table.
- Join when that data is needed.
- API change: `/api/v1/fleet/hosts` now returns only the requested
`additional` columns, unless `*` is provided as the sole argument.
Background:
A customer reported that MySQL binlogs grew huge and replication lag
went way up when data was stored in the `additional` column. In this
deployment MySQL was running with ROW replication. This would cause the
entire `additional` data to be copied on each update of the host checkin
time. While switching to STATEMENT or MIXED replication would likely
mitigate the issue, this was not an option in their environment.
Some datastore and service methods would return slices of structs,
rather than slices to pointers of structs (which most methods used).
Make this more consistent.
- Include only hosts that the user has access to in search targets API.
- Add parameter to specify whether `observer` hosts should be included.
- Generate counts based on which hosts user can access.
- Update API doc.
