Commit graph

207 commits

Author SHA1 Message Date
Victor Vrantchan
6cb1026d86 always return errors to the client as a map slice (#724)
keep the format for error returns consistent by always returning a
[]map[string]string for json errors. This simplifies the error handling
on the frontend.

Use "name":"base" as the name field for errors which do not have
a specific or known form field.
2016-12-29 19:40:12 -05:00
Victor Vrantchan
a13042e11b Redirect frontend routes to setup if setup is not configured. (#721)
Closes #617
2016-12-29 18:36:36 -05:00
Zachary Wasserman
ca72a5c67a Add check for all platforms in label query retrieval (#705)
Fixes #703
2016-12-29 12:11:26 -08:00
John Murphy
eec835a07a Osquery options 365 (#657) 2016-12-29 12:32:28 -06:00
Victor Vrantchan
52580ff872 Add invite email template and use the new invite mailer pattern (#711)
Closes #693
Closes #581
2016-12-28 11:55:03 -05:00
Zachary Wasserman
39ebd81dc5 Close orphaned distributed query campaign after failed publish attempt (#707)
A distributed query campaign can be "orphaned" (left in the QueryRunning state)
if the Kolide server restarts while it is running, or other weirdness occurs.
When this happens, no subscribers are waiting to read results written by
osqueryd agents, but the agents continue to receive the query. Previously, this
would cause us to error on ingestion.

The new behavior will instead set the campaign to completed when it detects
that it is orphaned. This should prevent sending queries for which there is no
subscriber.

- New NoSubscriber error interface in pubsub
- Detect NoSubscriber errors and close campaigns
- Tests on pubsub and service methods

Fixes #695
2016-12-27 10:35:19 -05:00
Victor Vrantchan
6b41d2b3f0 always print email messages in dev mode (#698) 2016-12-24 10:41:59 -05:00
Victor Vrantchan
f092c614cf add error logging to distributed query campaign handler (#694) 2016-12-22 20:31:45 -05:00
Zachary Wasserman
2c7fc2a6f2 Minor refactoring in HostStatus function (#692)
- Refactor function
- Add test
2016-12-22 13:32:40 -08:00
Mike Arpaia
ef22d81b50 Return query and name after creating a scheduled query (#690) 2016-12-22 14:29:29 -05:00
Victor Vrantchan
3658e4833e add prometheus metrics to every http endpoint in the app (#680)
tracking the following metrics:

http_request_duration_microseconds
http_request_size_bytes
http_response_size_bytes
http_requests_total
2016-12-22 12:39:44 -05:00
Victor Vrantchan
e7e57ddbc0 prevent password reuse when changing passwords (#678)
For #375
Closes #448
2016-12-22 12:08:29 -05:00
Victor Vrantchan
a47179f142 add a /healthz endpoint which checks that the app is in a healthy state (#674)
by pinging the mysql and redis backends.

For #93
2016-12-22 12:07:47 -05:00
Mike Stone
a84c40061a Prepends https protocol to the inmem org logo url (#685) 2016-12-22 10:03:20 -05:00
John Murphy
d653cdf281 Minor App Config API tweaks requested by @mikestone14 (#681)
* Minor App Config API tweaks requested by @mikestone14

* Refactored mail test into separate method, implemented code review changes
2016-12-22 08:12:34 -06:00
Victor Vrantchan
19fc70ed64 change logo_url for devMode (#683)
Closes #682
2016-12-21 16:22:10 -05:00
Victor Vrantchan
8ef11f51bf create user from invite: set admin property from invite. (#675)
Closes #589
2016-12-21 12:34:10 -05:00
Victor Vrantchan
11a5104d2c errors.Wrap mysql User store (#669) 2016-12-20 17:09:49 -05:00
John Murphy
62748bfbb1 App Settings - /admin/settings #363 (#590) 2016-12-20 15:54:30 -06:00
Victor Vrantchan
41610c1f66 catch sql.ErrNoRows in mysql/invites (#667) 2016-12-20 16:31:09 -05:00
Victor Vrantchan
9d49dbc465 change the implementation of ErrNotFound and AlreadyExists to a struct type (#665)
with an exposed interface.

Not checking for a specific sentinel error reduces coupling between packages
and allows adding context like the resource ID and resource type.
2016-12-20 13:35:22 -05:00
Zachary Wasserman
c8e6405220 Use redis for distributed query results when not in dev mode (#653)
- Add appropriate configs for Redis
- Use the Redis pubsub store by default, inmem in dev mode
2016-12-15 16:13:23 -08:00
Zachary Wasserman
d6765377c0 Rename inmem method receivers for consistency (#654)
This makes the inmem method receiver naming consistent with mysql. It also
eliminates potential confusion with the phrase "orm".
2016-12-15 16:12:43 -08:00
Zachary Wasserman
41120ebc00 Improve user endpoint validations (#642)
- Add empty string checks for NewUser
- Create validations for ModifyUser
- Use GravatarURL when creating new user

Fixes #620
2016-12-15 09:28:53 -08:00
Zachary Wasserman
216e1b58fa Fix parentheses in labels search SQL (#644)
Operator precedence was causing incorrect results to be returned. The failing
test was missed because the CI results did not appear in Github before merging.
2016-12-14 19:24:53 -08:00
Zachary Wasserman
b95c1dd6a0 Always return "All Hosts" label in targets unless selected (#637)
Closes #467
2016-12-14 18:27:22 -08:00
Mike Arpaia
1ada011f37 Remove existing bulk functionality (#641) 2016-12-14 17:22:17 -08:00
Mike Arpaia
56b3799457 Add forgotten JSON serialization tag (#640) 2016-12-14 14:29:20 -08:00
Zachary Wasserman
0e783e7887 Add change password endpoint (#628) 2016-12-14 10:11:43 -08:00
Mike Arpaia
bed279b919 Remove "selected" keyword from response (#635) 2016-12-13 15:59:59 -08:00
Mike Arpaia
f109b14f9d Moving query attributes from the query object to the pack-query relationship (#559)
* Moving query attributes from the query object to the pack-query relationship

* some additional tests

* http request parsing test

* QueryOptions in new test_util code

* initial scaffolding of new request structures

* service and datastore

* test outline

* l2 merge conflict scrub

* service tests for scheduled query service

* service and datastore tests

* most endpoints and transports

* order of values are not deterministic with inmem

* transport tests

* rename PackQuery to ScheduledQuery

* removing existing implementation of adding queries to packs

* accounting for the new argument to NewQuery

* fix alignment in sql query

* removing underscore

* add removed to the datastore

* removed differential from the schema
2016-12-13 14:22:05 -08:00
Zachary Wasserman
556cbf43bd Use canPerformActions authZ check appropriately in handlers (#625)
Fixes #282
2016-12-13 08:57:49 -08:00
Zachary Wasserman
0c62bda2f1 Fix session expiration logic with password reset (#619)
- Only expire sessions at reset request time when admin forces reset
- Expire sessions when reset completed

Prior to this, there was a possible DoS vector in which an attacker could
prevent a user from taking actions in the app by constantly requesting password
resets and expiring all the user's active sessions.

Fixes #612
2016-12-12 09:24:51 -08:00
Zachary Wasserman
b362682d90 Add service method for bulk deleting queries (#600)
- New datastore method for bulk deletion
- New service method calling this datastore method
- Endpoint, transport and handler connections for service method

Closes #389
2016-12-09 09:12:45 -08:00
Zachary Wasserman
d9190020fe Record author information with queries (#578) 2016-12-07 12:22:31 -08:00
Victor Vrantchan
6d9ad56e6d use validation error for expired invite tokens. (#587)
Closes #586
2016-12-07 11:25:48 -05:00
Victor Vrantchan
02945f1e3b set created_at time for invites in inmem (#588) 2016-12-07 11:20:21 -05:00
Victor Vrantchan
62a5e65c2e use random string instead of JWT for tokens. (#584)
uses a random URL encoded base64 string as the token for password reset
and invites.
2016-12-07 10:42:58 -05:00
John Murphy
7812b2f3bd Fixes host detail updates with MySQL backend (#568) 2016-12-07 03:51:11 +08:00
Zachary Wasserman
691eb55cf2 Return packs with queries (#575)
- New datastore method for loading packs associated with a query
- ListQueries and Query datastore methods now load packs

Addresses #388
2016-12-06 10:22:28 -08:00
Zachary Wasserman
bc6109fb56 Fixes for saved queries (#576)
- Only saved queries should be returned by ListQueries
- Bugfixes

Addresses #388
2016-12-06 10:16:04 -08:00
John Murphy
e6b1ed9ade Added MIA status for hosts that haven't been updated for 30 days (#570) 2016-12-07 01:37:22 +08:00
Zachary Wasserman
51e59712da Fixes in targets search (#565)
- Remove string formatting in favor of bindvars
- Fix errors when the query is empty
2016-12-05 11:16:23 -08:00
Victor Vrantchan
b1bca7c98c enbable API route after setup is complete (#564)
Fixes #563
2016-12-02 13:46:31 -05:00
Zachary Wasserman
b901c4c0d3 Update models/service method for saving queries (#553)
- Add saved state to query (to differentiate queries explicitly saved from
  those just run as distributed queries)
- Remove unique constraint on query name

Closes #390
2016-12-01 13:21:27 -08:00
Zachary Wasserman
e1db2d4c27 Add cleanup method for distributed queries and metadata (#547)
- New datastore method for expiring queries and deleting metadata
- Start goroutine in serve to run this method on an hourly interval

Closes #392
2016-12-01 10:31:16 -08:00
John Murphy
f66904f240 Host details 405 (#535) 2016-12-02 01:00:00 +08:00
Zachary Wasserman
ed3c696c6e Refactor campaign methods into separate file (#540) 2016-11-29 10:20:06 -08:00
Victor Vrantchan
a9000c5af3 print connection sleep interval duration in seconds (#534)
The duration was printed with a "ns" interval, even
though the sleep time was in seconds.
2016-11-28 10:35:05 -05:00
John Murphy
44ef92550f Added built in labels (#526) 2016-11-26 02:08:22 +08:00
Zachary Wasserman
0b612eedab Push query results and metadata over websocket connection (#519)
- New route `/api/v1/kolide/results/{id}` with upgrade to websocket connection
- Query results pushed over websocket as they are received from pubsub
- Target totals updates pushed over websocket every second
- New datastore method to support retrieiving target totals
- Websocket package includes helpers and patterns for communicating over websockets
2016-11-22 16:35:43 -08:00
Mike Arpaia
a036c5da9f Add attributes to packs (#524)
* Adds created_by attribute to packs

This PR also updated the distributed query code to use the pattern
established here (service checks context)

* add enable/disable state to packs

* add query_count to packs API responses

* add host_count to packs API responses (very, very poorly)

* pack description should not be required

* counting hosts in packs via mysql

* removing extraneous newline in test

* Switch case instead of if/if else

* add description to update query for SavePack method

* change AND to WHERE in query as per @zwass

* add ordering and list options as per @murphybytes' suggestion
2016-11-22 13:56:05 -08:00
Mike Arpaia
a8a7be7f20 Packs pages (#426)
* initial scaffolding

* pack info sidebar

* fixing the merge of the routes

* Remove radium from pack info sidepanel

* lint

* cards!

* redux entity config

* pack interface

* wiring up redux with fake dev data

* Add description attribute to packs

* move redux to top level page component to isolate data fetching

* initial scaffolding of all packs table

* adding redux entities back

* minimal

* alpha order in packs.js

* no newlines in HTML

* onclick handler to function on component class

* alpha order in router

* alpha order in paths.js

* no newline in side panel

* removing input field

* lint fixes
2016-11-21 11:49:36 -08:00
Zachary Wasserman
627b1ad6e6 Drop MySQL tables after test when migration fails (#517) 2016-11-18 11:14:10 -08:00
Zachary Wasserman
3cd841051b Add online count to target/label responses (#513)
Closes #508
2016-11-18 09:23:44 -08:00
John Murphy
531535c168 Refactored database migrations to use goose (#515)
* Refactored database migrations to use goose
2016-11-19 01:02:51 +08:00
Zachary Wasserman
dd16f6d918 Fix missing JSON tag for labels (#510) 2016-11-17 16:51:30 -08:00
Zachary Wasserman
eff7f60dc2 Fixes for bugs in MySQL migration (#501)
* Fix users table name in MySQL ListUsers
* Fix invalid SQL
* Implement MarkHostSeen
* Partial fix for targets autocompletion
2016-11-16 15:12:59 -08:00
Zachary Wasserman
34625ce4d0 Add service method/endpoint for creating distributed query campaign (#485) 2016-11-16 13:07:50 -08:00
Mike Arpaia
eb3f3cd765 removing some dependencies from glide (#493) 2016-11-16 10:38:31 -08:00
Mike Arpaia
587d5c9a09 removing gorm struct tags (#491) 2016-11-16 09:48:43 -08:00
John Murphy
6a825c11e3 Datastore refactor (#439)
Removed Gorm, replaced it with Sqlx

* Added SQL bundling command to Makfile

* Using go-kit logger

* Added soft delete capability

* Changed SearchLabel to accept a variadic param for optional omit list
instead of array

* Gorm removed

* Refactor table structures to use CURRENT_TIMESTAMP mysql function

* Moved Inmem datastore into it's own package

* Updated README

* Implemented code review suggestions from @zwass

* Removed reference to Gorm from glide.yaml
2016-11-16 21:47:49 +08:00
Mike Arpaia
752710ccf1 Return count of hosts in targets search for each label (#480) 2016-11-14 12:10:28 -08:00
Zachary Wasserman
9c38d6d19e Update osquery service methods for distributed queries (#476) 2016-11-14 10:22:54 -08:00
Victor Vrantchan
2a9e27e426 add mysql and redis services (#471)
enable redis and mysql tests in CircleCI environment
switch redis test to use UTC time - https://github.com/golang/go/issues/17885
2016-11-10 21:04:26 -05:00
Zachary Wasserman
0b6df0cd73 Fix changed ID column name in DistributedQueriesForHost query (#473) 2016-11-10 17:01:39 -08:00
Zachary Wasserman
5d8a42a959 Fix page in RequireSetup check (#465)
The previous value would require setup unless there were at least 2 users. Now
one user is sufficient to proceed past setup.
2016-11-09 16:35:49 -08:00
Zachary Wasserman
af6a5ee1b6 Add datastore methods for distributed queries (#458)
New datastore methods are introduced for creating/updating
distributed query campaigns, as well as determining the active
distributed queries for a given host.
2016-11-09 15:33:16 -08:00
Zachary Wasserman
c8b9b15ef2 Fix failing MySQL tests for packs (#457) 2016-11-09 15:31:47 -08:00
Victor Vrantchan
ac14215e21 create first time setup endpoint (#436)
The endpoint is only active if there are no users in the datastore.
While the endpoint is active, it also disables all the other API endpoints, and /config returns `{"require_setup":true}`
for #378
2016-11-09 12:19:07 -05:00
Zachary Wasserman
0388297fd8 Fix missing primary key designation in DistributedQueryExecutions (#450) 2016-11-09 09:09:29 -08:00
Zachary Wasserman
675c40ea15 Cleanup in service_users_test (#460)
- Use subtests where appropriate
- Attempt to fix #445
2016-11-09 08:52:25 -08:00
Zachary Wasserman
991fd6c146 Ensure tables are dropped after panic in test (#456)
Prior to this, a panic in the test function would leave data in the database.
Now we ensure that tables are dropped even after a panic.
2016-11-08 13:46:36 -08:00
Victor Vrantchan
def24499b5 store WebAddress config in datastore (#421)
moves web address config to datastore so that it can be configured by a user
in the Web UI.
rename OrgInfo struct to AppConfig.

For #363
For #378
2016-11-04 16:44:38 -04:00
Victor Vrantchan
b518447b31 correctly save user properties during ModifyUser call. (#395)
fixes #351

because RequestPasswordReset does it's own db query/save,
call it at the end, after all properties have already been saved.
2016-11-03 12:50:32 -04:00
Mike Arpaia
97de2bc5aa Label API Updates (#413)
close #403
close #404
close #412
2016-11-02 21:17:23 -04:00
Mike Arpaia
7ebebbb7b1 Target search endpoint (#339) 2016-11-02 10:59:53 -04:00
Zachary Wasserman
262a48f8eb Datastores for buffering distributed query results (#346)
A new datastore interface is needed for buffering incoming distributed query results to be sent to the client. This PR attempts to define and implement that interface.

It is intended that the ReadChannel() method be used by the goroutine that will push query results down a websocket to the client. Passing the results through this channel will allow that goroutine to perform a select on both the channel and the websocket, in order to properly handle IO.
2016-10-31 15:51:19 -07:00
Zachary Wasserman
0faa3a55fb Make context imports consistent (#359) 2016-10-31 15:18:05 -07:00
Mike Arpaia
c5339ddb76 Fixing bug where non-updated query object is returned (#361) 2016-10-31 18:00:04 -04:00
Mike Arpaia
4f83220870 Add description field to Query (#358) 2016-10-31 17:05:02 -04:00
Victor Vrantchan
950586c100 update datastore logger (#344)
remove dependency on logrus
adapt go-kit/log interface to gorm logger
allow debug logging flag
2016-10-25 11:30:14 -04:00
Victor Vrantchan
efffe50a55 update mysql test setup (#343)
reuse a single conneciton pool
close connection gracefully at the end of tests
setup and teardown tables in between each test
2016-10-25 11:01:14 -04:00
Zachary Wasserman
d2e31146c6 Fix bugs with password resets (#330)
- Permissions error with admin forced password reset
- Redirecting on successful admin forced password reset
- URL fix for forgot password reset
- Use JWT key for password reset request
2016-10-20 15:09:51 -04:00
Mike Arpaia
bdee0516e6 Removing sqlite from the codebase (#324) 2016-10-19 10:46:33 -04:00
Mike Arpaia
98ca32b783 Complete, working inmem datastore implementation (#319) 2016-10-17 15:30:47 -04:00
Zachary Wasserman
05cb15d63c Fix pointer handling in inmem nextID (#323) 2016-10-17 13:11:23 -04:00
Zachary Wasserman
f9bb9de665 Add ordering options for List* methods (#318) 2016-10-17 10:01:14 -04:00
Zachary Wasserman
1597ad23a4 Fix NewHost bug introduced in #312 (#317) 2016-10-16 07:56:20 -04:00
Mike Stone
beda051de2 Show invited users (#297)
* Adds loadAll action to redux entity config

* API Client get invites

* Add invites to the user management page

* Updates user block styles on user management page

* Submit modal form on enter

* Modify details form styles

* Enter submits edit user form

* Removes unused admin dashboard page

* API Client - revoke invites

* Delete invite entities in redux

* Revoke invites from admin manage users page

* Show success flash message after user invite is revoked
2016-10-14 17:08:57 -04:00
Zachary Wasserman
daeecfb244 Fix bug in inmem ID generation (#312)
Fixes #308
2016-10-14 09:15:04 -07:00
Zachary Wasserman
d9b98b22d6 Minor cleanup in service_invites_test (#313) 2016-10-14 09:01:26 -07:00
Zachary Wasserman
4b88ae6e2c Apply consistent naming conventions across server files (#310) 2016-10-14 08:59:27 -07:00
Zachary Wasserman
7f636aef4f Add pagination to List* endpoints (#309)
- Introduce kolide.ListOptions to store pagination params (in the future it can
  also store ordering/filtering params)
- Refactor service/datastore methods to take kolide.ListOptions
- Implement pagination
2016-10-13 11:21:47 -07:00
Zachary Wasserman
f9fa3e289f Return host 'status' with host endpoints (#307) 2016-10-12 14:41:35 -07:00
Victor Vrantchan
93eaae7454 Lowercase strings which should be case insensitive, like email and username. (#302)
Fixes #299
Closes #300
2016-10-12 12:35:34 -04:00
Zachary Wasserman
5d0cac882a Move API responses under top-level keys (#292) 2016-10-11 09:22:11 -07:00
Zachary Wasserman
24bb68b910 Rename GetAll* methods to List* (#291) 2016-10-07 10:26:48 -07:00
Zachary Wasserman
1cce02cf86 Fix bug causing empty lists to return as null in GetAll/List endpoints (#286)
Fixes #285
2016-10-06 09:24:12 -07:00
Victor Vrantchan
7221aac50f only set AdminForcedPasswordReset if payload value is true (#289)
Fixes #287
2016-10-06 08:43:09 -04:00