From f161a3c12cb3fa7b8bf904401c4473c866379ddc Mon Sep 17 00:00:00 2001 From: Zachary Wasserman Date: Thu, 3 May 2018 16:57:26 -0700 Subject: [PATCH 1/6] Update changelog for 1.0.8 release (#1753) --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 801717bd07..d41a8b8b9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,9 @@ +## Kolide Fleet 1.0.8 (May 3, 2018) + +* Osquery 3.0+ compatibility! + +* Include RFC822 From header in emails (for email authentication) + ## Kolide Fleet 1.0.7 (Mar 30, 2018) * Support FileAccesses in FIM configuration. From d1a2c2569758257c7ebe9917ce3398fcd5f0c5d6 Mon Sep 17 00:00:00 2001 From: dso <1205295+deeso@users.noreply.github.com> Date: Fri, 4 May 2018 11:56:51 -0500 Subject: [PATCH 2/6] adding the linux development setup (#1710) --- docs/development/linux.md | 52 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docs/development/linux.md diff --git a/docs/development/linux.md b/docs/development/linux.md new file mode 100644 index 0000000000..00134ac5c4 --- /dev/null +++ b/docs/development/linux.md @@ -0,0 +1,52 @@ +## Setting up a Linux Development Environment + +### Install some dependencies + +`sudo apt-get install xzip gyp libjs-underscore libuv1-dev dep11-tools deps-tools-cli` + +### Create a temp directory, download and place the `node` and `golang` bins + +``` +mkdir tmp +cd tmp +``` + +#### install `node` and `yarn` + +``` +wget https://nodejs.org/dist/v9.4.0/node-v9.4.0-linux-x64.tar.xz +xz -d node-v9.4.0-linux-x64.tar.xz +tar -xf node-v9.4.0-linux-x64.tar +sudo cp -rf node-v9.4.0-linux-x64/bin /usr/local/ +sudo cp -rf node-v9.4.0-linux-x64/include /usr/local +sudo cp -rf node-v9.4.0-linux-x64/lib /usr/local +sudo cp -rf node-v9.4.0-linux-x64/share /usr/local +npm install -g yarn +``` + +#### install `go` + +``` +wget https://dl.google.com/go/go1.9.3.linux-amd64.tar.gz +sudo tar -C /usr/local -xzf go1.9.3.linux-amd64.tar.gz +export PATH=$PATH:/usr/local/go/bin:~/go/bin/ +``` + +#### clean-up temp directory + +``` +cd .. +rm -rf tmp +``` + +### Clone and build depenencies + +``` +mkdir -p ~/go/src/github.com/kolide/ +git clone https://github.com/kolide/fleet.git +cd fleet +make deps +make generate +make build +sudo cp build/fleet /usr/bin/fleet +``` From 2482176d90130d62d0178d363e7e047f183bbdac Mon Sep 17 00:00:00 2001 From: Victor Vrantchan Date: Fri, 4 May 2018 13:01:45 -0400 Subject: [PATCH 3/6] update ubuntu docs and add notes on systemd (#1702) closes #1593 --- docs/infrastructure/fleet-on-centos.md | 5 +++ docs/infrastructure/fleet-on-ubuntu.md | 12 +++++-- docs/infrastructure/systemd.md | 47 ++++++++++++++++++++++++++ 3 files changed, 62 insertions(+), 2 deletions(-) create mode 100644 docs/infrastructure/systemd.md diff --git a/docs/infrastructure/fleet-on-centos.md b/docs/infrastructure/fleet-on-centos.md index 2b4e97b43b..33a5accf69 100644 --- a/docs/infrastructure/fleet-on-centos.md +++ b/docs/infrastructure/fleet-on-centos.md @@ -135,6 +135,11 @@ $ /usr/bin/fleet serve \ Now, if you go to [https://localhost:8080](https://localhost:8080) in your local browser, you should be redirected to [https://localhost:8080/setup](https://localhost:8080/setup) where you can create your first Fleet user account. +## Running Fleet with systemd + +See [systemd](./systemd.md) for documentation on running fleet as a background process and managing the fleet server logs. + + ## Installing and running osquery > Note that this whole process is outlined in more detail in the [Adding Hosts To Fleet](./adding-hosts-to-fleet.md) document. The steps are repeated here for the sake of a continuous tutorial. diff --git a/docs/infrastructure/fleet-on-ubuntu.md b/docs/infrastructure/fleet-on-ubuntu.md index 54772844db..6adcf68f8e 100644 --- a/docs/infrastructure/fleet-on-ubuntu.md +++ b/docs/infrastructure/fleet-on-ubuntu.md @@ -122,6 +122,11 @@ $ /usr/bin/fleet serve \ Now, if you go to [https://localhost:8080](https://localhost:8080) in your local browser, you should be redirected to [https://localhost:8080/setup](https://localhost:8080/setup) where you can create your first Fleet user account. +## Running Fleet with systemd + +See [systemd](./systemd.md) for documentation on running fleet as a background process and managing the fleet server logs. + + ## Installing and running osquery > Note that this whole process is outlined in more detail in the [Adding Hosts To Fleet](./adding-hosts-to-fleet.md) document. The steps are repeated here for the sake of a continuous tutorial. @@ -129,12 +134,15 @@ Now, if you go to [https://localhost:8080](https://localhost:8080) in your local To install osquery on Ubuntu, you can run the following: ``` -$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B -$ sudo add-apt-repository "deb [arch=amd64] https://osquery-packages.s3.amazonaws.com/xenial xenial main" +$ export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B +$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $OSQUERY_KEY +$ sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main' $ sudo apt-get update $ sudo apt-get install osquery ``` +If you're having trouble with the above steps, check the official [downloads](https://osquery.io/downloads) link for a direct download of the .deb. + You will need to set the osquery enroll secret and osquery server certificate. If you head over to the manage hosts page on your Fleet instance (which should be [https://localhost:8080/hosts/manage](https://localhost:8080/hosts/manage)), you should be able to click "Add New Hosts" and see a modal like the following: ![Add New Host](../images/add-new-host-modal.png) diff --git a/docs/infrastructure/systemd.md b/docs/infrastructure/systemd.md new file mode 100644 index 0000000000..32ae263f1b --- /dev/null +++ b/docs/infrastructure/systemd.md @@ -0,0 +1,47 @@ +## Running with systemd + +Once you've verified that you can run fleet in your shell, you'll likely want to keep fleet running in the background and after the server reboots. To do that we recommend using [systemd](https://coreos.com/os/docs/latest/getting-started-with-systemd.html). + +Below is a sample unit file. + +``` +[Unit] +Description=Kolide Fleet +After=network.target + +[Service] +ExecStart=/usr/local/bin/fleet serve \ + --mysql_address=127.0.0.1:3306 \ + --mysql_database=kolide \ + --mysql_username=root \ + --mysql_password=toor \ + --redis_address=127.0.0.1:6379 \ + --server_cert=/tmp/server.cert \ + --server_key=/tmp/server.key \ + --auth_jwt_key=this_string_is_not_secure_replace_it \ + --logging_json + +[Install] +WantedBy=multi-user.target +``` + +Once you created the file, you need to move it to `/etc/systemd/system/fleet.service` and start the service. + +``` +sudo mv fleet.service /etc/systemd/system/fleet.service +sudo systemctl start fleet.service +sudo systemctl status fleet.service + +sudo journalctl -u fleet.service -f +``` + +## Making changes + +Sometimes you'll need to update the systemd unit file defining the service. To do that, first open /etc/systemd/system/fleet.service in a text editor, and make your modifications. + +Then, run + +``` +sudo systemctl daemon-reload +sudo systemctl restart fleet.service +``` From 12922a89ccdbf7ac9232250b33e49001a6c4c0b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Donovan=20Sch=C3=B6nknecht?= Date: Fri, 4 May 2018 19:10:17 +0200 Subject: [PATCH 4/6] Remove default timestamp to fix strict error (#1720) --- .../tables/20161118212641_CreateTablePasswordResetRequests.go | 2 +- server/datastore/mysql/password_reset.go | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/server/datastore/mysql/migrations/tables/20161118212641_CreateTablePasswordResetRequests.go b/server/datastore/mysql/migrations/tables/20161118212641_CreateTablePasswordResetRequests.go index 05b05638ab..fc88041c1d 100644 --- a/server/datastore/mysql/migrations/tables/20161118212641_CreateTablePasswordResetRequests.go +++ b/server/datastore/mysql/migrations/tables/20161118212641_CreateTablePasswordResetRequests.go @@ -12,9 +12,9 @@ func Up_20161118212641(tx *sql.Tx) error { _, err := tx.Exec( "CREATE TABLE `password_reset_requests` (" + "`id` int(10) unsigned NOT NULL AUTO_INCREMENT," + + "`expires_at` timestamp NOT NULL," + "`created_at` timestamp DEFAULT CURRENT_TIMESTAMP," + "`updated_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP," + - "`expires_at` timestamp NOT NULL DEFAULT '1970-01-01 00:00:01'," + "`user_id` int(10) unsigned NOT NULL," + "`token` varchar(1024) NOT NULL," + "PRIMARY KEY (`id`)" + diff --git a/server/datastore/mysql/password_reset.go b/server/datastore/mysql/password_reset.go index a2a14e93d7..c6304401b3 100644 --- a/server/datastore/mysql/password_reset.go +++ b/server/datastore/mysql/password_reset.go @@ -8,8 +8,8 @@ import ( func (d *Datastore) NewPasswordResetRequest(req *kolide.PasswordResetRequest) (*kolide.PasswordResetRequest, error) { sqlStatement := ` INSERT INTO password_reset_requests - ( user_id, token) - VALUES (?,?) + ( user_id, token, expires_at) + VALUES (?,?, NOW()) ` response, err := d.db.Exec(sqlStatement, req.UserID, req.Token) if err != nil { From 178cab530ff8cbcff35c93fdac83485ac0a25b3b Mon Sep 17 00:00:00 2001 From: Mike Arpaia Date: Fri, 4 May 2018 11:39:36 -0600 Subject: [PATCH 5/6] Update the CLI docs on the master branch (#1756) --- docs/cli/README.md | 175 +++++++++++++++++++++++---------------------- 1 file changed, 89 insertions(+), 86 deletions(-) diff --git a/docs/cli/README.md b/docs/cli/README.md index ce51125ebb..a3ec43c443 100644 --- a/docs/cli/README.md +++ b/docs/cli/README.md @@ -50,32 +50,19 @@ Find more information at https://kolide.com/fleet ### Workflow ```bash -# Make sure you're currently using the current server (in this case: staging) -fleetctl config set-context staging +# Make sure you're currently using the current server (in this case: production linux hosts) +fleetctl config set-context production-linux # Edit the config file (or files) for your Fleet instance (or one of them) and apply the file -vim fleet-staging.yml -fleetctl apply -f ./fleet-staging.yml +vim fleet-linux.yml +fleetctl apply -f ./fleet-linux.yml # Commit the changes to an upstream source tree -git add fleet-staging.yml -git commit -m "new changes to staging fleet instance" +git add fleet-linux.yml +git commit -m "new changes to osquery production linux configuration" git push ``` -Alternatively, you can specify the context as a flag for easy use in parallel scripts or instances where you may have many Fleet environments: - -```bash -# Edit your Fleet config file -vim fleet.yml - -# First apply the configuration to your staging environment for testing -fleetctl apply -f ./fleet.yml --context=staging - -# Apply the configuration to both staging and production at the same time -fleetctl apply -f ./fleet.yml --context=staging,production -``` - ## Configuration File Format A Fleet configuration is defined using one or more declarative "messages" in yaml syntax. Each message can live in it's own file or multiple in one file, each separated by `---`. Each file/message contains a few required top-level keys: @@ -93,11 +80,10 @@ When you reason about how to manage these config files, consider following the [ - Group related objects into a single file whenever it makes sense. One file is often easier to manage than several. See the [config-single-file.yml](../../examples/config-single-file.yml) file as an example of this syntax. - Don’t specify default values unnecessarily – simple and minimal configs will reduce errors. -All of these files can be concatenated together into [one file](../../examples/config-single-file.yml) (separated by `---`), or they can be in [individual files with a directory structure](../../examples/config-many-files) like the following: +All of these files can be concatenated together into [one file](../../examples/config-single-file.yml) (seperated by `---`), or they can be in [individual files with a directory structure](../../examples/config-many-files) like the following: ``` |-- config.yml -|-- decorators.yml |-- labels.yml |-- packs | `-- osquery-monitoring.yml @@ -109,55 +95,74 @@ All of these files can be concatenated together into [one file](../../examples/c The following file describes configuration options passed to the osquery instance. All other configuration data will be over-written by the application of this file. ```yaml -apiVersion: k8s.kolide.com/v1alpha1 -kind: Options +apiVersion: kolide.com/v1alpha1 +kind: OsqueryOptions spec: config: - distributed_interval: 3 - distributed_tls_max_attempts: 3 - logger_plugin: tls - logger_tls_endpoint: /api/v1/osquery/log - logger_tls_period: 10 + options: + distributed_interval: 3 + distributed_tls_max_attempts: 3 + logger_plugin: tls + logger_tls_endpoint: /api/v1/osquery/log + logger_tls_period: 10 + decorators: + load: + - "SELECT version FROM osquery_info" + - "SELECT uuid AS host_uuid FROM system_info" + always: + - "SELECT user AS username FROM logged_in_users WHERE user <> '' ORDER BY time LIMIT 1" + interval: + 3600: "SELECT total_seconds AS uptime FROM uptime" overrides: - # Note configs in overrides take precedence over base configs + # Note configs in overrides take precedence over the default config defined + # under the config key above. With this config file, the base config would + # only be used for Windows hosts, while Mac and Linux hosts would pull + # these overrides. platforms: darwin: - disable_tables: chrome_extensions - docker_socket: /var/run/docker.sock - logger_tls_period: 60 - fim: - interval: 500 - groups: - - name: etc - paths: - - /etc/%% - - name: users - paths: - - /Users/%/Library/%% - - /Users/%/Documents/%% + options: + distributed_interval: 10 + distributed_tls_max_attempts: 10 + logger_plugin: tls + logger_tls_endpoint: /api/v1/osquery/log + logger_tls_period: 300 + disable_tables: chrome_extensions + docker_socket: /var/run/docker.sock + file_paths: + users: + - /Users/%/Library/%% + - /Users/%/Documents/%% + etc: + - /etc/%% + linux: - schedule_timeout: 60 - docker_socket: /etc/run/docker.sock -``` - -### Osquery Logging Decorators - -The following file describes logging decorators that should be applied on osquery instances. A decorator should reference an osquery query by name. Both of these resources can be included in the same file as such: - -```yaml ---- -apiVersion: k8s.kolide.com/v1/alpha1 -kind: Query -spec: - name: hostname - query: select hostname from system_info ---- -apiVersion: k8s.kolide.com/v1alpha1 -kind: Decorator -spec: - query: hostname - type: interval - interval: 10 + options: + distributed_interval: 10 + distributed_tls_max_attempts: 3 + logger_plugin: tls + logger_tls_endpoint: /api/v1/osquery/log + logger_tls_period: 60 + schedule_timeout: 60 + docker_socket: /etc/run/docker.sock + file_paths: + homes: + - /root/.ssh/%% + - /home/%/.ssh/%% + etc: + - /etc/%% + tmp: + - /tmp/%% + exclude_paths: + homes: + - /home/not_to_monitor/.ssh/%% + tmp: + - /tmp/too_many_events/ + decorators: + load: + - "SELECT * FROM cpuid" + - "SELECT * FROM docker_info" + interval: + 3600: "SELECT total_seconds AS uptime FROM uptime" ``` ### Host Labels @@ -165,9 +170,14 @@ spec: The following file describes the labels which hosts should be automatically grouped into. The label resource should reference the query by name. Both of these resources can be included in the same file as such: ```yaml +apiVersion: kolide.com/v1alpha1 +kind: OsqueryLabel +spec: + name: slack_not_running + query: slack_not_running --- -apiVersion: k8s.kolide.com/v1/alpha1 -kind: Query +apiVersion: kolide.com/v1/alpha1 +kind: OsqueryQuery spec: name: slack_not_running query: > @@ -177,12 +187,6 @@ spec: FROM processes WHERE name LIKE "%Slack%" ); ---- -apiVersion: k8s.kolide.com/v1alpha1 -kind: Label -spec: - name: slack_not_running - query: slack_not_running ``` ### Osquery Queries @@ -190,8 +194,8 @@ spec: For especially long or complex queries, you may want to define one query in one file. Continued edits and applications to this file will update the query as long as the `metadata.name` does not change. If you want to change the name of a query, you must first create a new query with the new name and then delete the query with the old name. Make sure the old query name is not defined in any packs before deleting it or an error will occur. ```yaml -apiVersion: k8s.kolide.com/v1alpha1 -kind: Query +apiVersion: kolide.com/v1alpha1 +kind: OsqueryQuery spec: name: docker_processes descriptions: The docker containers processes that are running on a system. @@ -203,12 +207,11 @@ spec: - darwin ``` -To define multiple queries in a file, concatenate multiple `Query` resources together in a single file with `---`. For example, consider a file that you might store at `queries/osquery_monitoring.yml`: +To define multiple queries in a file, concatenate multiple `OsqueryQuery` resources together in a single file with `---`. For example, consider a file that you might store at `queries/osquery_monitoring.yml`: ```yaml ---- -apiVersion: k8s.kolide.com/v1alpha1 -kind: Query +apiVersion: kolide.com/v1alpha1 +kind: OsqueryQuery spec: name: osquery_version description: The version of the Launcher and Osquery process @@ -217,22 +220,22 @@ spec: launcher: 0.3.0 osquery: 2.9.0 --- -apiVersion: k8s.kolide.com/v1alpha1 -kind: Query +apiVersion: kolide.com/v1alpha1 +kind: OsqueryQuery spec: name: osquery_schedule description: Report performance stats for each file in the query schedule. query: select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory, last_executed from osquery_schedule; --- -apiVersion: k8s.kolide.com/v1alpha1 -kind: Query +apiVersion: kolide.com/v1alpha1 +kind: OsqueryQuery spec: name: osquery_info description: A heartbeat counter that reports general performance (CPU, memory) and version. query: select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid; --- -apiVersion: k8s.kolide.com/v1alpha1 -kind: Query +apiVersion: kolide.com/v1alpha1 +kind: OsqueryQuery spec: name: osquery_events description: Report event publisher health and track event counters. @@ -244,8 +247,8 @@ spec: To define query packs, reference queries defined elsewhere by name. This is why the "name" of a query is so important. You can define many of these packs in many files. ```yaml -apiVersion: k8s.kolide.com/v1alpha1 -kind: Pack +apiVersion: kolide.com/v1alpha1 +kind: OsqueryPack spec: name: osquery_monitoring targets: From 4e8c3482c87ced108e07299fb12f08dd60270abd Mon Sep 17 00:00:00 2001 From: Caleb Coy Date: Fri, 4 May 2018 13:10:22 -0700 Subject: [PATCH 6/6] Fleet on ubuntu docs (#1757) --- docs/infrastructure/fleet-on-ubuntu.md | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/docs/infrastructure/fleet-on-ubuntu.md b/docs/infrastructure/fleet-on-ubuntu.md index 6adcf68f8e..ab6756042c 100644 --- a/docs/infrastructure/fleet-on-ubuntu.md +++ b/docs/infrastructure/fleet-on-ubuntu.md @@ -23,7 +23,7 @@ To install Fleet, run the following: ``` $ wget https://dl.kolide.co/bin/fleet_latest.zip $ unzip fleet_latest.zip 'linux/*' -d fleet -$ sudo cp fleet/linux/fleet /usr/bin/fleet +$ sudo cp fleet/linux/fleet_linux_amd64 /usr/bin/fleet ``` ## Installing and configuring dependencies @@ -36,7 +36,7 @@ To install the MySQL server files, run the following: $ sudo apt-get install mysql-server -y ``` -When asked for MySQL's root password, enter `toor` for the sake of this tutorial if you are having trouble thinking of a better password for the MySQL root user. +When asked for MySQL's root password, enter `toor` for the sake of this tutorial if you are having trouble thinking of a better password for the MySQL root user. If you decide to set your own password, be mindful that you will need to substitute it every time `toor` is used in this document. After installing `mysql-server`, the `mysqld` server should be running. You can verify this by running the following: @@ -81,18 +81,9 @@ $ /usr/bin/fleet prepare db \ The output should look like: -``` -OK 20161118193812_CreateTableAppConfigs.go -OK 20161118211713_CreateTableDistributedQueryCampaignTargets.go -... -OK 20170124230432_CreateTableEmailChanges.go -goose: no migrations to run. current version: 20170124230432 -OK 20161223115449_InsertOsqueryOptions.go -OK 20161229171615_InsertBuiltinLabels.go -goose: no migrations to run. current version: 20161229171615 -``` +`Migrations completed` -Before we can run the server, we need to generate some TLS keying material. If you already have tooling for generating valid TLS certificates, then you are encouraged to use that instead. You will need a TLS certificate and key for running the Fleet server. If you'd like to generate self-signed certificates, you can do this via: +Before we can run the server, we need to generate some TLS keying material. If you already have tooling for generating valid TLS certificates, then you are encouraged to use that instead. You will need a TLS certificate and key for running the Fleet server. If you'd like to generate self-signed certificates, you can do this via the following steps (note - you will be asked for severl bits of information, including name, contact info, and location, in order to generate the certificate): ``` $ openssl genrsa -out /tmp/server.key 4096 @@ -119,6 +110,7 @@ $ /usr/bin/fleet serve \ --server_key=/tmp/server.key \ --logging_json ``` +You will be prompted to add a value for `--auth_jwt_key`. A randomly generated key will be suggested, you can simply add the flag with the sugested key. Now, if you go to [https://localhost:8080](https://localhost:8080) in your local browser, you should be redirected to [https://localhost:8080/setup](https://localhost:8080/setup) where you can create your first Fleet user account.