fleet/docs/infrastructure/faq.md

FAQ for using/operating Fleet

Can multiple instances of the Fleet server be run behind a load-balancer?

Yes. Fleet scales horizontally out of the box as long as all of the Fleet servers are connected to the same MySQL and Redis instances.

Note that osquery logs will be distributed across the Fleet servers.

Where are my query results?

Live Queries

Live query results (executed in the web UI or fleetctl query) are pushed directly to the UI where the query is running. The results never go to a file unless you as the user manually save them.

Scheduled Queries

Scheduled query results (queries that are scheduled to run in Packs) are typically sent to the Fleet server, and will be available on the filesystem of the server at the path configurable by --osquery_result_log_file. This defaults to /tmp/osquery_result.

It is possible to configure osqueryd to log query results outside of Fleet. For results to go to Fleet, the --logger_plugin flag must be set to tls.

Troubleshooting

Expecting results, but not seeing anything in the logs?

• Try scheduling a query that always returns results (eg. SELECT * FROM time).
• Check whether the query is scheduled in differential mode. If so, new results will only be logged when the result set changes.
• Ensure that the query is scheduled to run on the intended platforms, and that the tables queried are supported by those platforms.
• Use live query to SELECT * FROM osquery_schedule to check whether the query has been scheduled on the host.
• Look at the status logs provided by osquery. In a standard configuration these are available on the filesystem of the Fleet server at the path configurable by --filesystem_status_log_file. This defaults to /tmp/osquery_status. The host will output a status log each time it executes the query.

Why aren’t my live queries being logged?

Live query results are never logged to the filesystem of the Fleet server. See Where are my query results?.

Why aren't my osquery agents connecting to Fleet?

This can be caused by a variety of problems. The best way to debug is usually to add --verbose --tls_dump to the arguments provided to osqueryd and look at the logs for the server communication.

Common problems

• Connection refused: The server is not running, or is not listening on the address specified. Is the server listening on an address that is available from the host running osquery? Do you have a load balancer that might be blocking connections? Try testing with curl.
• No node key returned: Typically this indicates that the osquery client sent an incorrect enroll secret that was rejected by the server. Check what osquery is sending by looking in the logs near this error.
• certificate verify failed: See How do I fix "certificate verify failed" errors from osqueryd.
• bad record MAC: When generating your certificate for your Fleet server, ensure you set the hostname to the FQDN or the IP of the server. This error is common when setting up Fleet servers and accepting defaults when generating certificates using openssl.

How do I fix "certificate verify failed" errors from osqueryd?

Osquery requires that all communication between the agent and Fleet are over a secure TLS connection. For the safety of osquery deployments, there is no (convenient) way to circumvent this check.

• Try specifying the path to the full certificate chain used by the server using the --tls_server_certs flag in osqueryd. This is often unnecessary when using a certificate signed by an authority trusted by the system, but is mandatory when working with self-signed certificates. In all cases it can be a useful debugging step.
• Ensure that the CNAME on the certificate matches the address at which the server is being accessed. If I try connect osquery via https://localhost:443, but my certificate is for https://fleet.example.com, the verification will fail.
• Is Fleet behind a load-balancer? Ensure that if the load-balancer is terminating TLS that this is the certificate provided to osquery.
• Does the certificate verify with curl? Try curl -v -X POST https://kolideserver:port/api/v1/osquery/enroll.

What do I do about "too many open files" errors?

This error usually indicates that the Fleet server has run out of file descriptors. Fix this by increasing the ulimit on the Fleet process. See the LimitNOFILE setting in the example systemd unit file for an example of how to do this with systemd.

I upgraded my database, but Fleet is still running slowly. What could be going on?

This could be caused by a mismatched connection limit between the Fleet server and the MySQL server that prevents Fleet from fully utilizing the database. First determine how many open connections your MySQL server supports. Now set the --mysql_max_open_conns and --mysql_max_idle_conns flags appropriately.

Why am I receiving a database connection error when attempting to "prepare" the database?

First, check if you have a version of MySQL installed that is at least 5.7. Then, make sure that you currently have a MySQL server running.

The next step is to make sure the credentials for the database match what is expected. Test your ability to connect to the database with mysql -u<username> -h<hostname_or_ip> -P<port> -D<database_name> -p.

If you're successful connecting to the database and still receive a database connection error, you may need to specify your database credentials when running fleet prepare db. It's encouraged to put your database credentials in environment variables or a config file.

fleet prepare db \
    --mysql_address=<database_address> \
    --mysql_database=<database_name> \
    --mysql_username=<username> \
    --mysql_password=<database_password>

How do I monitor a Fleet server?

Fleet provides standard interfaces for monitoring and alerting. See the Monitoring & Alerting documentation for details.

Why is the "Add User" button disabled?

The "Add User" button is disabled if SMTP (email) has not been configured for the Fleet server. Currently, there is no way to add new users without email capabilities.

One way to hack around this is to use a simulated mailserver like Mailhog. You can retrieve the email that was "sent" in the Mailhog UI, and provide users with the invite URL manually.

Is Fleet available as a SaaS product?

No. Currently, Fleet is only available as open-source software.

Has anyone stress tested Fleet?

Fleet has been stress tested to 150,000 online hosts and 400,000 total enrolled hosts. There are numerous production deployments in the thousands in the tens of thousands of hosts range, and there are production deployments in the high tens of thousands of hosts range.

How often do labels refresh? Is the refresh frequency configurable?

The update frequency for labels is configurable with the --osquery_label_update_interval flag (default 1 hour).

How do I get support for working with Fleet?

For bug reports, please use the Github issue tracker.

For questions and discussion, please join us in the #fleet channel of osquery Slack.