fleet/server/service/schedule
Scott Gress 1a1d7bae78
Clear cron schedule errors before each run (#26775)
For #26657

This PR fixes an issue that causes cron monitoring alerts to be sent
repeatedly after the first instance; that is, if a cron job fails once
then the monitor reports the failure every time it runs until the server
is restarted. This was due to the errors being held in the Schedule
object which persists for the lifetime of the process, rather than being
recreated for each run. The solution is to clear the errors from the
Schedule object before each run.

Added a test that fails on main and passes on this branch.
2025-03-03 16:41:48 -06:00
..
README.md feat: initial readme for cron jobs (#23563) 2024-11-06 09:13:45 -05:00
schedule.go Clear cron schedule errors before each run (#26775) 2025-03-03 16:41:48 -06:00
schedule_test.go Clear cron schedule errors before each run (#26775) 2025-03-03 16:41:48 -06:00
testing_utils.go Monitor and alert on errors in cron jobs (#24347) 2024-12-19 15:55:29 -06:00

schedule: the Fleet cron job machinery

Fleet has several pieces of functionality that are implemented as cron jobs, which run on a schedule. Package schedule implements the machinery needed for queueing and running these jobs.

List of cron jobs

See server/fleet/cron_schedules.go for a list of the currently implemented cron jobs and information about what they do.

Cron jobs are created and registered in the cmd/fleet package because they have to be run at server start. The actual implementation of the cron job logic is usually elsewhere however, typically in a service layer method (and related datastore methods).

How to add a new cron job

See this PR for a nice example of how to add a simple cron job.

  1. Do you need a new cron job? You can add sub-jobs to an existing cron job; for example, if you're adding some functionality for cleaning up unused data, you might want to implement it as a sub-job in the cleanups_then_aggregation cron.
  2. Add a cron job name. If you determine that you do need a new cron job, create a descriptive name in cron_schedules.go. Make sure you leave a comment explaining what the job does.
  3. Implement your functionality. Do this wherever it makes sense. In the example PR, the functionality exists in the server/mdm/maintainedapps/ingest.go file. However, you'll most likely implement a service layer method and related datastore layer methods.
  4. Add a function that returns a *schedule.Schedule in cmd/fleet/cron.go. This function will be used to register your cron job so it can actually run. This function should call whatever you implemented in step 3. This is also where you can set the interval on which your cron job will run.
  5. Register the cron job in cmd/fleet/serve.go. You'll use cronSchedules.StartCronSchedule to register the cron job by passing it an anonymous function that calls the function you wrote in step 3.