chronos-forecasting/ci/evaluate/backtest_config.yaml

# From In-domain
- name: taxi_30min # 30 min
  hf_repo: autogluon/chronos_datasets
  offset: -48
  prediction_length: 48
  num_rolls: 1
# From Zero-shot
- name: ETTh # Hourly
  hf_repo: autogluon/chronos_datasets_extra
  offset: -24
  prediction_length: 24
  num_rolls: 1
- name: monash_covid_deaths # Daily
  hf_repo: autogluon/chronos_datasets
  offset: -30
  prediction_length: 30
  num_rolls: 1
- name: monash_nn5_weekly # Weekly
  hf_repo: autogluon/chronos_datasets
  offset: -8
  prediction_length: 8
  num_rolls: 1
- name: monash_fred_md # Monthly
  hf_repo: autogluon/chronos_datasets
  offset: -12
  prediction_length: 12
  num_rolls: 1
- name: monash_m3_quarterly # Quarterly
  hf_repo: autogluon/chronos_datasets
  offset: -8
  prediction_length: 8
  num_rolls: 1
- name: monash_tourism_yearly # Yearly
  hf_repo: autogluon/chronos_datasets
  offset: -4
  prediction_length: 4
  num_rolls: 1
Add workflow to run evaluation on a subset of datasets (#222) Issue #, if available: Description of changes: This PR adds a workflow that will run the evaluation script on `chronos-bolt-small` for a subset of datasets specified in `ci/evaluate/backtest_configs.yaml`. After evaluation, a comment will be made on the PR. The workflow will only run if the `run-eval` label is present on a PR. The end-to-end workflow has been split into two workflows: - `eval-model.yml`: only has read access (can be run from forks). This will evaluate the model and upload the metrics CSV file as a Github artifact. - `eval-pr-comment.yml`: has read and write access (can only be run when in the `main` branch). This will be triggered when the first job finishes, will download the CSV from the eval job and make the comment. According to [this post](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/), splitting into two jobs as done here is the recommended and secure way to do this. NOTE: The first steps works as expected, but we can only test the second step after the merging because this workflow needs to be part of the `main` branch for this to work. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Abdul Fatir Ansari <ansarnd@amazon.de> 2024-12-02 09:05:57 +00:00			`# From In-domain`
			`- name: taxi_30min # 30 min`
			`hf_repo: autogluon/chronos_datasets`
			`offset: -48`
			`prediction_length: 48`
			`num_rolls: 1`
			`# From Zero-shot`
			`- name: ETTh # Hourly`
			`hf_repo: autogluon/chronos_datasets_extra`
			`offset: -24`
			`prediction_length: 24`
			`num_rolls: 1`
			`- name: monash_covid_deaths # Daily`
			`hf_repo: autogluon/chronos_datasets`
			`offset: -30`
			`prediction_length: 30`
			`num_rolls: 1`
			`- name: monash_nn5_weekly # Weekly`
			`hf_repo: autogluon/chronos_datasets`
			`offset: -8`
			`prediction_length: 8`
			`num_rolls: 1`
			`- name: monash_fred_md # Monthly`
			`hf_repo: autogluon/chronos_datasets`
			`offset: -12`
			`prediction_length: 12`
			`num_rolls: 1`
			`- name: monash_m3_quarterly # Quarterly`
			`hf_repo: autogluon/chronos_datasets`
			`offset: -8`
			`prediction_length: 8`
			`num_rolls: 1`
			`- name: monash_tourism_yearly # Yearly`
			`hf_repo: autogluon/chronos_datasets`
			`offset: -4`
			`prediction_length: 4`
			`num_rolls: 1`