mirror of
https://github.com/amazon-science/chronos-forecasting
synced 2026-05-23 17:48:23 +00:00
eac768ce28
*Issue #, if available:* *Description of changes:* This PR adds a workflow that will run the evaluation script on `chronos-bolt-small` for a subset of datasets specified in `ci/evaluate/backtest_configs.yaml`. After evaluation, a comment will be made on the PR. The workflow will only run if the `run-eval` label is present on a PR. The end-to-end workflow has been split into two workflows: - `eval-model.yml`: only has read access (can be run from forks). This will evaluate the model and upload the metrics CSV file as a Github artifact. - `eval-pr-comment.yml`: has read and write access (can only be run when in the `main` branch). This will be triggered when the first job finishes, will download the CSV from the eval job and make the comment. According to [this post](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/), splitting into two jobs as done here is the recommended and secure way to do this. **NOTE**: The first steps works as expected, but we can only test the second step after the merging because this workflow needs to be part of the `main` branch for this to work. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Abdul Fatir Ansari <ansarnd@amazon.de>
37 lines
No EOL
903 B
YAML
37 lines
No EOL
903 B
YAML
# From In-domain
|
|
- name: taxi_30min # 30 min
|
|
hf_repo: autogluon/chronos_datasets
|
|
offset: -48
|
|
prediction_length: 48
|
|
num_rolls: 1
|
|
# From Zero-shot
|
|
- name: ETTh # Hourly
|
|
hf_repo: autogluon/chronos_datasets_extra
|
|
offset: -24
|
|
prediction_length: 24
|
|
num_rolls: 1
|
|
- name: monash_covid_deaths # Daily
|
|
hf_repo: autogluon/chronos_datasets
|
|
offset: -30
|
|
prediction_length: 30
|
|
num_rolls: 1
|
|
- name: monash_nn5_weekly # Weekly
|
|
hf_repo: autogluon/chronos_datasets
|
|
offset: -8
|
|
prediction_length: 8
|
|
num_rolls: 1
|
|
- name: monash_fred_md # Monthly
|
|
hf_repo: autogluon/chronos_datasets
|
|
offset: -12
|
|
prediction_length: 12
|
|
num_rolls: 1
|
|
- name: monash_m3_quarterly # Quarterly
|
|
hf_repo: autogluon/chronos_datasets
|
|
offset: -8
|
|
prediction_length: 8
|
|
num_rolls: 1
|
|
- name: monash_tourism_yearly # Yearly
|
|
hf_repo: autogluon/chronos_datasets
|
|
offset: -4
|
|
prediction_length: 4
|
|
num_rolls: 1 |