mirror of
https://github.com/amazon-science/chronos-forecasting
synced 2026-05-24 10:08:33 +00:00
eac768ce28
*Issue #, if available:* *Description of changes:* This PR adds a workflow that will run the evaluation script on `chronos-bolt-small` for a subset of datasets specified in `ci/evaluate/backtest_configs.yaml`. After evaluation, a comment will be made on the PR. The workflow will only run if the `run-eval` label is present on a PR. The end-to-end workflow has been split into two workflows: - `eval-model.yml`: only has read access (can be run from forks). This will evaluate the model and upload the metrics CSV file as a Github artifact. - `eval-pr-comment.yml`: has read and write access (can only be run when in the `main` branch). This will be triggered when the first job finishes, will download the CSV from the eval job and make the comment. According to [this post](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/), splitting into two jobs as done here is the recommended and secure way to do this. **NOTE**: The first steps works as expected, but we can only test the second step after the merging because this workflow needs to be part of the `main` branch for this to work. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Abdul Fatir Ansari <ansarnd@amazon.de>
35 lines
1.1 KiB
YAML
35 lines
1.1 KiB
YAML
# Evaluates Chronos-Bolt (Small) model on selected datasets
|
|
name: Evaluate
|
|
|
|
on:
|
|
# Runs only with read privilages for the GITHUB_TOKEN
|
|
pull_request:
|
|
branches: ["main"] # Run on PRs to main branch
|
|
|
|
jobs:
|
|
evaluate-and-post:
|
|
if: contains(github.event.pull_request.labels.*.name, 'run-eval') # Only run if 'run-eval' label is added
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout Repository
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.11'
|
|
|
|
- name: Install Dependencies
|
|
run: pip install ".[evaluation]" -f https://download.pytorch.org/whl/cpu/torch_stable.html
|
|
|
|
- name: Run Eval Script
|
|
run: python scripts/evaluation/evaluate.py ci/evaluate/backtest_config.yaml eval-ci-metrics.csv --chronos-model-id=amazon/chronos-bolt-small --device=cpu --torch-dtype=float32
|
|
|
|
- name: Upload CSV
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: eval-metrics
|
|
path: eval-ci-metrics.csv
|
|
retention-days: 1
|
|
overwrite: true
|