mirror of
https://github.com/amazon-science/chronos-forecasting
synced 2026-05-24 10:08:33 +00:00
eac768ce28
*Issue #, if available:* *Description of changes:* This PR adds a workflow that will run the evaluation script on `chronos-bolt-small` for a subset of datasets specified in `ci/evaluate/backtest_configs.yaml`. After evaluation, a comment will be made on the PR. The workflow will only run if the `run-eval` label is present on a PR. The end-to-end workflow has been split into two workflows: - `eval-model.yml`: only has read access (can be run from forks). This will evaluate the model and upload the metrics CSV file as a Github artifact. - `eval-pr-comment.yml`: has read and write access (can only be run when in the `main` branch). This will be triggered when the first job finishes, will download the CSV from the eval job and make the comment. According to [this post](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/), splitting into two jobs as done here is the recommended and secure way to do this. **NOTE**: The first steps works as expected, but we can only test the second step after the merging because this workflow needs to be part of the `main` branch for this to work. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Abdul Fatir Ansari <ansarnd@amazon.de>
54 lines
No EOL
1.6 KiB
YAML
54 lines
No EOL
1.6 KiB
YAML
# Post evaluation results from the "Evaluate" workflow as a PR comment
|
|
name: Post Eval Metrics
|
|
|
|
on:
|
|
# Runs with read & write privilages for the GITHUB_TOKEN
|
|
workflow_run:
|
|
workflows: ["Evaluate"]
|
|
types:
|
|
- completed
|
|
|
|
jobs:
|
|
comment-eval-results:
|
|
if: >
|
|
github.event.workflow_run.event == 'pull_request' &&
|
|
github.event.workflow_run.conclusion == 'success'
|
|
runs-on: ubuntu-latest
|
|
permissions:
|
|
actions: read # for downloading artifacts
|
|
pull-requests: write # for posting PR comment
|
|
|
|
steps:
|
|
- name: Download Eval Metrics
|
|
uses: actions/download-artifact@v4
|
|
with:
|
|
name: eval-metrics
|
|
path: eval-metrics-artifact/
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
run-id: ${{ github.event.workflow_run.id }}
|
|
|
|
- name: Display structure of downloaded files
|
|
run: ls -R
|
|
|
|
- name: Read CSV
|
|
id: csv
|
|
uses: juliangruber/read-file-action@v1
|
|
with:
|
|
path: eval-metrics-artifact/eval-ci-metrics.csv
|
|
|
|
- name: Create Markdown Table
|
|
uses: petems/csv-to-md-table-action@master
|
|
id: csv-table-output
|
|
with:
|
|
csvinput: ${{ steps.csv.outputs.content }}
|
|
|
|
- name: Post Table as a Comment
|
|
uses: peter-evans/create-or-update-comment@v4
|
|
with:
|
|
token: ${{ secrets.GITHUB_TOKEN }}
|
|
repository: ${{ github.repository }}
|
|
issue-number: ${{ github.event.pull_request.number }}
|
|
body: |
|
|
### Evaluation Metrics
|
|
${{steps.csv-table-output.outputs.markdown-table}}
|
|
reactions: rocket |