Skip to content

Conversation

@sgoedecke
Copy link
Collaborator

Adds an example GitHub Action workflow that uses the new gh models eval command, so we have something to link to. Ideally this would be a fully-fledged reusable workfow a la actions/ai-inference but I don't have the time to do that for the foreseeable future.

Copilot AI review requested due to automatic review settings June 5, 2025 22:36
@sgoedecke sgoedecke requested a review from a team as a code owner June 5, 2025 22:36
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds a new example GitHub Actions workflow that runs gh models eval on changed prompt files and updates the README to reference this example.

  • Introduce examples/evals_action.yml demonstrating how to evaluate prompts in a PR using the gh models eval command.
  • Link the new workflow example in README.md.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

File Description
examples/evals_action.yml New workflow for running prompt evaluations and commenting results
README.md Added reference to the evals_action.yml example
Comments suppressed due to low confidence (1)

README.md:83

  • The description refers to the eval command, but the workflow uses gh models eval. Consider clarifying this to avoid confusion.
Here's a sample GitHub Action that uses the `eval` command to automatically run the evals in any PR that updates a prompt file: [evals_action.yml](/examples/evals_action.yml).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@sgoedecke sgoedecke merged commit 4a20abf into main Jun 5, 2025
4 checks passed
@sgoedecke sgoedecke deleted the sgoedecke/add-evals-action-example branch June 5, 2025 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants