Update Examples and CI workflow

[JoyboyBrian]

This pull request refactors the reward rubric scoring system to support multiple LLM providers, specifically OpenAI and Anthropic, and simplifies the configuration and usage. It removes the previous YAML-based rubric configuration and dataset files, replaces the monolithic implementation with provider-specific Python modules, and updates documentation and CI workflows to reflect these changes.

Provider-specific rubric scoring (major refactor):

• Removed the generic YAML-based configuration and dataset files (reward_rubric_config.yaml, reward_rubric_example.json, sample_data.jsonl) and the old reward_rubric.py implementation, in favor of two new provider-specific modules: reward_rubric_openai.py and reward_rubric_anthropic.py, each with a hardcoded rubric and scoring logic. [1] [2] [3] [4] [5]

• Added scripts run_reward_rubric_openai.sh and run_reward_rubric_anthropic.sh to run the respective scorer modules, replacing the old run_reward_rubric.sh script. [1] [2]

Documentation updates:

• Updated README.md to document the new provider-specific scoring modules, usage examples for both shell scripts and direct Python calls, and revised CI/CD setup instructions for multi-provider support. [1] [2] [3] [4]

CI workflow improvements:

• Updated .github/workflows/reward_rubric.yml to trigger on changes to any files in reward_rubric/ or the new scripts, and to run separate jobs for OpenAI and Anthropic rubric scoring, each with its own API key and output validation. [1] [2]

Project structure cleanup:

• Cleaned up the directory listing in README.md to reflect the new files and removed references to obsolete assets.

This refactor makes it easier to add new providers, simplifies configuration, and improves clarity for both local development and CI.

[gemini-code-assist]

Summary of Changes

Hello @JoyboyBrian, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant refactoring of the reward rubric scoring system, transitioning from a single, YAML-configured implementation to a more modular, provider-specific approach. The core change involves creating distinct Python modules for OpenAI and Anthropic, which simplifies the integration and management of different LLM providers. This update also includes comprehensive documentation and CI workflow adjustments to reflect and support the new multi-provider architecture, making the system more flexible and easier to extend for future LLM integrations.

Highlights

• Multi-Provider LLM Support: The reward rubric scoring system has been refactored to support multiple LLM providers, specifically OpenAI and Anthropic, moving away from a monolithic, YAML-based configuration.
• Simplified Configuration: The previous generic reward_rubric.py and YAML configuration files have been replaced with dedicated, provider-specific Python modules (reward_rubric_openai.py and reward_rubric_anthropic.py), each containing hardcoded rubric and scoring logic for simplicity.
• Updated Examples and Documentation: New shell scripts (run_reward_rubric_openai.sh, run_reward_rubric_anthropic.sh) and comprehensive updates to the README.md provide clear usage examples and documentation for the new provider-specific scoring modules.
• Enhanced CI Workflow: The CI workflow (.github/workflows/reward_rubric.yml) has been improved to trigger on relevant file changes and now runs separate jobs for OpenAI and Anthropic rubric scoring, ensuring multi-provider validation.
• Project Structure Cleanup: Obsolete configuration files (reward_rubric_config.yaml, reward_rubric_example.json, sample_data.jsonl) and the generic reward_rubric.py have been removed, streamlining the project structure.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/reward_rubric.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request does a great job of refactoring the reward rubric scoring system to support multiple LLM providers, namely OpenAI and Anthropic. The move away from a single implementation with YAML configuration to provider-specific Python modules greatly simplifies the configuration and improves modularity. The documentation and CI workflow updates are also well-aligned with these changes. My review includes a few suggestions to enhance the robustness of the new Python modules, primarily concerning API key handling and type safety.

[Copilot]

Pull Request Overview

This PR refactors the reward rubric scoring system from a YAML-based configuration approach to provider-specific Python modules, supporting both OpenAI and Anthropic LLM providers with simplified hardcoded configurations.

Key changes:

• Replaced monolithic reward_rubric.py with separate reward_rubric_openai.py and reward_rubric_anthropic.py modules
• Removed YAML configuration, JSON examples, and JSONL dataset files in favor of inline configuration
• Updated CI workflow to run separate jobs for each provider with appropriate API key validation

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
reward_rubric/reward_rubric_openai.py	New OpenAI-specific rubric scorer with hardcoded GPT model configuration
reward_rubric/reward_rubric_anthropic.py	New Anthropic-specific rubric scorer with hardcoded Claude model configuration
scripts/run_reward_rubric_openai.sh	Shell script to execute OpenAI rubric scoring with example data
scripts/run_reward_rubric_anthropic.sh	Shell script to execute Anthropic rubric scoring with example data
scripts/run_reward_rubric.sh	Removed old generic rubric runner script
reward_rubric/reward_rubric.py	Removed monolithic implementation with YAML-based configuration
reward_rubric/reward_rubric_config.yaml	Removed YAML configuration file
reward_rubric/reward_rubric_example.json	Removed JSON example file
reward_rubric/sample_data.jsonl	Removed JSONL dataset file
README.md	Updated documentation to reflect provider-specific approach with usage examples
.github/workflows/reward_rubric.yml	Updated CI to run separate jobs for OpenAI and Anthropic with dedicated API keys

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.