Skip to content

Comments

feat: add schema validation to promptfooconfig.yaml#1185

Merged
mldangelo merged 18 commits intomainfrom
feat/validation
Jul 18, 2024
Merged

feat: add schema validation to promptfooconfig.yaml#1185
mldangelo merged 18 commits intomainfrom
feat/validation

Conversation

@mldangelo
Copy link
Member

@mldangelo mldangelo commented Jul 17, 2024

This update introduces schema validation to promptfooconfig.yaml (in eval only). Currently, it issues a warning if the schema validation fails, using a standard Zod warning. I tested this on all of our example promptfooconfig.yaml files with a script like the following:

#!/bin/bash

# Find all promptfooconfig.yaml files
config_files=$(git ls-files | grep promptfooconfig.yaml)

# Loop through each config file and run promptfoo eval
for config in $config_files
do
  echo "Running evaluation for $config"
  promptfoo eval -c "$config"
done

echo "All evaluations complete"

I made a slight modification to process.exit() in eval.ts.

In the future, we should:

  • Add description fields to the Zod schema
  • Include more transform steps to simplify downstream dynamics
  • Generate a JSON schema that we can integrate into text editors to provide users with instant validation

CC @cmrfrd

@mldangelo mldangelo requested a review from typpo July 17, 2024 22:48
@mldangelo mldangelo self-assigned this Jul 17, 2024
@mldangelo mldangelo merged commit d967700 into main Jul 18, 2024
@mldangelo mldangelo deleted the feat/validation branch July 18, 2024 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant