Skip to content

Commit

Permalink
chore(docs): update --config yaml file references to match actual beh…
Browse files Browse the repository at this point in the history 
…avior (promptfoo#2170)

Signed-off-by: Evan Harris <echarris@smcm.edu>
  • Loading branch information
@eharris128
eharris128 authored Nov 27, 2024
1 parent cf67b92 commit 6d5ebae
Show file tree
Hide file tree
Showing 3 changed files with 38 additions and 38 deletions.
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ If you're looking to customize your usage, you have a wide set of parameters at
| `-r, --providers <name or path...>` | One of: openai:chat, openai:completion, openai:model-name, localai:chat:model-name, localai:completion:model-name. See [API providers][providers-docs] |
| `-o, --output <path>` | Path to [output file](https://www.promptfoo.dev/docs/configuration/parameters#output-file) (csv, json, yaml, html) |
| `--tests <path>` | Path to [external test file](https://www.promptfoo.dev/docs/configurationexpected-outputsassertions#load-an-external-tests-file) |
| `-c, --config <paths>` | Path to one or more [configuration files](https://www.promptfoo.dev/docs/configuration/guide). `promptfooconfig.js/json/yaml` is automatically loaded if present |
| `-c, --config <paths>` | Path to one or more [configuration files](https://www.promptfoo.dev/docs/configuration/guide). `promptfooconfig.yaml` is automatically loaded if present |
| `-j, --max-concurrency <number>` | Maximum number of concurrent API calls |
| `--table-cell-max-length <number>` | Truncate console table cells to this length |
| `--prompt-prefix <path>` | This prefix is prepended to every prompt |
Expand Down
72 changes: 36 additions & 36 deletions site/docs/usage/command-line.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,42 +25,42 @@ The `promptfoo` command line utility supports the following subcommands:

By default the `eval` command will read the `promptfooconfig.yaml` configuration file in your current directory. But, if you're looking to override certain parameters you can supply optional arguments:

| Option | Description |
| ----------------------------------- | ------------------------------------------------------------------------------- |
| `-a, --assertions <path>` | Path to assertions file |
| `-c, --config <paths...>` | Path to configuration file(s). Automatically loads promptfooconfig.js/json/yaml |
| `--delay <number>` | Delay between each test (in milliseconds) |
| `--description <description>` | Description of the eval run |
| `--env-file, --env-path <path>` | Path to .env file |
| `--filter-failing <path>` | Path to JSON output file with failing tests |
| `-n, --filter-first-n <number>` | Only run the first N tests |
| `--filter-pattern <pattern>` | Only run tests whose description matches the regex pattern |
| `--filter-providers <providers>` | Only run tests with these providers |
| `--filter-sample <number>` | Only run a random sample of N tests |
| `--filter-targets <targets>` | Only run tests with these targets |
| `--grader <provider>` | Model that will grade outputs |
| `-j, --max-concurrency <number>` | Maximum number of concurrent API calls |
| `--model-outputs <path>` | Path to JSON containing list of LLM output strings |
| `--no-cache` | Do not read or write results to disk cache |
| `--no-progress-bar` | Do not show progress bar |
| `--no-table` | Do not output table in CLI |
| `--no-write` | Do not write results to promptfoo directory |
| `-o, --output <paths...>` | Path(s) to output file (csv, txt, json, jsonl, yaml, yml, html) |
| `-p, --prompts <paths...>` | Paths to prompt files (.txt) |
| `--prompt-prefix <path>` | Prefix prepended to every prompt |
| `--prompt-suffix <path>` | Suffix appended to every prompt |
| `-r, --providers <name or path...>` | Provider names or paths to custom API caller modules |
| `--remote` | Force remote inference wherever possible (used for red teams) |
| `--repeat <number>` | Number of times to run each test |
| `--share` | Create a shareable URL |
| `--suggest-prompts <number>` | Generate N new prompts and append them to the prompt list |
| `--table` | Output table in CLI |
| `--table-cell-max-length <number>` | Truncate console table cells to this length |
| `-t, --tests <path>` | Path to CSV with test cases |
| `--var <key=value>` | Set a variable in key=value format |
| `-v, --vars <path>` | Path to CSV with test cases (alias for --tests) |
| `--verbose` | Show debug logs |
| `-w, --watch` | Watch for changes in config and re-run |
| Option | Description |
| ----------------------------------- | ----------------------------------------------------------------------- |
| `-a, --assertions <path>` | Path to assertions file |
| `-c, --config <paths...>` | Path to configuration file(s). Automatically loads promptfooconfig.yaml |
| `--delay <number>` | Delay between each test (in milliseconds) |
| `--description <description>` | Description of the eval run |
| `--env-file, --env-path <path>` | Path to .env file |
| `--filter-failing <path>` | Path to JSON output file with failing tests |
| `-n, --filter-first-n <number>` | Only run the first N tests |
| `--filter-pattern <pattern>` | Only run tests whose description matches the regex pattern |
| `--filter-providers <providers>` | Only run tests with these providers |
| `--filter-sample <number>` | Only run a random sample of N tests |
| `--filter-targets <targets>` | Only run tests with these targets |
| `--grader <provider>` | Model that will grade outputs |
| `-j, --max-concurrency <number>` | Maximum number of concurrent API calls |
| `--model-outputs <path>` | Path to JSON containing list of LLM output strings |
| `--no-cache` | Do not read or write results to disk cache |
| `--no-progress-bar` | Do not show progress bar |
| `--no-table` | Do not output table in CLI |
| `--no-write` | Do not write results to promptfoo directory |
| `-o, --output <paths...>` | Path(s) to output file (csv, txt, json, jsonl, yaml, yml, html) |
| `-p, --prompts <paths...>` | Paths to prompt files (.txt) |
| `--prompt-prefix <path>` | Prefix prepended to every prompt |
| `--prompt-suffix <path>` | Suffix appended to every prompt |
| `-r, --providers <name or path...>` | Provider names or paths to custom API caller modules |
| `--remote` | Force remote inference wherever possible (used for red teams) |
| `--repeat <number>` | Number of times to run each test |
| `--share` | Create a shareable URL |
| `--suggest-prompts <number>` | Generate N new prompts and append them to the prompt list |
| `--table` | Output table in CLI |
| `--table-cell-max-length <number>` | Truncate console table cells to this length |
| `-t, --tests <path>` | Path to CSV with test cases |
| `--var <key=value>` | Set a variable in key=value format |
| `-v, --vars <path>` | Path to CSV with test cases (alias for --tests) |
| `--verbose` | Show debug logs |
| `-w, --watch` | Watch for changes in config and re-run |

The `eval` command will return exit code `100` when there is at least 1 test case failure. It will return exit code `1` for any other error. The exit code for failed tests can be overridden with environment variable `PROMPTFOO_FAILED_TEST_EXIT_CODE`.

Expand Down
2 changes: 1 addition & 1 deletion src/commands/eval.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -425,7 +425,7 @@ export function evalCommand(
// Core configuration
.option(
'-c, --config <paths...>',
'Path to configuration file. Automatically loads promptfooconfig.js/json/yaml',
'Path to configuration file. Automatically loads promptfooconfig.yaml',
)
.option('--env-file, --env-path <path>', 'Path to .env file')

Expand Down

0 comments on commit 6d5ebae

Please sign in to comment.