EvalScope integration: one-command evaluation support for MiniWoB #400
Yunnglin
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi ServiceNow/BrowserGym maintainers,
EvalScope now supports MiniWoB. This gives users another way to run the benchmark when they want a standardized evaluation workflow around OpenAI-compatible model endpoints, saved predictions, scoring, and report generation.
This may be useful for users who find this repository while setting up MiniWoB and want a quick path to evaluate a hosted or locally served model.
For MiniWoB, EvalScope runs the benchmark through BrowserGym and uses a native function-calling browser-action agent workflow.
Example:
evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets miniwob \ --limit 10Docs: https://evalscope.readthedocs.io/en/latest/third_party/miniwob.html
EvalScope repo: https://github.com/modelscope/evalscope
Some benchmarks may require additional assets, media files, browser/runtime dependencies, or judge-model configuration for a full run; the EvalScope guide above includes the benchmark-specific setup notes.
If this is useful for your community, we can follow up with a small README/Docs PR linking to the EvalScope usage guide.
All reactions