Unable to run benchmark tasks using "eval-dev-quality evaluate" command on windows due to usage of ":" in the directory creation syntax. Check the attached screenshot of the error.
Steps to reproduce
- Install Go, Git
- Clone the repository and install application.
- Run "eval-dev-quality evaluate --model=openrouter/meta-llama/llama-3-70b-instruct" command.
System Configuration
Windows 10 pro
Findings
I found that in evaluat.go file (line 67), following format is used for directory creation.
command.ResultPath = strings.ReplaceAll(command.ResultPath, "%datetime%", evaluationTimestamp.Format("2006-01-02-15:04:05"))
Usage of ":", I believe is not supported for directory creation in windows.
