Skip to content

Unable to run benchmark tasks on windows due to incorrect directory creation syntax #101

@mkovelamudi

Description

@mkovelamudi

Unable to run benchmark tasks using "eval-dev-quality evaluate" command on windows due to usage of ":" in the directory creation syntax. Check the attached screenshot of the error.

Steps to reproduce

  1. Install Go, Git
  2. Clone the repository and install application.
  3. Run "eval-dev-quality evaluate --model=openrouter/meta-llama/llama-3-70b-instruct" command.

System Configuration
Windows 10 pro

Findings
I found that in evaluat.go file (line 67), following format is used for directory creation.
command.ResultPath = strings.ReplaceAll(command.ResultPath, "%datetime%", evaluationTimestamp.Format("2006-01-02-15:04:05"))

Usage of ":", I believe is not supported for directory creation in windows.

image

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions