Display errors in evalite.config.ts resolution

[silas]

We use vite-tsconfig-paths in our project to support compilerOptions.paths, but it seems like evalite isn't picking it up.

Looking at the runEvalite code, I don't see any obvious reason why plugins wouldn't get passed along.

It's totally possible vite-tsconfig-paths is doing some magic that isn't supported in evalite.

Are vitest plugins supported, did I miss something obvious?

You can see a full example here: https://github.com/silas/evalite-vite-tsconfig-paths

package.json

{
  "devDependencies": {
    "ai": "6.0.0-beta.99",
    "evalite": "1.0.0-beta.5",
    "vite-tsconfig-paths": "^5.1.4",
    "vitest": "^4.0.9"
  }
}

vitest.config.ts

import tsconfigPaths from "vite-tsconfig-paths";
import { defineConfig } from "vitest/config";

export default defineConfig({
  plugins: [tsconfigPaths()],
});

evalite.config.ts

import { defineConfig } from "evalite/config";
import viteConfig from "./vite.config.ts";

export default defineConfig({
  viteConfig: viteConfig,
});

vitest.spec.ts (works as expected):

import { expect, test } from "vitest";

import { EXPECTED } from "@/data";

test("should work", () => {
  expect("Hello World!").toBe(EXPECTED);
});

path.eval.ts (fails):

import { evalite } from "evalite";
import { exactMatch } from "evalite/scorers";

import { EXPECTED } from "@/data";

evalite("My Eval", {
  data: [{ input: "Hello", expected: EXPECTED }],
  task: async (input) => {
    return input + " World!";
  },
  scorers: [
    {
      scorer: ({ output, expected }) =>
        exactMatch({ actual: output, expected }),
    },
  ],
});

When running this evaluation I get the following error:

$ pnpm evalite

 EVALITE running...

Sourcemap for "/Users/demo/src/evalite-vite-tsconfig-paths/node_modules/.pnpm/evalite@1.0.0-beta.5_better-sqlite3@11.10.0/node_modules/evalite/dist/env-setup-file.js" points to missing source files
 ✖ path.eval.ts  (unknown evals)
Error: Cannot find package '@/data' imported from '/Users/demo/src/evalite-vite-tsconfig-paths/path.eval.ts'
 ❯ path.eval.ts:4:1
      2| import { exactMatch } from "evalite/scorers";
      3|
      4| import { EXPECTED } from "@/data";
       | ^
      5|
      6| evalite("My Eval", {

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { code: 'ERR_MODULE_NOT_FOUND' }
 ⏳ no-path.eval.ts  (1 eval)
 100% no-path.eval.ts  (1 eval)

      Score  100%
 Eval Files  2
      Evals  1
   Duration  3ms

[silas]

Ah, my import was wrong. I was using import viteConfig from "./vite.config.ts"; and it should have been import viteConfig from "./vitest.config.ts";.

Looks like this error gets suppressed, which is why I was confused.

[silas]

Might be worth just checking if configPath exists and not suppressing all import errors when loading evalite.config.ts in loadEvaliteConfig (although maybe there is more complexity to this).

evalite/packages/evalite/src/config.ts

Lines 50 to 67 in 44b791e

	const configPath = path.join(cwd, fileName);
	try {
	const loaded = (await jiti.import(configPath)) as any;
	const config = loaded.default || loaded.evalite || loaded;
	if (config && typeof config === "object") {
	return config as Evalite.Config;
	}
	} catch (error: any) {
	// File not found is expected, ignore it
	if (
	error.code === "ERR_MODULE_NOT_FOUND" ||
	error.code === "ENOENT" ||
	error.message?.includes("Cannot find module")
	) {
	continue;
	}

[mattpocock]

Fixed