feat: Plumb kwargs through to evaluate and evaluate_dataframe

[anticorrelator]

• allows passing arbitrary kwargs into evaluator's evaluate and async_evaluate methods
• when appropriate, these are passed onto LLM.generate_classification as invocation parameters
• per-evaluator kwargs can be passed into evaluate_dataframe with eval_kwargs, a mapping from evaluator names to kwarg dictionaries

---

Note

Adds **kwargs passthrough to evaluator evaluate/async methods and dataframe evaluators (with per-evaluator eval_kwargs), forwarding them to LLM classification calls.

• Evaluators:
  • Evaluator.evaluate/async_evaluate now accept **kwargs and forward to _evaluate/_async_evaluate (including thread-wrapper path).
  • _evaluate/_async_evaluate signatures updated across base, LLM, and decorator-generated evaluators to accept **kwargs.
• LLM Evaluation:
  • ClassificationEvaluator forwards **kwargs to LLM.generate_classification/async_generate_classification.
• DataFrame Runners:
  • evaluate_dataframe and async_evaluate_dataframe accept eval_kwargs (per-evaluator) and **kwargs (global); merged and passed to each evaluator during task execution.

^{Written by Cursor Bugbot for commit 3d1dd97. This will update automatically on new commits. Configure here.}

[cursor]

Bug: Decorator Ignores Function Parameters

The _evaluate and _async_evaluate methods generated by the create_evaluator decorator accept **kwargs but don't forward them to the underlying user-defined function. This means any kwargs provided to function-based evaluators are silently ignored, which can cause unexpected behavior for users expecting their functions to receive these parameters.

 

[graphite-app]

The order of merging kwargs needs to be reversed to match the documented behavior. Currently, general kwargs will override evaluator-specific kwargs, but the docstring states that eval_kwargs should take precedence.

To fix this, change:

task_kwargs: Dict[str, Any] = (
    eval_kwargs.get(evaluator.name, {}) if eval_kwargs else {}
)
task_kwargs.update(kwargs)

to:

task_kwargs = kwargs.copy()
task_kwargs.update(eval_kwargs.get(evaluator.name, {}) if eval_kwargs else {})

This ensures that evaluator-specific settings from eval_kwargs will override the general kwargs as intended.

Suggested change

	task_kwargs: Dict[str, Any] = (
	eval_kwargs.get(evaluator.name, {}) if eval_kwargs else {}
	)
	task_kwargs.update(kwargs)
	task_kwargs: Dict[str, Any] = kwargs.copy()
	task_kwargs.update(
	eval_kwargs.get(evaluator.name, {}) if eval_kwargs else {}
	)

Spotted by Diamond



Is this helpful? React 👍 or 👎 to let us know.