feat: init v1 client format #16

abhishekg999 · 2025-11-05T23:49:28Z

No description provided.

gemini-code-assist · 2025-11-05T23:51:06Z

Summary of Changes

Hello @abhishekg999, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the initial version of the Judgeval SDK V1 API, which includes a new JudgmentClient for interacting with the Judgeval service. It also deprecates the existing scorer classes and Tracer, guiding users to migrate to the new V1 API. The changes include version updates, new class implementations, and an example demonstrating the new API's usage.

Highlights

Version Update: The pull request updates the judgeval-java library to version 0.3.0 in both the core library and the example project.
V1 Implementation: Introduces the initial implementation of the V1 API for the Judgeval SDK, providing a new way to interact with the service.
Deprecation of Existing Scorers and Tracer: Marks existing scorer classes (AnswerCorrectnessScorer, AnswerRelevancyScorer, DerailmentScorer, FaithfulnessScorer, InstructionAdherenceScorer, CustomScorer, PromptScorer, TracePromptScorer) and the Tracer class as deprecated, recommending the use of the new V1 JudgmentClient instead.
Example Usage: Adds a new example, V1QuickStart.java, demonstrating the usage of the V1 API, including initializing the client, creating tracers, and using different types of scorers.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces the v1 API for the Judgeval Java SDK, centered around the new JudgmentClient. The changes include adding the new client, corresponding factories and builders, a quick start example for v1, and deprecating the old API classes with helpful migration guides. The new API design using a central client and factories is a good improvement.

My review focuses on improving correctness, robustness, and developer experience. I've identified a potential runtime error in CustomScorer, a memory leak risk in PromptScorerFactory's cache, and several areas for improvement in the new example code and API class implementations. I've also found that the migration examples in the Javadocs for deprecated classes are incorrect and will cause compilation errors, which I've suggested fixes for.

gemini-code-assist · 2025-11-05T23:54:13Z

...eval-java/src/main/java/com/judgmentlabs/judgeval/v1/scorers/custom_scorer/CustomScorer.java

+    @Override
+    public ScorerConfig getScorerConfig() {
+        throw new UnsupportedOperationException("CustomScorer does not use ScorerConfig");
+    }


Overriding getScorerConfig() to throw an UnsupportedOperationException will cause a runtime crash when this scorer is used with asyncEvaluate or asyncTraceEvaluate, as the tracer mechanism relies on this method to build the evaluation payload. The default implementation from the parent APIScorer class should be sufficient. Please remove this override.

gemini-code-assist · 2025-11-05T23:54:13Z

...va/src/main/java/com/judgmentlabs/judgeval/v1/scorers/prompt_scorer/PromptScorerFactory.java

+    private final String                                                                           apiKey;
+    private final String                                                                           organizationId;
+    private final boolean                                                                          isTrace;
+    private static final Map<CacheKey, com.judgmentlabs.judgeval.internal.api.models.PromptScorer> cache = new ConcurrentHashMap<>();


Using a static ConcurrentHashMap for caching without any eviction policy can lead to a memory leak in long-running applications or in environments where many different scorers are fetched over time. Consider using a cache with an eviction policy (e.g., size-based or time-based eviction) like Guava's CacheBuilder or Caffeine. Alternatively, if the factory's lifecycle is managed, make the cache an instance field rather than static.

Its expected that the total number of Scorers is finite and if loaded, will be used multiple times. I.E This is more of a utility to avoid the behavior of the USER loading pulling all scorers at once into a new class, then importing from there.

.../src/main/java/com/judgmentlabs/judgeval/scorers/api_scorers/custom_scorer/CustomScorer.java

.../src/main/java/com/judgmentlabs/judgeval/scorers/api_scorers/prompt_scorer/PromptScorer.java

...-java/src/main/java/com/judgmentlabs/judgeval/scorers/api_scorers/AnswerRelevancyScorer.java

judgeval-java/src/main/java/com/judgmentlabs/judgeval/v1/evaluation/EvaluationFactory.java

gemini-code-assist · 2025-11-05T23:54:15Z

judgeval-java/src/main/java/com/judgmentlabs/judgeval/v1/scorers/ScorersFactory.java

+        return new CustomScorerFactory();
+    }
+
+    public BuiltInScorersFactory builtIn() {
+        return new BuiltInScorersFactory();


The customScorer() and builtIn() methods create new factory instances on every call. Since these factories (CustomScorerFactory, BuiltInScorersFactory) are stateless, they can be instantiated once and stored as final fields in ScorersFactory to avoid unnecessary object creation.

private final CustomScorerFactory customScorerFactory = new CustomScorerFactory(); private final BuiltInScorersFactory builtInScorersFactory = new BuiltInScorersFactory(); public CustomScorerFactory customScorer() { return customScorerFactory; } public BuiltInScorersFactory builtIn() { return builtInScorersFactory; }

gemini-code-assist · 2025-11-05T23:54:16Z

examples/src/main/java/examples/v1_quick_start/V1QuickStart.java

+        try {
+            Thread.sleep(10000);
+        } catch (InterruptedException e) {
+            e.printStackTrace();
+        }


Using Thread.sleep() to wait for asynchronous operations to complete is unreliable. The tracer might not have finished sending all data within the 10-second window, or the sleep might be unnecessarily long. Use the Tracer's built-in forceFlush() method to ensure all buffered spans are sent before the application exits.

// Wait for async traces to be exported before exiting. tracer.forceFlush(10000);

gemini-code-assist · 2025-11-05T23:54:16Z

...eval-java/src/main/java/com/judgmentlabs/judgeval/v1/scorers/prompt_scorer/PromptScorer.java

+    private final String              judgmentApiKey;
+    private final String              organizationId;


The judgmentApiKey and organizationId fields are initialized in the constructor but are never used. They should be removed to avoid confusion and unnecessary code. The corresponding assignments in the constructor and fields in the builder should also be removed.

gemini-code-assist · 2025-11-05T23:54:16Z

examples/src/main/java/examples/v1_quick_start/V1QuickStart.java

+            try {
+                PromptScorer traceScorer = client.scorers()
+                        .tracePromptScorer()
+                        .create()
+                        .name("trace-quality-scorer")
+                        .prompt("Does the entire trace show high quality reasoning?")
+                        .threshold(0.8)
+                        .build();
+                System.out.println("   Created TracePromptScorer: " + traceScorer.getName());
+            } catch (Exception e) {
+                System.out.println("   TracePromptScorer creation demo");
+            }


This try-catch block is unnecessary. Creating a scorer using the builder pattern (create()...build()) is a local operation that does not perform I/O and is not expected to throw an exception. This block appears to be a copy-paste from the example for fetching an existing scorer, which does perform a network request. Removing the try-catch will make the code cleaner.

PromptScorer traceScorer = client.scorers() .tracePromptScorer() .create() .name("trace-quality-scorer") .prompt("Does the entire trace show high quality reasoning?") .threshold(0.8) .build(); System.out.println(" Created TracePromptScorer: " + traceScorer.getName());

adivate2021

lgtm just some minor comments

judgeval-java/src/main/java/com/judgmentlabs/judgeval/v1/data/APIScorerType.java

judgeval-java/src/main/java/com/judgmentlabs/judgeval/v1/scorers/built_in/DerailmentScorer.java

judgeval-java/src/main/java/com/judgmentlabs/judgeval/v1/evaluation/Evaluation.java

...va/src/main/java/com/judgmentlabs/judgeval/v1/scorers/prompt_scorer/PromptScorerFactory.java

judgeval-java/src/main/java/com/judgmentlabs/judgeval/v1/tracer/BaseTracer.java

adivate2021

lgtm

feat: init v1

af21c5d

gemini-code-assist bot reviewed Nov 5, 2025

View reviewed changes

abhishekg999 added 5 commits November 5, 2025 16:09

guide

53453a9

updates

f362421

chore: tests

a406e5b

fix: tests

08beb2c

cleanup

af8bece

abhishekg999 changed the title ~~feat: init v1~~ feat: init v1 client format Nov 6, 2025

abhishekg999 added 3 commits November 5, 2025 21:20

format

2337c48

chore: remove api key

f5db678

wip

6e4c21a

adivate2021 requested changes Nov 7, 2025

View reviewed changes

abhishekg999 added 6 commits November 7, 2025 21:00

fix: trace scorer

c6ad9fe

reapply gitignore

51c62c2

chore: format

9f6a4bf

fix: implement custom scorer

5c8d54b

fix: remove derailment

7530ea1

chore: dont throw on prompt scorer get

20901c2

abhishekg999 requested a review from adivate2021 November 9, 2025 00:49

cleanup: no model

3c7ea84

adivate2021 approved these changes Nov 9, 2025

View reviewed changes

abhishekg999 added 2 commits November 10, 2025 22:47

judgment client -> judgeval

d1ecd58

chore: initialize true default

3dcf889

abhishekg999 merged commit f727451 into main Nov 11, 2025
2 checks passed

abhishekg999 deleted the ahh/0.3.0 branch November 11, 2025 22:53

		private final String judgmentApiKey;
		private final String organizationId;

feat: init v1 client format #16

feat: init v1 client format #16

Uh oh!

Conversation

abhishekg999 commented Nov 5, 2025

Uh oh!

gemini-code-assist bot commented Nov 5, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

abhishekg999 Nov 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

adivate2021 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adivate2021 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants