design and implementation plan #3410
Conversation
Summary of ChangesHello @robfrank, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request establishes the foundational documentation for a new performance benchmark within ArcadeDB. It includes a detailed design specification for an LDBC Social Network Benchmark-inspired graph, outlining its schema, data generation process, and a series of lookup and traversal queries. Complementing this, a comprehensive implementation plan is provided, breaking down the development into manageable tasks, ensuring a structured approach to building a robust and measurable performance testing framework for ArcadeDB's graph capabilities. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
🧪 CI InsightsHere's what we observed from your CI run for 9d36998. 🟢 All jobs passed!But CI Insights is watching 👀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive design document and a detailed implementation plan for a new LDBC-inspired graph benchmark. The plans are well-structured and thorough, covering schema, data generation, benchmark phases, and reporting.
My review focuses on ensuring the correctness of the benchmark design and improving the clarity of the implementation plan. I've identified a significant issue in the 'Friends-of-friends' query (4a) where the proposed SQL query is not logically equivalent to its Cypher counterpart, which would skew benchmark results. I've also suggested improvements to the implementation plan regarding hardcoded paths and code readability.
Overall, this is an excellent planning effort that will set a strong foundation for the benchmark implementation.
| benchmark("4a", "Friends of friends", COMPLEX_TRAVERSAL_ITERATIONS, | ||
| "SELECT expand(both('KNOWS').both('KNOWS')) FROM Person WHERE id = :id", | ||
| "MATCH (p:Person {id: $id})-[:KNOWS]-()-[:KNOWS]-(fof) " + | ||
| "WHERE fof <> p AND NOT (p)-[:KNOWS]-(fof) RETURN DISTINCT fof"); |
There was a problem hiding this comment.
The implementation plan for query 4a ('Friends of friends') carries over an incorrect SQL query from the design document. The query SELECT expand(both('KNOWS').both('KNOWS')) FROM Person WHERE id = :id does not exclude direct friends, which makes the SQL benchmark not comparable to the Cypher benchmark for the same task. Given the complexity of writing a correct and efficient equivalent in SQL, it might be better to mark it as Cypher-only for now, similar to other complex queries.
| benchmark("4a", "Friends of friends", COMPLEX_TRAVERSAL_ITERATIONS, | |
| "SELECT expand(both('KNOWS').both('KNOWS')) FROM Person WHERE id = :id", | |
| "MATCH (p:Person {id: $id})-[:KNOWS]-()-[:KNOWS]-(fof) " + | |
| "WHERE fof <> p AND NOT (p)-[:KNOWS]-(fof) RETURN DISTINCT fof"); | |
| benchmark("4a", "Friends of friends", COMPLEX_TRAVERSAL_ITERATIONS, | |
| null, // SQL equivalent is complex and needs to filter out 1-hop neighbors | |
| "MATCH (p:Person {id: $id})-[:KNOWS]-()-[:KNOWS]-(fof) " + | |
| "WHERE fof <> p AND NOT (p)-[:KNOWS]-(fof) RETURN DISTINCT fof"); |
|
|
||
| **Step 3: Compile to verify skeleton** | ||
|
|
||
| Run: `cd /Users/frank/projects/arcade/worktrees/ldbc-bechmark && mvn compile test-compile -pl engine -q` |
There was a problem hiding this comment.
The mvn command contains a hardcoded, user-specific absolute path: /Users/frank/projects/arcade/worktrees/ldbc-bechmark. This makes the command not directly runnable for other developers. It's better to use relative paths or placeholders. This applies to all shell commands in this document.
| Run: `cd /Users/frank/projects/arcade/worktrees/ldbc-bechmark && mvn compile test-compile -pl engine -q` | |
| Run: `mvn compile test-compile -pl engine -q` |
| private void prepareSampleIds() { | ||
| final ThreadLocalRandom rnd = ThreadLocalRandom.current(); | ||
| final int sampleSize = 100; | ||
|
|
||
| samplePersonIds = new long[sampleSize]; | ||
| samplePostIds = new long[sampleSize]; | ||
| sampleForumIds = new long[sampleSize]; | ||
| sampleCityNames = new String[sampleSize]; | ||
| sampleFirstNames = new String[sampleSize]; | ||
|
|
||
| for (int i = 0; i < sampleSize; i++) { | ||
| samplePersonIds[i] = rnd.nextLong(NUM_PERSONS); | ||
| samplePostIds[i] = rnd.nextLong(NUM_POSTS); | ||
| sampleForumIds[i] = rnd.nextLong(NUM_FORUMS); | ||
| sampleCityNames[i] = "City_" + (CONTINENTS.length + Math.min(COUNTRIES.length, NUM_PLACES / 3) + rnd.nextInt(Math.max(1, NUM_PLACES - CONTINENTS.length - Math.min(COUNTRIES.length, NUM_PLACES / 3)))); | ||
| sampleFirstNames[i] = FIRST_NAMES[rnd.nextInt(FIRST_NAMES.length)]; | ||
| } | ||
| } |
There was a problem hiding this comment.
The logic to generate a random city name is complex and duplicated from the data generation methods. This can be simplified for readability and maintainability by pre-calculating firstCityId and numCities and reusing them, just like in generatePersons() and generateOrganisations().
private void prepareSampleIds() {
final ThreadLocalRandom rnd = ThreadLocalRandom.current();
final int sampleSize = 100;
// Pre-compute city ID range for simplicity and consistency
final int firstCountryId = CONTINENTS.length;
final int numCountries = Math.min(COUNTRIES.length, NUM_PLACES / 3);
final int firstCityId = firstCountryId + numCountries;
final int numCities = NUM_PLACES - firstCityId;
samplePersonIds = new long[sampleSize];
samplePostIds = new long[sampleSize];
sampleForumIds = new long[sampleSize];
sampleCityNames = new String[sampleSize];
sampleFirstNames = new String[sampleSize];
for (int i = 0; i < sampleSize; i++) {
samplePersonIds[i] = rnd.nextLong(NUM_PERSONS);
samplePostIds[i] = rnd.nextLong(NUM_POSTS);
sampleForumIds[i] = rnd.nextLong(NUM_FORUMS);
if (numCities > 0) {
sampleCityNames[i] = "City_" + (firstCityId + rnd.nextInt(numCities));
} else {
sampleCityNames[i] = "City_0"; // Fallback if no cities
}
sampleFirstNames[i] = FIRST_NAMES[rnd.nextInt(FIRST_NAMES.length)];
}
}
Pull Request Review: LDBC Graph Benchmark Design & Implementation PlanThank you for this comprehensive planning work! I have reviewed both design and implementation documents against the project CLAUDE.md guidelines and best practices. Overall AssessmentStrengths:
Code Quality & Best PracticesAlignment with CLAUDE.md
Potential Issues1. Hard-coded Path (Critical) The implementation plan shows absolute paths like /Users/frank/projects/arcade/worktrees/ldbc-bechmark Issue: This is specific to your machine and will fail in CI/CD and for other developers. Fix: Remove absolute paths from all bash commands in the implementation plan. 2. Missing Regression Test Requirement
Performance ConsiderationsGood Practices
Concerns1. Missing WAL Configuration Design doc mentions WAL disabled during generation but implementation has no code for this. Fix: Add WAL disable/enable in populateGraph() method 2. Index Lookups in Tight Loops Millions of index lookups in KNOWS/LIKES generation could be slow. Consider batching or caching vertex references. 3. Memory Pressure With 30K persons × 40 KNOWS edges = ~600K bidirectional edges, this could create memory pressure. Suggestion: Add explicit transaction boundaries or increase commit frequency for KNOWS/LIKES generation. Test CoverageMissing Test Assertions Current implementation has no assertions - it just runs queries and collects metrics. Recommendation: Add verification tests to verify graph integrity, edge counts, and bidirectional relationships. Specific Technical Issues1. Potential NPE in formatRow If iterations < 20, percentile index calculation could fail. Add validation for small sample sizes. 2. Resource Leaks IndexCursor might need explicit closing. Check if it implements AutoCloseable. 3. Data Quality Birthday generation can create invalid dates like 1985-2-31. Consider using Java Time API. 4. Benchmark Warmup 5 warmup iterations may be insufficient for JIT compilation. JMH typically uses 10-20. Recommendations SummaryMust Fix:
Should Fix: Nice to Have: ConclusionThis is well-thought-out design work with excellent structure. Main issues:
Once these are addressed, this will be a valuable addition to ArcadeDB performance test suite. Approval Status: Please update the plan documents to address the Must Fix items before proceeding with implementation. |
Coverage summary from CodacySee diff coverage on Codacy
Coverage variation details
Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: Diff coverage details
Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: See your quality gate settings Change summary preferences |
Pull Request Review: Graph Benchmark Design & Implementation PlanOverviewThis PR adds comprehensive documentation for an LDBC-inspired graph benchmark. The design is well-thought-out and aligns well with ArcadeDB's architecture and the project's guidelines in CLAUDE.md. ✅ Strengths1. Excellent Design Documentation
2. Adherence to Project Guidelines
3. Performance Considerations
4. Comprehensive Coverage
🔍 Issues & RecommendationsCRITICAL: Hardcoded Path in Implementation PlanLocation: Throughout Task 1-11 in implementation plan Issue: The implementation plan contains hardcoded absolute paths specific to the author's machine: cd /Users/frank/projects/arcade/worktrees/ldbc-bechmarkThis appears in:
Recommendation: Update all commands to use a generic working directory variable or relative paths: cd ${ARCADEDB_ROOT} # or just assume proper working directory
mvn compile test-compile -pl engine -qMedium Priority Issues1. Typo in PathThe path contains "bechmark" instead of "benchmark": This typo propagates throughout the implementation plan. 2. Git Commit Commands in Implementation PlanLocation: Each task ends with commit commands The implementation plan includes explicit
Recommendation: Either:
3. Missing Import in Phase 1 TestLocation: docs/plans/2026-02-11-graph-benchmark-impl.md:1632 The implementation plan correctly notes adding: import static org.assertj.core.api.Assertions.assertThat;But this should be mentioned earlier in Task 1 skeleton imports, since it's used in Task 9's phase1_verifyGraphIntegrity test. Recommendation: Move this import to Task 1's initial skeleton. 4. Micrometer Version AssumptionLocation: docs/plans/2026-02-11-graph-benchmark-impl.md:322 The plan assumes Recommendation: Verify this version exists in the parent pom or specify the version explicitly if not defined. 5. Memory Considerations for Large ScaleThe medium scale creates ~3M edges and ~780K vertices. Recommendation: Consider documenting:
6. Test Isolation ConcernsThe test preserves the database between runs which is good for benchmarking, but:
Recommendation: Add:
Minor/Nitpick Issues7. Claude Attribution in CodeLocation: docs/plans/2026-02-11-graph-benchmark-impl.md:292 The implementation plan references:
This is metadata for AI tooling. While the code correctly excludes Claude as author (per CLAUDE.md guidelines), this comment in the markdown might be confusing for human readers. Recommendation: Consider moving this to a comment format or removing it from the final documentation. 8. Console Output FormattingThe fancy box-drawing characters in Recommendation: Test on Windows command prompt and consider a fallback format using plain ASCII. 9. Missing Edge Count in Stats
Recommendation: Consider adding edge counts (though this may be expensive - so optional). 10. Query 4d: Shortest Path RobustnessLocation: docs/plans/2026-02-11-graph-benchmark-impl.md:1560 The shortest path query might fail if no path exists between two random persons (disconnected graph). Recommendation: Either:
📋 Testing RecommendationsBefore merging, verify:
🎯 Security ConsiderationsNo security concerns - this is performance test code with synthetic data. 📊 Final AssessmentQuality Score: 8.5/10 This is high-quality design work with excellent alignment to project standards. The only significant issue is the hardcoded paths in the implementation plan, which makes it less useful as a generic implementation guide. Recommendation: Approve with changes
The actual implementation following this plan should produce a valuable benchmark tool for ArcadeDB. Review conducted following CLAUDE.md guidelines for ArcadeDB project standards. |
Single JUnit 5 test class that generates an LDBC Social Network Benchmark-inspired graph (8 vertex types, 14 edge types, ~790K vertices) and benchmarks ArcadeDB across creation, lookups, and traversals in both SQL and OpenCypher. Database is preserved between runs so only the first execution pays the generation cost. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Code Review: Graph Benchmark ImplementationOverviewThis PR adds a comprehensive LDBC SNB-inspired graph benchmark for ArcadeDB. The implementation creates a realistic social network graph and benchmarks both SQL and OpenCypher queries. Strengths ✅
Critical Issues 🔴1. Performance Issue: Repeated Index Lookups (Multiple locations)Many generation methods perform individual index lookups inside tight loops, resulting in millions of index lookups during generation. Recommendation: Cache vertex references in arrays to avoid ~3M+ index lookups. 2. Missing Exception Handling (GraphBenchmark.java:914-962)The 3. Invalid Date Generation (GraphBenchmark.java:600-604)Birthday generation can create invalid dates like "2000-2-30". Use proper date formatting. Major Issues 🟡4. No Schema Validation on Database Reuse (GraphBenchmark.java:148-161)If schema changes between runs, opening an existing database could cause errors. Add schema validation. 5. No Result Validation (GraphBenchmark.java:964-984)The RecommendationsMust Fix Before Merge:
Should Fix: Nice to Have: Overall AssessmentThis is a high-quality benchmark implementation with good design and comprehensive coverage. The main concerns are around performance optimizations and error handling. With the required fixes, this will be an excellent addition to ArcadeDB's test suite. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3410 +/- ##
==========================================
+ Coverage 62.44% 62.94% +0.49%
==========================================
Files 1251 1251
Lines 83627 83627
Branches 17118 17118
==========================================
+ Hits 52223 52639 +416
+ Misses 24075 23604 -471
- Partials 7329 7384 +55 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
2 participants
What does this PR do?