Use ARRAY instead of IN queries for Better Perf in Flat PG Collections by suddendust · Pull Request #258 · hypertrace/document-store

suddendust · 2025-12-10T09:50:05Z

Optimize IN/NOT IN Queries with`= ANY(ARRAY[])`

Overview

Replaced IN () syntax with = ANY(ARRAY[]) for better PostgreSQL query performance across flat and nested JSON collections.

SQL Examples

Flat/Top-level Scalar Fields

Before:

"item" IN (?, ?, ?)                    -- 3 parameters
"price" IN (?, ?, ?)                   -- 3 parameters

After:

"item" = ANY(?)                        -- 1 parameter (text array)
"price" = ANY(?)                       -- 1 parameter (int4 array)

Array Fields (overlap operator)

Before:

"tags" && ARRAY[?, ?, ?]::text[]       -- 3 parameters
"numbers" && ARRAY[?, ?]               -- 2 parameters

After:

"tags" && ?                            -- 1 parameter (text array)
"numbers" && ?                         -- 1 parameter (int4 array)

NOT IN (flat collections)

Before:

"item" NOT IN (?, ?)                   -- 2 parameters

After:

"item" IS NULL OR NOT ("item" = ANY(?)) -- 1 parameter (text array)

JSONB Array Fields

Before:

"props_dot_tags" = ANY(ARRAY[?::jsonb, ?::jsonb])  -- 2 parameters with explicit casts

After:

"props_dot_tags" = ANY(ARRAY[?::jsonb, ?::jsonb])  -- unchanged (JSONB requires explicit casts)

Testing

[x] Add integration tests.
[x] Tested in a live env.

codecov · 2025-12-10T09:53:42Z

Codecov Report

❌ Patch coverage is 77.90698% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.41%. Comparing base (4e8e31c) to head (6f0874e).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...ld/PostgresInRelationalFilterParserArrayField.java	63.63%	4 Missing and 4 partials ⚠️
...PostgresInRelationalFilterParserJsonPrimitive.java	50.00%	3 Missing and 2 partials ⚠️
...d/PostgresInRelationalFilterParserScalarField.java	68.75%	2 Missing and 3 partials ⚠️
...resContainsRelationalFilterParserNonJsonField.java	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #258      +/-   ##
============================================
- Coverage     80.48%   80.41%   -0.07%     
- Complexity     1330     1337       +7     
============================================
  Files           231      231              
  Lines          6020     6060      +40     
  Branches        537      545       +8     
============================================
+ Hits           4845     4873      +28     
- Misses          808      813       +5     
- Partials        367      374       +7

Flag	Coverage Δ
integration	`80.41% <77.90%> (-0.07%)`	⬇️
unit	`58.21% <47.67%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

puneet-traceable · 2025-12-10T10:03:38Z

...ostgres/query/v1/parser/filter/nonjson/field/PostgresInRelationalFilterParserArrayField.java

  /**
   * Generates SQL for scalar IN operator (used when array field has been unnested). Example:
-   * "tags_unnested" IN (?, ?, ?)
+   * "tags_unnested" = ANY(ARRAY[?, ?, ?])


@suddendust Why do we have multiple placeholders in ANY? This should be just one placeholder. If the placeholders are variable query plan will not be cached.

Also, even if we use IN, postgres query planner is automatically converting queries to ANY.

You're right. I am actually testing this. Basically, using just one placeholder for the entire array.

puneet-traceable · 2025-12-11T11:04:09Z

...ment-store/src/main/java/org/hypertrace/core/documentstore/postgres/utils/PostgresUtils.java

+   */
+  public static String inferSqlTypeFromValue(Object[] values) {
+    if (values.length == 0) {
+      return "text";


will this cause issues when type of the column is not text?

Yes, there'd be an issue, found this in texting. Fixed this now.

puneet-traceable · 2025-12-11T11:07:30Z

...ment-store/src/main/java/org/hypertrace/core/documentstore/postgres/utils/PostgresUtils.java

+   * @param values Array of values to infer type from
+   * @return PostgreSQL internal type name: "int4", "float8", "bool", or "text"
+   */
+  public static String inferSqlTypeFromValue(Object[] values) {


not very happy about this. How big is the lift to pass type information from entity service?

We now just need to start passing in the type info in IdentifierExpression. That'd entail changes in document-store and then the corresponding changes in entity-service.

…ace_in_with_any_for_perf

…to replace_in_with_any_for_perf

…alar fields

…to replace_in_with_any_for_perf

puneet-traceable · 2025-12-18T12:51:27Z

...ostgres/query/v1/parser/filter/nonjson/field/PostgresInRelationalFilterParserArrayField.java


-    String placeholders =
+      if (values.length == 0) {
+        return "1 = 0";


curious, what is this for?

Use ARRAY instead of IN queries for perf

49a35f1

suddendust requested review from avinashkolluru, kotharironak, skjindal93 and suresh-prakash as code owners December 10, 2025 09:50

Fixed PostgresQueryParserTest

2c3e834

puneet-traceable reviewed Dec 10, 2025

View reviewed changes

suddendust changed the title ~~Use ARRAY instead of IN queries for Better Perf in Flat PG Collections~~ [Draft] Use ARRAY instead of IN queries for Better Perf in Flat PG Collections Dec 10, 2025

Use array parms correctly

712b605

suddendust requested a review from puneet-traceable December 11, 2025 09:52

WIP

6b8579a

suddendust changed the title ~~[Draft] Use ARRAY instead of IN queries for Better Perf in Flat PG Collections~~ Use ARRAY instead of IN queries for Better Perf in Flat PG Collections Dec 11, 2025

puneet-traceable reviewed Dec 11, 2025

View reviewed changes

suddendust and others added 10 commits December 11, 2025 17:10

Handle empty arrays

201b442

Merge branch 'main' into replace_in_with_any_for_perf

571db94

Merge branch 'main' of github.com:hypertrace/document-store into repl…

b3effa7

…ace_in_with_any_for_perf

Remove instanceof checks by using compile-type type

211fb41

Merge remote-tracking branch 'myfork/replace_in_with_any_for_perf' in…

8e92e5b

…to replace_in_with_any_for_perf

Added backward compat tests for IN operator on top-level array and sc…

18df9d0

…alar fields

Fix failing test case

b599564

Merge branch 'main' into replace_in_with_any_for_perf

3502662

Added PostgresTypeExtractorTest

ffed724

Merge remote-tracking branch 'myfork/replace_in_with_any_for_perf' in…

6f0874e

…to replace_in_with_any_for_perf

suresh-prakash approved these changes Dec 18, 2025

View reviewed changes

puneet-traceable reviewed Dec 18, 2025

View reviewed changes

puneet-traceable approved these changes Dec 18, 2025

View reviewed changes

suresh-prakash merged commit 35fac78 into hypertrace:main Dec 18, 2025
5 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use ARRAY instead of IN queries for Better Perf in Flat PG Collections#258

Use ARRAY instead of IN queries for Better Perf in Flat PG Collections#258
suresh-prakash merged 14 commits intohypertrace:mainfrom
suddendust:replace_in_with_any_for_perf

suddendust commented Dec 10, 2025 •

edited

Loading

Uh oh!

codecov bot commented Dec 10, 2025 •

edited

Loading

Uh oh!

puneet-traceable Dec 10, 2025

Uh oh!

suddendust Dec 10, 2025

Uh oh!

puneet-traceable Dec 11, 2025

Uh oh!

suddendust Dec 11, 2025 •

edited

Loading

Uh oh!

puneet-traceable Dec 11, 2025

Uh oh!

suddendust Dec 11, 2025

Uh oh!

puneet-traceable Dec 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

suddendust commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Optimize IN/NOT IN Queries with= ANY(ARRAY[])

Overview

SQL Examples

Flat/Top-level Scalar Fields

Array Fields (overlap operator)

NOT IN (flat collections)

JSONB Array Fields

Testing

Uh oh!

codecov bot commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

puneet-traceable Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

suddendust Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

puneet-traceable Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

suddendust Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

puneet-traceable Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

suddendust Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

puneet-traceable Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

suddendust commented Dec 10, 2025 •

edited

Loading

Optimize IN/NOT IN Queries with`= ANY(ARRAY[])`

codecov bot commented Dec 10, 2025 •

edited

Loading

suddendust Dec 11, 2025 •

edited

Loading