Skip to content

Conversation

ahkcs
Copy link
Contributor

@ahkcs ahkcs commented Sep 18, 2025

Description

This PR introduces the multisearch command, a new PPL command that allows combining results from multiple search subsearches into a single unified result set.

Resolves: #4348

Key Features:

  • Combines results from 2+ search subsearches with different filtering criteria
  • Supports ALL PPL commands within subsearches (where, eval, fields, stats, sort, dedup, etc.)
  • Enables timestamp-based result interleaving for time-series data analysis
  • Uses efficient UNION ALL + ORDER BY architecture for optimal performance

Example Usage:

-- Basic age group analysis
source=accounts | multisearch
  [search source=accounts | where age < 30 | eval age_group = "young"]
  [search source=accounts | where age >= 30 | eval age_group = "adult"]
| stats count by age_group

-- Success rate monitoring pattern
source=logs | multisearch
  [search source=logs | where status="success" | eval result="success"]
  [search source=logs | where status!="success" | eval result="total"]
| stats count(eval(result="success")) as success_count, count() as total_count

@ahkcs ahkcs force-pushed the feat/multisearch_cmd branch from 92ab992 to 5c43a2b Compare September 23, 2025 23:57
ahkcs added 3 commits October 1, 2025 10:27
Signed-off-by: Kai Huang <ahkcs@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
@ahkcs ahkcs force-pushed the feat/multisearch_cmd branch from ae6eff9 to bd249bf Compare October 1, 2025 17:43
Signed-off-by: Kai Huang <ahkcs@amazon.com>
dai-chen
dai-chen previously approved these changes Oct 1, 2025
@@ -0,0 +1,200 @@
{"index":{"_id":"1"}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np: do we really need so many test data to demonstrate the command usage in documentation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking we can reuse this dataset for future doctest for timestamp

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated to trim the dataset to 20 records

Signed-off-by: Kai Huang <ahkcs@amazon.com>
ykmr1224
ykmr1224 previously approved these changes Oct 1, 2025
Signed-off-by: Kai Huang <ahkcs@amazon.com>
@dai-chen
Copy link
Collaborator

dai-chen commented Oct 1, 2025

Do we add and benchmark Big5 query for each new command like this?

@ahkcs
Copy link
Contributor Author

ahkcs commented Oct 1, 2025

Do we add and benchmark Big5 query for each new command like this?

I think we can add IT tests to benchmark Big5 queries like this, I'll open a separate PR for this:
#4163

@ahkcs
Copy link
Contributor Author

ahkcs commented Oct 1, 2025

Do we add and benchmark Big5 query for each new command like this?

For Multisearch command, I think the result will be exactly the same as Append command as they share the same implementation

@RyanL1997 RyanL1997 merged commit ca5a5bd into opensearch-project:main Oct 2, 2025
51 of 52 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4332-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ca5a5bd6ee36ecdbbb2d3dc7a4b73ba28ecafa95
# Push it to GitHub
git push --set-upstream origin backport/backport-4332-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4332-to-2.19-dev.

ahkcs added a commit to ahkcs/sql that referenced this pull request Oct 2, 2025
---------

Signed-off-by: Kai Huang <ahkcs@amazon.com>
(cherry picked from commit ca5a5bd)
penghuo pushed a commit that referenced this pull request Oct 3, 2025
…4431)

* Support `multisearch` command in calcite (#4332)

---------

Signed-off-by: Kai Huang <ahkcs@amazon.com>
(cherry picked from commit ca5a5bd)

* fix ExplainIT

Signed-off-by: Kai Huang <ahkcs@amazon.com>

* put missing file back

Signed-off-by: Kai Huang <ahkcs@amazon.com>

---------

Signed-off-by: Kai Huang <ahkcs@amazon.com>
@LantaoJin LantaoJin added the backport-manually Filed a PR to backport manually. label Oct 9, 2025
asifabashar added a commit to asifabashar/sql that referenced this pull request Oct 10, 2025
* main-apple: (218 commits)
  Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
  Create fab-radar.yml
  PPL `fillnull` command enhancement (opensearch-project#4421)
  reverting to _doc + _id (opensearch-project#4435)
  Support `multisearch` command in calcite (opensearch-project#4332)
  Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
  [SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
  Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
  [Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
  Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
  Version bump: 3.3 (opensearch-project#4417)
  Add max/min eval functions (opensearch-project#4333)
  Support time modifiers in search command  (opensearch-project#4224)
  Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
  refactor span (opensearch-project#4334)
  Move release notes categories (opensearch-project#3818)
  [Doc] Enable doctest with Calcite (opensearch-project#4379)
  Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
  Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
  Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev backport-failed backport-manually Filed a PR to backport manually. calcite calcite migration releated feature PPL Piped processing language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Support multisearch command for PPL

7 participants