Support time modifiers in search command

[yuancu]

Description

Support time modifier in search command.

Examples:

source=any earliest='2020-12-10'
source=any earliest='2020-12-10' latest='2025-09-10 13:00:12'
source=any latest=now
source=logs earliest=-7d
source=logs earliest='-1d@d'

Queries with earliest and latest time modifiers will be converted to a query string query with comparison conditions on the implicit timestamp field @timestamp. For example, query source=time_test earliest=-1year latest='-50d@w3' is converted to the following DSL:

{
  "query": {
    "query_string": {
      "query": "(@timestamp:>=now-1y) AND (@timestamp:<=now-50d/w-5d)",
    }
  }
}

Work items

• support absolute time like 2012-10-10
• support relative time range like now, -30m, -2h@h, @w0
• support chained time range offsets like -mon@mon+7d
• Unix epoch time like 1 (UTC January 1, 1970 at 12:00:01 AM)

Related Issues

Resolves #4135

Implementation Walk-through

Implementation Details

1. Time Modifier Processing

The core functionality is implemented in the visitTimeModifierValue and visitTimeModifierExpression methods in AstExpressionBuilder.java:

• visitTimeModifierValue: Converts time values to OpenSearch date math expressions
• visitTimeModifierExpression: Creates comparison conditions for @timestamp field in query string

2. Time Format Conversion

The PR extends DateTimeUtils.java with the parseRelativeTime method, which transforms PPL time expressions to OpenSearch date math expressions.

3. Query String Query Creation

Time modifiers are converted to search comparisons with a query string on the implicit @timestamp field, which is now defined as a constant in OpenSearchConstants.java. For example, search earliest=-10days@month latest=now() is converted to a query string query like the following:

{
  "query": {
    "query_string": {
      "query": "(@timestamp:>=-10d/M) AND (@timestamp:<=now)"
    }
  }
}

PPL Time Modifiers to OpenSearch Date Math Examples

• Absolute Times

  PPL Time Modifier	OpenSearch Date Math	Explanation
  '2023-01-01'	2023-01-01	Simple date
  '2023-01-01 13:45:30'	2023-01-01T13:45:30Z	Date with time

• Relative Times

  PPL Time Modifier	OpenSearch Date Math	Explanation
  now	now	Current time
  -30s	now-30s	30 seconds ago
  -1h	now-1h	1 hour ago
  +1d	now+1d	1 day in future

• Snapping to Time Units (Using @)

  PPL Time Modifier	OpenSearch Date Math	Explanation
  -1d@d	now-1d/d	1 day ago, rounded to day
  @h	now/h	Round to current hour
  @m	now/M	Round to current month

• Week Days

  PPL Time Modifier	OpenSearch Date Math	Explanation
  @w0	/w-1d or /w+6d	Previous Sunday (depending on current day)
  @w1	/w	Start of week (Monday)
  @w4	/w+3d	Thursday of current week

• Special Cases

PPL Time Modifier	OpenSearch Date Math	Explanation
@q	/M or /M-1M or /M-2M	Start of current quarter
-2q	-6M	2 quarters ago (6 months)
1234.567	1234567	Unix timestamp (seconds → milliseconds)

• Complex Expressions

PPL Time Modifier	OpenSearch Date Math	Explanation
-1d+1y@mon	now-1d+1y/M	1 day ago + 1 year, rounded to month
-3d@d-2h+10m	now-3d/d-2h+10m	3 days ago, day start, minus 2 hours, plus 10 minutes

Limitations

Currently, the following formats has to be quoted:

• chained offsets. e.g. -1day@month+1h or +1year-4day

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[vamsimanohar]

can you review this: #4152
I am changing the grammar for search expression and its functionality.
We might need to reevaluate the implementation.

[yuancu]

Hi @vamsimanohar, thank you for reminding! I think the functionality does not overlap, I'll re-implement this based on your grammar.

[vamsimanohar]

@yuancu Few things to keep in mind.

We fundamentally want search command to be always push down and translate toquery_stringfunction in DSL which follows Lucene query syntax. Anything we add to search command should get translated to lucene query.

I think it should be possible with your changes.

[yuancu]

This dependency is added for the access of org.opensearch.common.time.DateMathParser -- I want to make sure that the parsed OpenSearch date math like now-1d/M-2M returns the intended instant for -1d@q

[penghuo]

parseRelativeTime() is called by PPL module only, right? if move parseRelativeTime to PPL module, does opensearch depedency still needed?

[yuancu]

Yes, it is only called by PPL module. If I still want to assure the correctness of the parsed instant, I'll have to move this dependency to the PPL module.

Is there any concern over this dependency? If it's for the performance, as the dependency is only test time, I assume there would be no harm to the runtime performance.

[vamsimanohar]

we want core modules to not depend on any opensearch dependencies. This helps in our future plans to publish ppl as a library or running sql engine as a separate service.

If we can avoid, its better to avoid.

[yuancu]

Thanks for the explanation! I've removed the dependency and related tests.

[vamsimanohar]

@yuancu Did we cover every usecase mentioned in the description? Lets try to close this PR by end of the day. I can stay late and get on a call with you for review.

You probably might need to refactor from this PR:#4334

[yuancu]

@yuancu Did we cover every usecase mentioned in the description? Lets try to close this PR by end of the day. I can stay late and get on a call with you for review.

You probably might need to refactor from this PR:#4334

Unit test and explaining IT covers all of them. I did not cover some relative time modifier in integration test because they either have something to do with the current time, which may be flaky to mock and test. I'll try to cover more of them in the integration test. E.g. there exists unknown latency after I write a timestamp and before the query is executed.

The requested change has been addressed.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4224-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e4685137de681a5f513a88c681b406ad5a786c27
# Push it to GitHub
git push --set-upstream origin backport/backport-4224-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4224-to-2.19-dev.