Skip to content

Conversation

LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Sep 25, 2025

Description

Followup of #4372

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin LantaoJin changed the title Pr/issues/4314 test [Doc] Enable doctest with Calcite Sep 25, 2025
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin LantaoJin added the documentation Improvements or additions to documentation label Sep 25, 2025
Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: Lantao Jin <ltjin@amazon.com>
| FLOOR(282474973688888.022) | FLOOR(9223372036854775807.022) | FLOOR(9223372036854775807.0000001) |
|----------------------------+--------------------------------+------------------------------------|
| 282474973688888 | 9223372036854775807 | 9223372036854775807 |
| 282474973688888.0 | 9.223372036854776e+18 | 9.223372036854776e+18 |
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the double values which are out of the double's precision. Keep the behavior same as Java's Math.floor(double) and Splunk's floor() math function.

Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: Lantao Jin <ltjin@amazon.com>
] as b
| stats avg(salary) by span(age, 10) as age_span, b.country;
fetched rows / total rows = 5/5
PPL> source = state_country as a | where country = 'USA' OR country = 'England' | left join ON a.name = b.name [ source = occupation | where salary > 0 | fields name, country, salary | sort salary | head 3 ] as b | stats avg(salary) by span(age, 10) as age_span, b.country;
Copy link
Member Author

@LantaoJin LantaoJin Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why this query failed in linux/doctest only. I cannot reproduce in local and IT. Will keep the PPL> to avoid failure.

    ppl_cmd.process("source = state_country as a | where country = 'USA' OR country = 'England' | left join ON a.name = b.name [ source = occupation | where salary > 0 | fields name, country, salary | sort salary | head 3 ] as b | stats avg(salary) by span(age, 10) as age_span, b.country")Expected:
    fetched rows / total rows = 3/3
    +-------------+----------+-----------+
    | avg(salary) | age_span | b.country |
    |-------------+----------+-----------|
    | null        | 40       | null      |
    | 70000.0     | 30       | USA       |
    | 100000.0    | 70       | England   |
    +-------------+----------+-----------+
Got:
    fetched rows / total rows = 2/2
    +-------------+----------+-----------+
    | avg(salary) | age_span | b.country |
    |-------------+----------+-----------|
    | 70000.0     | 30       | USA       |
    | 100000.0    | 70       | England   |
    +-------------+----------+-----------+
----------------------------------------------------------------------

Comment on lines 10 to +12
"user/ppl/cmd/grok.rst",
"user/ppl/cmd/head.rst",
"user/ppl/cmd/parse.rst",
"user/ppl/cmd/patterns.rst",
"user/ppl/cmd/rare.rst",
"user/ppl/cmd/search.rst",
"user/ppl/cmd/sort.rst",
"user/ppl/cmd/syntax.rst",
"user/ppl/cmd/trendline.rst",
"user/ppl/cmd/top.rst",
"user/ppl/cmd/where.rst",
"user/ppl/cmd/join.rst",
"user/ppl/cmd/lookup.rst",
"user/ppl/cmd/subquery.rst",
"user/ppl/general/identifiers.rst",
"user/ppl/general/datatypes.rst",
"user/ppl/functions/datetime.rst",
"user/ppl/functions/expressions.rst",
"user/ppl/functions/ip.rst",
"user/ppl/functions/json.rst",
"user/ppl/functions/math.rst",
"user/ppl/functions/relevance.rst",
"user/ppl/functions/string.rst"
"user/ppl/cmd/patterns.rst"
Copy link
Member Author

@LantaoJin LantaoJin Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the #4380 closed. but grok, parse, patterns are still not able to 100% match the result of doctest in v2. @songkant-aws can you fix it or link an issue. Or it just needs to update the doc.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reopen #4380

| 2024-07-01 00:00:00 | web-01 | 2 |
| 2024-07-01 00:00:00 | web-02 | 2 |
+---------------------+--------+-------+
os> source=events | timechart span=1h count() by host
Copy link
Member Author

@LantaoJin LantaoJin Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@selsong , when I change PPL> to os> in timechart.rst. I found many results cannot not match the expected results added in doc. timechart command is a Calcite enabled only command, which means the results in this file shouldn't be impacted by the action of setting calcite enabled as default. Please check why the original expected results cannot match the actual ones. cc @penghuo

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add issue to track. #4409


PPL query::

PPL> source=events | timechart span=1m avg(cpu_usage)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@selsong there are many tests which cannot run in timechart.rst, see the queries starting with PPL>. cc @penghuo

@Swiddis Swiddis merged commit 9143a44 into opensearch-project:main Sep 29, 2025
51 of 52 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4379-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9143a449f38bf08470ecc25826c474d8fd38d6c8
# Push it to GitHub
git push --set-upstream origin backport/backport-4379-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4379-to-2.19-dev.

penghuo pushed a commit to penghuo/os-sql that referenced this pull request Oct 7, 2025
penghuo pushed a commit to penghuo/os-sql that referenced this pull request Oct 7, 2025
(cherry picked from commit 9143a44)
Signed-off-by: Peng Huo <penghuo@gmail.com>
LantaoJin added a commit to LantaoJin/search-plugins-sql that referenced this pull request Oct 9, 2025
@LantaoJin LantaoJin added the backport-manually Filed a PR to backport manually. label Oct 9, 2025
LantaoJin added a commit to LantaoJin/search-plugins-sql that referenced this pull request Oct 9, 2025
(cherry picked from commit 9143a44)
Signed-off-by: Lantao Jin <ltjin@amazon.com>
LantaoJin added a commit that referenced this pull request Oct 9, 2025
(cherry picked from commit 9143a44)

Signed-off-by: Lantao Jin <ltjin@amazon.com>
asifabashar added a commit to asifabashar/sql that referenced this pull request Oct 10, 2025
* main-apple: (218 commits)
  Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
  Create fab-radar.yml
  PPL `fillnull` command enhancement (opensearch-project#4421)
  reverting to _doc + _id (opensearch-project#4435)
  Support `multisearch` command in calcite (opensearch-project#4332)
  Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
  [SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
  Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
  [Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
  Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
  Version bump: 3.3 (opensearch-project#4417)
  Add max/min eval functions (opensearch-project#4333)
  Support time modifiers in search command  (opensearch-project#4224)
  Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
  refactor span (opensearch-project#4334)
  Move release notes categories (opensearch-project#3818)
  [Doc] Enable doctest with Calcite (opensearch-project#4379)
  Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
  Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
  Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev backport-failed backport-manually Filed a PR to backport manually. documentation Improvements or additions to documentation v3.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants