Add tracing instrumentation for indexing paths #10273

rayshrey · 2023-09-29T14:57:49Z

Description

Added tracing instrumentation for indexing paths.

Sample spans:

{
    "traceID": "47a376fb3e446903a8c1100c38edfde1",
    "spanID": "5f3b3a3227c7c87d",
    "operationName": "bulkShardAction",
    "references": [
        {
            "refType": "CHILD_OF",
            "traceID": "47a376fb3e446903a8c1100c38edfde1",
            "spanID": "67488bb3020a56ff"
        }
    ],
    "startTime": 1697732790132198,
    "duration": 162497,
    "tags": [
        {
            "key": "otel.library.name",
            "type": "string",
            "value": "org.opensearch.telemetry"
        },
        {
            "key": "refresh_policy",
            "type": "string",
            "value": "false"
        },
        {
            "key": "bulk_request_items",
            "type": "int64",
            "value": 6
        },
        {
            "key": "index",
            "type": "string",
            "value": "moviess"
        },
        {
            "key": "shard_id",
            "type": "int64",
            "value": 0
        },
        {
            "key": "thread.name",
            "type": "string",
            "value": "opensearch[runTask-0][transport_worker][T#5]"
        },
        {
            "key": "node_id",
            "type": "string",
            "value": "KHWh8yCPSC-cT0CfS8AgRA"
        },
        {
            "key": "span.kind",
            "type": "string",
            "value": "server"
        },
        {
            "key": "internal.span.format",
            "type": "string",
            "value": "proto"
        }
    ],
    "logs": [],
    "processID": "p1",
    "warnings": null
}

{
    "traceID": "47a376fb3e446903a8c1100c38edfde1",
    "spanID": "7ff2c5c2910a11c7",
    "operationName": "dispatchedShardOperationOnPrimary",
    "references": [
        {
            "refType": "CHILD_OF",
            "traceID": "47a376fb3e446903a8c1100c38edfde1",
            "spanID": "0fa4e84f91da6d81"
        }
    ],
    "startTime": 1697732790135496,
    "duration": 137502,
    "tags": [
        {
            "key": "otel.library.name",
            "type": "string",
            "value": "org.opensearch.telemetry"
        },
        {
            "key": "refresh_policy",
            "type": "string",
            "value": "false"
        },
        {
            "key": "bulk_request_items",
            "type": "int64",
            "value": 6
        },
        {
            "key": "index",
            "type": "string",
            "value": "moviess"
        },
        {
            "key": "shard_id",
            "type": "int64",
            "value": 0
        },
        {
            "key": "thread.name",
            "type": "string",
            "value": "opensearch[runTask-0][write][T#1]"
        },
        {
            "key": "node_id",
            "type": "string",
            "value": "KHWh8yCPSC-cT0CfS8AgRA"
        },
        {
            "key": "span.kind",
            "type": "string",
            "value": "server"
        },
        {
            "key": "internal.span.format",
            "type": "string",
            "value": "proto"
        }
    ],
    "logs": [],
    "processID": "p1",
    "warnings": null
}

    "traceID": "47a376fb3e446903a8c1100c38edfde1",
    "spanID": "7c925fc99accc3fb",
    "operationName": "dispatchedShardOperationOnReplica",
    "references": [
        {
            "refType": "CHILD_OF",
            "traceID": "47a376fb3e446903a8c1100c38edfde1",
            "spanID": "cf1e69713797cc94"
        }
    ],
    "startTime": 1697732790269806,
    "duration": 22670,
    "tags": [
        {
            "key": "otel.library.name",
            "type": "string",
            "value": "org.opensearch.telemetry"
        },
        {
            "key": "shard_id",
            "type": "int64",
            "value": 0
        },
        {
            "key": "refresh_policy",
            "type": "string",
            "value": "false"
        },
        {
            "key": "thread.name",
            "type": "string",
            "value": "opensearch[runTask-2][write][T#3]"
        },
        {
            "key": "node_id",
            "type": "string",
            "value": "prD53FEMSKam7RdppmaR_Q"
        },
        {
            "key": "index",
            "type": "string",
            "value": "moviess"
        },
        {
            "key": "bulk_request_items",
            "type": "int64",
            "value": 6
        },
        {
            "key": "span.kind",
            "type": "string",
            "value": "server"
        },
        {
            "key": "internal.span.format",
            "type": "string",
            "value": "proto"
        }
    ],
    "logs": [],
    "processID": "p1",
    "warnings": [
        "clock skew adjustment disabled; not applying calculated delta of -3.266521ms"
    ]
}

Related Issues

Resolves #8555 partially

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-29T15:01:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL:
CommitID: 6a39f5c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-29T15:38:19Z

Compatibility status:

Checks if related components are compatible with change 533c716

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

sarthakaggarwal97 · 2023-09-29T17:08:26Z

@rayshrey I think the PR needs refactoring, can you please do a ./gradlew :server:precommit and ./gradlew :server:spotlessApply to check for the issues.
Please add the entry in the change log as well!

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java

server/src/main/java/org/opensearch/action/support/replication/TransportReplicationAction.java

server/src/main/java/org/opensearch/telemetry/tracing/AttributeNames.java

server/src/main/java/org/opensearch/telemetry/tracing/SpanBuilder.java

sarthakaggarwal97 · 2023-09-29T17:25:10Z

@rayshrey can you also please attach the outputs of the traces?

Gaganjuneja · 2023-10-01T13:11:34Z

Please make SpanName also meaningful so that if somebody just looking at the span should be able to recognise that what's happening in that Span.

rayshrey · 2023-10-03T04:54:35Z

Please make SpanName also meaningful so that if somebody just looking at the span should be able to recognise that what's happening in that Span.

@Gaganjuneja We are using shardPrimaryWrite, shardReplicaWrite and bulkShardAction as the span names currently.
According to me, these convey what is happening in the span from Indexing perspective clearly.

Do you have any suggestions/references for the span names ?

github-actions · 2023-10-03T10:46:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26683/
CommitID: 2ffce93
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sarthakaggarwal97 · 2023-10-19T18:04:57Z

@reta lgtm!

github-actions · 2023-10-19T20:17:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testRestartPrimary

URL: https://build.ci.opensearch.org/job/gradle-check/28483/
CommitID: 8db4a65
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/main/java/org/opensearch/action/resync/TransportResyncReplicationAction.java

Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

github-actions · 2023-10-20T11:21:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/28567/
CommitID: f46af51

server/src/main/java/org/opensearch/action/resync/TransportResyncReplicationAction.java

Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

github-actions · 2023-10-20T13:22:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/28582/
CommitID: d5581c9

rayshrey · 2023-10-20T13:37:07Z

@reta Thanks for the approval. Can we resolve the open conversations now and merge this PR (since we have got approvals from all the concerned folks) ?

server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java

reta · 2023-10-20T14:18:41Z

@reta Thanks for the approval. Can we resolve the open conversations now and merge this PR (since we have got approvals from all the concerned folks) ?

One minor thing, sorry about that, we missed it

Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

github-actions · 2023-10-20T15:01:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

URL: https://build.ci.opensearch.org/job/gradle-check/28594/
CommitID: 533c716
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-10-20T15:07:43Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-10273-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e691df09c66dcc1693897543fd7633c4b208ce48
# Push it to GitHub
git push --set-upstream origin backport/backport-10273-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-10273-to-2.x.

reta · 2023-10-20T15:11:55Z

@rayshrey please send manual backport to 2.x, thank you

) * Add tracing instrumentation for indexing paths Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix failing tests and review changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix test failures due to Span not being properly closed Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Changes to spans in primary and replica actions Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Review comments fixes and refactoring Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Precommit auto-changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Add refresh policy as attribute Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix changelog entry Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Instrument primary/replica write in TransportWriteAction instead of TransportShardBulkAction Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Modify SpanBuilder Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * spotlessApply and precommit Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Change span names Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Pass Noop Tracer instead of injected tracer Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Reverting previous changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Remove tracer variable from TransportShardBulkAction Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> --------- Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

monusingh-1 · 2023-10-27T09:53:55Z

@rayshrey please send manual backport to 2.x, thank you

@rayshrey backport to 2.x

) * Add tracing instrumentation for indexing paths Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix failing tests and review changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix test failures due to Span not being properly closed Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Changes to spans in primary and replica actions Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Review comments fixes and refactoring Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Precommit auto-changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Add refresh policy as attribute Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix changelog entry Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Instrument primary/replica write in TransportWriteAction instead of TransportShardBulkAction Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Modify SpanBuilder Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * spotlessApply and precommit Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Change span names Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Pass Noop Tracer instead of injected tracer Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Reverting previous changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Remove tracer variable from TransportShardBulkAction Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> --------- Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

* Add tracing instrumentation for indexing paths * Fix failing tests and review changes * Fix test failures due to Span not being properly closed * Changes to spans in primary and replica actions * Review comments fixes and refactoring * Precommit auto-changes * Add refresh policy as attribute * Fix changelog entry * Instrument primary/replica write in TransportWriteAction instead of TransportShardBulkAction * Modify SpanBuilder * spotlessApply and precommit * Change span names * Pass Noop Tracer instead of injected tracer * Reverting previous changes * Remove tracer variable from TransportShardBulkAction --------- Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

) * Add tracing instrumentation for indexing paths Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix failing tests and review changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix test failures due to Span not being properly closed Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Changes to spans in primary and replica actions Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Review comments fixes and refactoring Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Precommit auto-changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Add refresh policy as attribute Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix changelog entry Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Instrument primary/replica write in TransportWriteAction instead of TransportShardBulkAction Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Modify SpanBuilder Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * spotlessApply and precommit Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Change span names Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Pass Noop Tracer instead of injected tracer Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Reverting previous changes Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Remove tracer variable from TransportShardBulkAction Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> --------- Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

sarthakaggarwal97 reviewed Sep 29, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java Outdated Show resolved Hide resolved

sarthakaggarwal97 reviewed Sep 29, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/support/replication/TransportReplicationAction.java Outdated Show resolved Hide resolved

sarthakaggarwal97 reviewed Sep 29, 2023

View reviewed changes

server/src/main/java/org/opensearch/telemetry/tracing/AttributeNames.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/telemetry/tracing/SpanBuilder.java Outdated Show resolved Hide resolved

rayshrey marked this pull request as ready for review October 3, 2023 12:16

rayshrey requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, peternied, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja and dreamer-89 as code owners October 3, 2023 12:16

Merge branch 'main' into indexing-tracing-instrumentation

8db4a65

mgodwan reviewed Oct 20, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/resync/TransportResyncReplicationAction.java Show resolved Hide resolved

Pass Noop Tracer instead of injected tracer

f46af51

Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

mgodwan approved these changes Oct 20, 2023

View reviewed changes

reta requested changes Oct 20, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/resync/TransportResyncReplicationAction.java Outdated Show resolved Hide resolved

Reverting previous changes

d5581c9

Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

mgodwan approved these changes Oct 20, 2023

View reviewed changes

reta approved these changes Oct 20, 2023

View reviewed changes

reta reviewed Oct 20, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java Outdated Show resolved Hide resolved

Remove tracer variable from TransportShardBulkAction

533c716

Signed-off-by: Shreyansh Ray <rayshrey@amazon.com>

reta merged commit e691df0 into opensearch-project:main Oct 20, 2023
16 checks passed

reta added the backport 2.x Backport to 2.x branch label Oct 20, 2023

opensearch-trigger-bot bot added the backport-failed label Oct 20, 2023

rayshrey mentioned this pull request Oct 27, 2023

[Backport 2.x] Add tracing instrumentation for indexing paths #10962

Merged

8 tasks

rayshrey deleted the indexing-tracing-instrumentation branch November 2, 2023 09:50

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tracing instrumentation for indexing paths #10273

Add tracing instrumentation for indexing paths #10273

rayshrey commented Sep 29, 2023 •

edited

Loading

github-actions bot commented Sep 29, 2023

github-actions bot commented Sep 29, 2023 •

edited

Loading

sarthakaggarwal97 commented Sep 29, 2023

sarthakaggarwal97 commented Sep 29, 2023

Gaganjuneja commented Oct 1, 2023

rayshrey commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

sarthakaggarwal97 commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

github-actions bot commented Oct 20, 2023

github-actions bot commented Oct 20, 2023

rayshrey commented Oct 20, 2023

reta commented Oct 20, 2023

github-actions bot commented Oct 20, 2023

opensearch-trigger-bot bot commented Oct 20, 2023

reta commented Oct 20, 2023

monusingh-1 commented Oct 27, 2023

Add tracing instrumentation for indexing paths #10273

Add tracing instrumentation for indexing paths #10273

Conversation

rayshrey commented Sep 29, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Sep 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 29, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

sarthakaggarwal97 commented Sep 29, 2023

sarthakaggarwal97 commented Sep 29, 2023

Gaganjuneja commented Oct 1, 2023

rayshrey commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

sarthakaggarwal97 commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 20, 2023

Gradle Check (Jenkins) Run Completed with:

rayshrey commented Oct 20, 2023

reta commented Oct 20, 2023

github-actions bot commented Oct 20, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Oct 20, 2023

reta commented Oct 20, 2023

monusingh-1 commented Oct 27, 2023

rayshrey commented Sep 29, 2023 •

edited

Loading

github-actions bot commented Sep 29, 2023 •

edited

Loading