OpenTelemetry instrumentation example #296

emanuelef · 2023-07-13T06:27:21Z

What was changed

Added example to show how to instrument OpenTelemetry in the worker and client

Why?

Currently there is no sample to show how to instrument a Temporal worker and a client with OpenTelemetry.
This can save some time and clarifies what is the minimum needed to be able to export traces.

There's also an open issue:
#269

Checklist

Closes
How was this tested:

Run as most of the samples, no need to set OTLP endpoint and see the spans in the stdout.
Also tested with Honeycomb.io.

Any docs updates needed?

Updated README.md

otel/README.md

otel/otel.go

cretz · 2023-07-13T21:31:46Z

otel/starter/main.go

+	"go.temporal.io/sdk/client"
+	"go.temporal.io/sdk/interceptor"
+
+	otelworkflow "github.com/temporalio/samples-go/otel"


Suggested change

otelworkflow "github.com/temporalio/samples-go/otel"

"github.com/temporalio/samples-go/otel"

No need for package alias and doesn't seem purely about workflows

it will now clash with "go.temporal.io/sdk/contrib/opentelemetry"

cretz · 2023-07-13T21:32:30Z

otel/starter/main.go

+		log.Fatalln("Unable to execute workflow", err)
+	}
+
+	log.Println("Started workflow", "WorkflowID", we.GetID(), "RunID", we.GetRunID())


The standard Go logger doesn't usually expect key-value pair type logs like structured logging does

This line is common to many other examples in the repo like: https://github.com/search?q=repo%3Atemporalio%2Fsamples-go+log.Println%28%22Started+workflow%22&type=code

👍 We get it wrong in those too, oh well. Can ignore.

otel/otel-setup.go

otel/starter/main.go

cretz · 2023-07-13T21:35:27Z

Thanks! Looks great, just added some comments.

cretz

Looks good! I think we can just drop the JSON and I'll merge. Thanks for the help here!

cretz · 2023-07-14T21:10:59Z

@emanuelef - I think you may need a go mod tidy run from the repo root

emanuelef · 2023-07-15T05:26:21Z

When I do I get:

go: finding module for package go.temporal.io/server/tools/cli/plugin
github.com/temporalio/samples-go/snappycompress/plugin imports
        go.temporal.io/server/tools/cli/plugin: module go.temporal.io/server@latest found (v1.21.2), but does not contain package go.temporal.io/server/tools/cli/plugin

cretz · 2023-07-17T12:40:47Z

I think this is because you inadvertently updated API and server. Try to set go.temporal.io/api and go.temporal.io/server back to what they are in main's go.mod manually (I guess go.temporal.io/api v1.7.1-0.20220131203817-08fe71b1361d and go.temporal.io/server v1.15.2), then run go mod tidy. Otherwise, we can open another PR to update server (looks like some packages moved around in there).

cretz · 2023-07-24T13:52:41Z

If OpenTelemetry sample cannot be added without updating server, I am afraid this is going to have to wait on #264 to be solved.

emanuelef · 2023-07-27T10:55:17Z

If OpenTelemetry sample cannot be added without updating server, I am afraid this is going to have to wait on #264 to be solved.

Is this something expected to be done soon ?

cretz · 2023-07-31T13:59:15Z

I don't have a specific timeline, sorry

Update Go SDK to v1.24.0

* created batch-sliding-window * renamed package * Added activities and workflows * Ported activities and workflows from Java * sliding window progress * simplistic test passes * Failure to drain reproduction * Added error handling to record_processor_workflow * added sort * Fixed batch partitions * Added description to README * Fixed nondeterminism errors

Add session failure sample

Update Go SDK to v1.25.0

emanuelef · 2024-01-09T22:18:35Z

@Quinn-With-Two-Ns can you have a look ? With the latest changes the tests (when running locally with make test) are passing.
The open telemetry example is running as well.
But I'm still having errors reported on grpc-proxy and codec-server due to the following imported modules:
go.temporal.io/server/common/authorization
go.temporal.io/server/common/config

Thanks

cretz

Looks mostly good, we need to get CI working though

README.md

cretz · 2024-01-11T21:58:58Z

go.mod

 	github.com/pborman/uuid v1.2.1
-	github.com/prometheus/client_golang v1.16.0


Hrmm, wonder why this got removed. I think CI is failing here, may need a go mod tidy

This is present as indirect: https://github.com/emanuelef/samples-go/blob/598b901b62e521741e3a78a27f071abcdc2188f9/go.mod#L52C1-L52C1

Tried to run go mod tidy anyway and got:

go: github.com/temporalio/samples-go/codec-server/codec-server imports go.temporal.io/server/common/config imports go.temporal.io/server/common/telemetry imports go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc imports go.opentelemetry.io/otel/exporters/otlp/internal: module go.opentelemetry.io/otel/exporters/otlp@latest found (v0.20.1), but does not contain package go.opentelemetry.io/otel/exporters/otlp/internal go: github.com/temporalio/samples-go/codec-server/codec-server imports go.temporal.io/server/common/config imports go.temporal.io/server/common/telemetry imports go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc imports go.opentelemetry.io/otel/exporters/otlp/otlpmetric/internal/oconf imports go.opentelemetry.io/otel/exporters/otlp/internal/envconfig: module go.opentelemetry.io/otel/exporters/otlp@latest found (v0.20.1), but does not contain package go.opentelemetry.io/otel/exporters/otlp/internal/envconfig

go mod tidy has to work on our samples. It appears that our server expects an older version of go.opentelemetry.io/otel (https://github.com/temporalio/temporal/blob/v1.22.3/go.mod#L39). I think if you put to go.mod back as it was and just add that version of otel and then go mod tidy it should figure it out. You may have to work with it. But you can't do a go get -u or similar overall because it updated lots of unrelated dependencies here I think.

Fixed and now go mod tidy doesn't complain, thanks.

opentelemetry/worker/main.go

Co-authored-by: Chad Retz <chad.retz@gmail.com>

emanuelef · 2024-01-12T20:50:03Z

@cretz ci-build is now passing

emanuelef added 2 commits July 12, 2023 22:22

Working on OTeL instrumentation example

911122e

working example

800fd7f

emanuelef marked this pull request as draft July 13, 2023 06:27

emanuelef and others added 5 commits July 13, 2023 07:42

fix run command lines

94d81de

Working on README

ec111aa

Update README.md

4482e38

more documentation

e93ad95

fix example formatting

092e3bd

emanuelef marked this pull request as ready for review July 13, 2023 21:04

added comment in documentation

e086549

cretz reviewed Jul 13, 2023

View reviewed changes

emanuelef added 8 commits July 14, 2023 05:40

renamed to opentelemetry

c1caf06

more comments

5197bae

embed image

e109cdf

reorder imports

06dad84

reordered imports

c58b216

renamed files

c2d3c32

removed default host

57e0e06

refactoring to avoid global interceptor

898c142

emanuelef requested a review from cretz July 14, 2023 05:20

clean up

5a3186f

cretz reviewed Jul 14, 2023

View reviewed changes

removed JSON sample

1910f4a

emanuelef requested a review from cretz July 14, 2023 20:06

cretz approved these changes Jul 14, 2023

View reviewed changes

emanuelef requested a review from cretz July 15, 2023 09:35

emanuelef mentioned this pull request Sep 27, 2023

Update Temporal server version to v1.22 #293

Closed

emanuelef mentioned this pull request Dec 30, 2023

Add open telemetry sample #320

Closed

Quinn-With-Two-Ns and others added 14 commits January 3, 2024 21:00

Update Go SDK to v1.24.0 (temporalio#299)

e9328de

Update Go SDK to v1.24.0

Add session failure sample (temporalio#303)

df619e7

Add session failure sample

bump Server to 1.22 (temporalio#305)

e980844

Update Go SDK to v1.25.0 (temporalio#306)

7b707f9

Update Go SDK to v1.25.0

Add start delay sample (temporalio#309)

5282b70

fix: os.WriteFile path error (temporalio#310)

e622a88

Rename Sticky Activities sample (temporalio#316)

bd0f7f4

Update server to v1.22.3 (temporalio#321)

fa5c2aa

fix dependencies

b474b26

more imports

2e29d58

added more modules

6431ae0

Merge branch 'main' into otel-example

51dff88

fix go.sum

eef709d

cretz reviewed Jan 11, 2024

View reviewed changes

emanuelef and others added 5 commits January 12, 2024 04:58

Update README.md

a877939

Co-authored-by: Chad Retz <chad.retz@gmail.com>

Update opentelemetry/worker/main.go

b21a7cc

Co-authored-by: Chad Retz <chad.retz@gmail.com>

Update opentelemetry/worker/main.go

598b901

Co-authored-by: Chad Retz <chad.retz@gmail.com>

fix go.mod

015fd4b

fix ci-build

68fe937

emanuelef requested a review from cretz January 12, 2024 15:43

Quinn-With-Two-Ns merged commit 2590097 into temporalio:main Jan 16, 2024
4 checks passed

Quinn-With-Two-Ns mentioned this pull request Jan 21, 2024

[Feature Request] Add a sample with opentelemetry tracing #269

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenTelemetry instrumentation example #296

OpenTelemetry instrumentation example #296

emanuelef commented Jul 13, 2023 •

edited

Loading

cretz Jul 13, 2023

emanuelef Jul 14, 2023

cretz Jul 13, 2023

emanuelef Jul 14, 2023

cretz Jul 14, 2023

cretz commented Jul 13, 2023

cretz left a comment

cretz commented Jul 14, 2023

emanuelef commented Jul 15, 2023 •

edited

Loading

cretz commented Jul 17, 2023 •

edited

Loading

cretz commented Jul 24, 2023

emanuelef commented Jul 27, 2023

cretz commented Jul 31, 2023

emanuelef commented Jan 9, 2024 •

edited

Loading

cretz left a comment

cretz Jan 11, 2024

emanuelef Jan 12, 2024 •

edited

Loading

cretz Jan 12, 2024 •

edited

Loading

emanuelef Jan 12, 2024

emanuelef commented Jan 12, 2024

	otelworkflow "github.com/temporalio/samples-go/otel"
	"github.com/temporalio/samples-go/otel"

		github.com/pborman/uuid v1.2.1
		github.com/prometheus/client_golang v1.16.0

OpenTelemetry instrumentation example #296

OpenTelemetry instrumentation example #296

Conversation

emanuelef commented Jul 13, 2023 • edited Loading

What was changed

Why?

Checklist

cretz Jul 13, 2023

Choose a reason for hiding this comment

emanuelef Jul 14, 2023

Choose a reason for hiding this comment

cretz Jul 13, 2023

Choose a reason for hiding this comment

emanuelef Jul 14, 2023

Choose a reason for hiding this comment

cretz Jul 14, 2023

Choose a reason for hiding this comment

cretz commented Jul 13, 2023

cretz left a comment

Choose a reason for hiding this comment

cretz commented Jul 14, 2023

emanuelef commented Jul 15, 2023 • edited Loading

cretz commented Jul 17, 2023 • edited Loading

cretz commented Jul 24, 2023

emanuelef commented Jul 27, 2023

cretz commented Jul 31, 2023

emanuelef commented Jan 9, 2024 • edited Loading

cretz left a comment

Choose a reason for hiding this comment

cretz Jan 11, 2024

Choose a reason for hiding this comment

emanuelef Jan 12, 2024 • edited Loading

Choose a reason for hiding this comment

cretz Jan 12, 2024 • edited Loading

Choose a reason for hiding this comment

emanuelef Jan 12, 2024

Choose a reason for hiding this comment

emanuelef commented Jan 12, 2024

emanuelef commented Jul 13, 2023 •

edited

Loading

emanuelef commented Jul 15, 2023 •

edited

Loading

cretz commented Jul 17, 2023 •

edited

Loading

emanuelef commented Jan 9, 2024 •

edited

Loading

emanuelef Jan 12, 2024 •

edited

Loading

cretz Jan 12, 2024 •

edited

Loading