Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[crash] stdout json exporter crashes when many spans generated at once #2264

Closed
lizthegrey opened this issue Sep 28, 2021 · 0 comments · Fixed by #2265
Closed

[crash] stdout json exporter crashes when many spans generated at once #2264

lizthegrey opened this issue Sep 28, 2021 · 0 comments · Fixed by #2265
Labels
bug Something isn't working

Comments

@lizthegrey
Copy link
Member

Description

Crash due to insufficient locking in #2196 which switched from a thread-safe JSON package method to a non-thread-safe JSON exporter object that uses a single buffer

Environment

  • OS: Linux
  • Architecture: amd64
  • Go Version: 1.17.1
  • opentelemetry-go version: v1.0.0

Steps To Reproduce

Generate many spans at once, with JSON and pretty-printing on, and simplespanexporter used as the export method e.g. https://glitch.com/edit/#!/intro-to-o11y-go?path=src%2Fmain.go%3A85%3A37

Expected behavior

No crash, all spans exported successfully and in sequence

Observed behavior

goroutine 71 [running]:
bytes.(*Buffer).WriteByte(0xc000172cb0, 0xc0)
	/tmp/go/src/bytes/buffer.go:269 +0x85
encoding/json.newline(0xc0004c4340, {0x0, 0x0}, {0xa199c0, 0x1}, 0x3)
	/tmp/go/src/encoding/json/indent.go:63 +0x45
encoding/json.Indent(0xc0005de330, {0xc00048e000, 0x439, 0x0}, {0x0, 0x0}, {0xa199c0, 0x1})
	/tmp/go/src/encoding/json/indent.go:118 +0x245
encoding/json.(*Encoder).Encode(0xc0000889b0, {0x935fe0, 0xc000142300})
	/tmp/go/src/encoding/json/stream.go:225 +0x18b
go.opentelemetry.io/otel/exporters/stdout/stdouttrace.(*Exporter).ExportSpans(0xc0003250b0, {0x98ef9b6c6f889cc8, 0xe096794a2d1d49d}, {0xc0005e1760, 0x0, 0x1})
	/app/pkg/mod/go.opentelemetry.io/otel/exporters/stdout/stdouttrace@v1.0.0/trace.go:86 +0x1dc
go.opentelemetry.io/otel/sdk/trace.(*simpleSpanProcessor).OnEnd(0xc00007d580, {0xac0df8, 0xc000142180})
	/app/pkg/mod/go.opentelemetry.io/otel/sdk@v1.0.0/trace/simple_span_processor.go:58 +0x1e6
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc000142000, {0x0, 0x0, 0x0})
	/app/pkg/mod/go.opentelemetry.io/otel/sdk@v1.0.0/trace/span.go:280 +0x9e2
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace.(*clientTracer).end(0xc0004a45a0, {0xc0005f2fc0, 0xc00014b4d8}, {0x0, 0x0}, {0xc0005a3600, 0x2, 0x2})
	/app/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace@v0.24.0/clienttrace.go:224 +0x46d
@lizthegrey lizthegrey added the bug Something isn't working label Sep 28, 2021
lizthegrey added a commit to honeycombio/opentelemetry-go that referenced this issue Sep 28, 2021
Aneurysm9 pushed a commit that referenced this issue Sep 30, 2021
#2265)

* lock accesses to encoder

fixes #2264

* move locking outside loop to avoid deadlock

* Update CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
1 participant