Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[internal/otelarrow] Flaky TestMRUReset #34252

Closed
djaglowski opened this issue Jul 25, 2024 · 7 comments · Fixed by #36070
Closed

[internal/otelarrow] Flaky TestMRUReset #34252

djaglowski opened this issue Jul 25, 2024 · 7 comments · Fixed by #36070
Labels
bug Something isn't working internal/otelarrow Stale

Comments

@djaglowski
Copy link
Member

Component(s)

internal/otelarrow

What happened?

Failing here and on main.

Likely introduced in #33579

Collector version

latest

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

=== FAIL: compression/zstd TestMRUReset (0.00s)
    mru_test.go:83: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/compression/zstd/mru_test.go:83
        	Error:      	Not equal: 
        	            	expected: 0
        	            	actual  : 1
        	Test:       	TestMRUReset

=== FAIL: test  (0.00s)
panic: test timed out after 10m0s
running tests:
	TestIntegrationSelfTracing (9m30s)

goroutine 66621 [running]:
testing.(*M).startAlarm.func1()
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:2259 +0x259
created by time.goFunc
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/time/sleep.go:176 +0x45

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc0003041a0, {0x1416bb7dd, 0x1a}, 0x141713ec8)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1649 +0x871
testing.runTests.func1(0x0?)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:2054 +0x85
testing.tRunner(0xc0003041a0, 0xc00051faf8)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1595 +0x262
testing.runTests(0xc000360f00?, {0x1413ffd40, 0x3, 0x3}, {0x1c?, 0x1e?, 0x1423243e0?})
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:2052 +0x8ae
testing.(*M).Run(0xc000360f00)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1925 +0xcd8
main.main()
	_testmain.go:51 +0x2be

goroutine 28393 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc042bf0f48?)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc042bf0f40)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/waitgroup.go:116 +0xa8
github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test.testIntegrationTraces({0x14199c020?, 0xc052cb0f50}, 0xc000380ea0, {0x141a6d1cf?, 0xf?}, 0x141c727bb?, 0x141714008, 0x141714020, 0x141714038)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/test/e2e_test.go:218 +0x972
github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test.TestIntegrationSelfTracing(0x0?)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/test/e2e_test.go:476 +0xbf
testing.tRunner(0xc000380ea0, 0x141713ec8)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 1
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1648 +0x846

goroutine 28398 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc011e0c1d8?)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc011e0c1d0)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/waitgroup.go:116 +0xa8
google.golang.org/grpc.(*Server).stop(0xc011e0c000, 0x1)
	C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1921 +0x4ac
google.golang.org/grpc.(*Server).GracefulStop(...)
	C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1883
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver.(*otelArrowReceiver).Shutdown(0xc0000e33b0, {0xc0009e5d68?, 0x1400511b4?})
ERROR rerun aborted because previous run had a suspected panic and some test may not have run
make[2]: *** [../../Makefile.Common:126: test] Error 3
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/otelarrow.go:172 +0x5f
github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent.(*SharedComponent).Shutdown.func1()
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/sharedcomponent/sharedcomponent.go:72 +0x6d
sync.(*Once).doSlow(0xc000d87b7c, 0xc0009e5ec0)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/once.go:74 +0xf1
sync.(*Once).Do(0xc000d87b7c, 0xc0009e5eb8?)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/once.go:65 +0x45
github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent.(*SharedComponent).Shutdown(0xc000d87b60, {0x14199c020?, 0xc052cb0f50})
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/sharedcomponent/sharedcomponent.go:71 +0xb9
github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test.testIntegrationTraces.func1()
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/test/e2e_test.go:179 +0x172
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test.testIntegrationTraces in goroutine 28393
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/test/e2e_test.go:175 +0x3ab

goroutine 66582 [chan send, 9 minutes]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*inFlightData).replyToCaller(...)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:485
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*inFlightData).consumeDone(0xc001598180, {0x14199bfe8, 0xc001111380}, 0xc008b39f10)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:480 +0x2e7
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).consumeAndRespond(0xc001f908c0, {0x14199bfe8?, 0xc001111380}, {0x14148ab40, 0xc02d5345a0}, 0xc001598180)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:659 +0x1fe
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*receiverStream).recvOne in goroutine 60128
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:642 +0xe5d

goroutine 60127 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc03dbb94c8?)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc03dbb94c0)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/waitgroup.go:116 +0xa8
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).anyStream(0xc001f908c0, {0x47fd8bc0, 0xc0550655a0}, {0xc0001be140, 0x49})
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:405 +0xeac
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).ArrowTraces(0x140d8bd25?, {0x1419a2978?, 0xc0550655a0})
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:309 +0x78
github.com/open-telemetry/otel-arrow/api/experimental/arrow/v1._ArrowTracesService_ArrowTraces_Handler({0x141619820?, 0xc001f908c0}, {0x14199f9f0?, 0xc040565220})
	C:/Users/runneradmin/go/pkg/mod/github.com/open-telemetry/otel-arrow@v0.24.0/api/experimental/arrow/v1/arrow_service_grpc.pb.go:122 +0xb6
go.opentelemetry.io/collector/config/configgrpc.(*ServerConfig).toServerOption.enhanceStreamWithClientInformation.func10({0x141619820, 0xc001f908c0}, {0x1419a0098, 0xc0000e21e0}, 0x14147bf40?, 0x141715ce8)
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/config/configgrpc@v0.105.1-0.20240717163034-43ed6184f9fe/configgrpc.go:451 +0x87
google.golang.org/grpc.(*Server).processStreamingRPC(0xc011e0c000, {0x14199bfe8, 0xc0008db5c0}, {0x1419a3500, 0xc000c6a300}, 0xc0007dd7a0, 0xc040e3e360, 0x1413ed640, 0x0)
	C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1680 +0x1f51
google.golang.org/grpc.(*Server).handleStream(0xc011e0c000, {0x1419a3500, 0xc000c6a300}, 0xc0007dd7a0)
	C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1794 +0x1a25
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1029 +0x15d
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 57553
	C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1040 +0x225

goroutine 60129 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc0000170c0?)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc0000170b8)
	C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/waitgroup.go:116 +0xa8
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*receiverStream).flushSender(0xc0000170b0, {0x47fd8bc0, 0xc0550655a0}, 0x0?, 0x0?)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:726 +0x58
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*receiverStream).srvSendLoop(0x0?, {0x14199c020, 0xc000aa90e0}, {0x47fd8bc0, 0xc0550655a0}, 0x0?, 0xc05400d560)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:745 +0x177
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).anyStream.func3()
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:394 +0x1ce
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).anyStream in goroutine 60127
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:388 +0x985

goroutine 28366 [chan receive, 9 minutes]:
google.golang.org/grpc.(*Server).Serve.func1()
	C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:855 +0xad
google.golang.org/grpc.(*Server).Serve(0xc011e0c000, {0x141999210?, 0xc040e34780})
	C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:915 +0xd73
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver.(*otelArrowReceiver).startGRPCServer.func1()
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/otelarrow.go:91 +0xf1
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver.(*otelArrowReceiver).startGRPCServer in goroutine 28398
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/otelarrow.go:88 +0x2a9

goroutine 66584 [chan send, 9 minutes]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*inFlightData).replyToCaller(...)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:485
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*inFlightData).consumeDone(0xc001599500, {0x14199bfe8, 0xc001111950}, 0xc002ed5f10)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:480 +0x2e7
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).consumeAndRespond(0xc001f908c0, {0x14199bfe8?, 0xc001111950}, {0x14148ab40, 0xc02d535368}, 0xc001599500)
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:659 +0x1fe
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*receiverStream).recvOne in goroutine 60128
	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:642 +0xe5d
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test	600.474s


### Additional context

_No response_
@djaglowski djaglowski added bug Something isn't working needs triage New item requiring triage labels Jul 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@axw axw mentioned this issue Jul 26, 2024
Merged
@mwear mwear removed the needs triage New item requiring triage label Jul 29, 2024
@github-actions github-actions bot mentioned this issue Jul 30, 2024
Closed
@pjanotti pjanotti mentioned this issue Jul 31, 2024
Merged
2 tasks
pjanotti added a commit to pjanotti/opentelemetry-service-contrib that referenced this issue Jul 31, 2024
@pjanotti
[chore] Skip TestMRUReset until open-telemetry#34252 is fixed
794eafd
@crobert-1 crobert-1 mentioned this issue Aug 1, 2024
Merged
dmitryax pushed a commit that referenced this issue Aug 1, 2024
@pjanotti
[chore] Skip TestMRUReset until #34252 is fixed (#34362)
07a7efd 
This is failing too much on Windows skipping the test until it is fixed.
dmitryax pushed a commit that referenced this issue Aug 1, 2024
@axw
[internal/otelarrow] [chore] fix TestMRUReset flakiness (#34260)
42f9faa 
**Description:**

Ensure the monotonic clock has advanced prior to resetting, to ensure
the item generation is before the reset.
The monotonic clock is guaranteed to be non-decreasing, but may be
identical in consecutive measurements.

**Link to tracking Issue:**


#34252

**Testing:**

I couldn't reproduce the issue locally - this issue would depend on the
system clock frequency.

**Documentation:**

N/A
@pjanotti
Copy link
Contributor

pjanotti commented Aug 1, 2024 •
edited
Loading

[EDIT: removed the comment here and moved it to issue #34719]

@pjanotti
Copy link
Contributor

pjanotti commented Aug 14, 2024 •
edited
Loading

[EDIT: removed the comment here and moved it to issue https://github.com//issues/34719]

@pjanotti pjanotti mentioned this issue Aug 14, 2024
Merged
@axw
Copy link
Contributor

axw commented Aug 15, 2024

@pjanotti are you referring to any/all flaky tests here? I don't see any mention of TestMRUReset in those failures.
I think we should close this issue and create new ones for other flaky tests.

@pjanotti
Copy link
Contributor

pjanotti commented Aug 16, 2024 •
edited
Loading

@axw yeah, makes more sense, moved to #34719

@github-actions GitHub Actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Oct 16, 2024
@jmacd
Copy link
Contributor

jmacd commented Oct 29, 2024

It appears that any timing-based test is unreliable on Windows.

@jmacd jmacd mentioned this issue Oct 29, 2024
Merged
codeboten pushed a commit that referenced this issue Oct 29, 2024
@jmacd
(internal/otelarrow) Fix timing test flake (#36070)
b54b755 
#### Description

Fixes MRU_Reset test.

#### Link to tracking issue

Fixes #34252 

#### Testing

This repairs a test by ensuring that a clock advances. 

#### Documentation

n/a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working internal/otelarrow Stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

(internal/otelarrow) Fix timing test flake jmacd/opentelemetry-collector-contrib
5 participants
@axw @mwear @jmacd @djaglowski @pjanotti