windows_event_log source permanently freezes after ~3 minutes of inactivity (PR #24305)

[SwimSalt]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Describe the bug

The new pull-based windows_event_log source works perfectly during continuous event bursts. However, if the event channel experiences a period of inactivity (around 2.5 to 3 minutes), the source never wakes up again when new events arrive. The Vector process stays at 0% CPU indefinitely, requiring a manual service restart to resume ingestion. I have consistently reproduced this 3 times in a row.

During this idle time, the internal hyper/h2 networking client gracefully closes the idle connection (GoAway NO_ERROR). It is unclear if this networking idle timeout propagates backpressure that interrupts the Win32 pulling loop, or if the Win32 Event Log API itself naturally drops the handle state after 3 minutes.

Technical Theory

Looking at the new implementation in PR #24305, this behaves like a lost-wakeup deadlock with the Win32 Event Log API.
If EvtNext isn't drained completely until it returns ERROR_NO_MORE_ITEMS (for instance, if slight backpressure from the network disconnect interrupts the drain loop), the Windows Event Log service will never signal the event handle again.
Additionally, if WaitForMultipleObjects is using an INFINITE timeout, the thread cannot self-recover from a missed signal. Implementing a safe timeout (e.g., 5 seconds) and ensuring a strict ERROR_NO_MORE_ITEMS drain loop before waiting again might solve this frozen state.

To Reproduce

1. Configure Vector with a windows_event_log source reading from ForwardedEvents (WEF) and outputting to a remote vector sink via TCP/gRPC.

2. Let Vector ingest a burst of events.

3. Incoming events stop for ~3 minutes.

4. Observe the vector.log in debug mode: h2::codec::framed_write sends a GoAway NO_ERROR frame due to the idle period.

5. Generate new Windows events in the channel.

6. Vector remains completely silent and does not resume ingestion.

Configuration

api:
  enabled: true
  address: "127.0.0.1:8686"

sources:
  windows_events:
    type: "windows_event_log"
    channels: ["ForwardedEvents"]
    read_existing_events: true
    events_per_second: 5000
    max_event_data_length: 32768

sinks:
  linux_aggregator:
    type: "vector"
    inputs: ["windows_events"]
    address: "<LINUX_AGGREGATOR_IP>:9000"
    compression: true
    request:
      timeout_secs: 15
      concurrency: 2
    buffer:
      type: "disk"
      max_size: 1048576000
      when_full: "block"

Version

vector 0.55.0 (x86_64-pc-windows-msvc 9e89738 2026-04-07 05:39:05.160677444)

Debug Output

Here is the exact timeline where the ingestion freezes. The stream successfully finishes a burst at 06:19:54, followed by ~2.5 minutes of silence. The idle connection is closed at 06:22:36. After this, nothing happens anymore even if new events are generated :

{"frame":"Data { stream_id: StreamId(3539) }","host":"WINDOWS-SERVER","message":"send","metadata":{"kind":"event","level":"DEBUG","module_path":"h2::codec::framed_write","target":"h2::codec::framed_write"},"timestamp":"2026-04-12T06:19:54.013218200Z"}
{"frame":"Data { stream_id: StreamId(3539), flags: (0x1: END_STREAM) }","host":"WINDOWS-SERVER","message":"send","metadata":{"kind":"event","level":"DEBUG","module_path":"h2::codec::framed_write","target":"h2::codec::framed_write"},"timestamp":"2026-04-12T06:19:54.013295Z"}
{"frame":"Headers { stream_id: StreamId(3539), flags: (0x4: END_HEADERS) }","host":"WINDOWS-SERVER","message":"received","metadata":{"kind":"event","level":"DEBUG","module_path":"h2::codec::framed_read","target":"h2::codec::framed_read"},"timestamp":"2026-04-12T06:19:54.054499400Z"}
{"frame":"Data { stream_id: StreamId(3539) }","host":"WINDOWS-SERVER","message":"received","metadata":{"kind":"event","level":"DEBUG","module_path":"h2::codec::framed_read","target":"h2::codec::framed_read"},"timestamp":"2026-04-12T06:19:54.054556600Z"}
{"frame":"Headers { stream_id: StreamId(3539), flags: (0x5: END_HEADERS | END_STREAM) }","host":"WINDOWS-SERVER","message":"received","metadata":{"kind":"event","level":"DEBUG","module_path":"h2::codec::framed_read","target":"h2::codec::framed_read"},"timestamp":"2026-04-12T06:19:54.054569500Z"}
{"frame":"GoAway { error_code: NO_ERROR, last_stream_id: StreamId(0) }","host":"WINDOWS-SERVER","message":"send","metadata":{"kind":"event","level":"DEBUG","module_path":"h2::codec::framed_write","target":"h2::codec::framed_write"},"timestamp":"2026-04-12T06:22:36.623332800Z"}
{"error":"GoAway(b\"\", NO_ERROR, Library)","host":"WINDOWS-SERVER","message":"Connection::poll; connection error","metadata":{"kind":"event","level":"DEBUG","module_path":"h2::proto::connection","target":"h2::proto::connection"},"timestamp":"2026-04-12T06:22:36.624688400Z"}

Example Data

No response

Additional Context

• Running on Windows Server 2025

• I have not yet attempted to reproduce this with a local console or file sink.

• However, the disk buffer is definitely not full when the freeze occurs. I know this because the moment I manually restart the Vector service (which is still running - no issue here), it instantly wakes up and aggressively bursts all the accumulated backlogged events to the aggregator without any issues. I guess this strongly isolates the bug to the source loop going to sleep and never waking up, rather than a permanent downstream bottleneck.

References

#24305 @tot19

[tot19]

Thanks for filing this. Will take a look and get back to you

[tot19]

If I put up a fix on a new branch, would you be able to test it to see if it resolves the bug? @SwimSalt

Thank you for all of the great info you provided here.

[SwimSalt]

Yes, absolutely! I'd be happy to test it. @tot19

[tot19]

Hi @SwimSalt — thanks for the detailed report and especially for the correct root-cause analysis. Your diagnosis is exactly right: it's a classic lost-wakeup race between Windows' SetEvent notification and our ResetEvent call.

I've opened #25195 with a fix. The short version:

• The Windows Event Log service signals our pull-mode wait handle via SetEvent each time a new matching event is recorded. Because the handle is manual-reset, SetEvent on an already-signaled handle is a no-op.
• Today, pull_events drains EvtNext to ERROR_NO_MORE_ITEMS and only then calls ResetEvent. If a new event arrives between the last EvtNext and the ResetEvent, the OS's SetEvent is absorbed (no-op) and then our ResetEvent clears the signal — the notification is silently lost and the next WaitForMultipleObjects times out indefinitely.
• The fix moves ResetEvent to run before the drain. Any SetEvent raised by the OS during the drain window is preserved, so the next WaitForMultipleObjects returns immediately. On paths that break out of the drain early (channel budget exhausted, bookmark update failure), we re-SetEvent so the next iteration still revisits that channel without waiting for a fresh OS signal.

This is the same ordering that the well-known correct Win32 pattern uses (reset-before-drain, not reset-after-drain). Microsoft's canonical pull-subscription sample actually has the same bug and should arguably be updated too.

The correlation you noticed with the h2 GoAway timeout is coincidental — the race fires after any brief quiescent period; the GoAway just happens to mark the moment external traffic slowed enough for the race window to open.

If you have a chance, could you build from that branch (tot19/fix-windows-event-log-lost-wakeup-25194 on my fork, or once the PR is merged) and confirm the source no longer freezes under your reproduction? A nightly build from master after merge would work too.

Also added a regression test (test_signal_not_lost_when_set_during_drain) that pins the invariant deterministically using std::sync::Barrier, so this can't silently regress.

[tot19]

Ok, ignore the Claude verbosity lol ... but should be ready for you now. Thanks again! @SwimSalt.

CC @pront

[tot19]

Running integration here - https://github.com/tot19/vector/actions/runs/24443619236
And unit here - https://github.com/tot19/vector/actions/runs/24443749151

[SwimSalt]

@tot19 Thanks for the fast fix!

I'll grab the Windows msi installer artifact from your CI runs to test it out as soon as it's available, since I'm not set up to compile from source locally at the moment. I'll report back here once I've run it against my environment.

Also, I just wanted to say thank you for taking the time to build and implement this windows_event_log feature !