Spurious dips/zeros across multiple monotonic windows_process_* counters (Alloy v1.10.1 / windows_exporter v0.30.8)

[rkyser]

Problem Statement

Environment

• Agent: Alloy v1.10.1
• Embedded exporter: windows_exporter v0.30.8
• OS: Windows Server 2022 on Azure Virtual Machine Scale Set (target host)
• Alloy+windows_exporter for scraping, Grafana for visualization

Problem

We’re seeing intermittent, single-scrape drops to 0 or a small fixed value (e.g. 3396) in metrics that should be monotonic counters. Immediately before/after, the series resumes at the great-or-equal expected value.

This occurs across multiple per-process counters, for example:

• windows_process_page_faults_total (see screenshot below)
• windows_process_cpu_time_total

What we know

• The PID label remains constant across the dips — so it is not a process restart.
• The dips are one scrape wide (… → high → 0/3396 → high …).
• We also see scrape gaps (e.g. 10:08–10:15) but those are distinct from the spurious values. (honestly, less concerned about gaps at this point)
• Multiple different windows_process_* metrics are affected, not just one.

Questions

1. Is it expected for the exporter to emit bad samples? Or am I somehow misinterpreting the data?
2. Are there additional steps that I could take to troubleshoot this issue? What might you suggest?

Screenshot with Dips

Environment

• windows_exporter Version: v0.30.8 (based on inspecting Alloy Agent v1.10.1 go.sum file)
• Windows Server Version: 2022

[jkroepke]

windows_exporter 0.30 uses Process V2 performance data. Any data from performance collectors are exposed without interpret them. It could be possible that they deliver wrong data. Generally, Windows itself collect the data, windows_exporter just expose them.

The new Process V2 Performance Data still had some issues. Thats one reason, why I implement a toggle to use the old V1 collector in server 2022 or higher.

#2064

---

Another potential issue is #2098 which was fixed and backported to 0.30.9.

[rkyser]

If I am understanding this, you think the bad data is coming from the Windows Process V2 API?

We are trying the Alloy agent to scrape from the latest windows_exporter to see if that makes a difference.

[jkroepke]

We are trying the Alloy agent to scrape from the latest windows_exporter to see if that makes a difference.

i guess with --collector.process.counter-version=1 you see a difference. In that specific case, windows_exporter directly fetches and decodes the data from the registry.

[rkyser]

I was reviewing the PDH docs for PdhGetRawCounterArrayW return values and noticed the following statement.

PDH_MORE_DATA
The ItemBuffer buffer is not large enough to contain the object name. This return value is expected if lpdwBufferSize is zero on input. If the specified size on input is greater than zero but less than the required size, you should not rely on the returned size to reallocate the buffer.

It seems like the recommendation is to:

1. call PdhGetRawCounterArrayW with lpdwBufferSize = 0
2. get the lpdwBufferSize size back from the API
3. allocate new buffer based on lpdwBufferSize
4. call PdhGetRawCounterArrayW with the updated lpdwBufferSize and allocated buffer

---

Conversely, the current implementation of windows_exporter (master / 691f64f5cccaa3966c66ee8f9036470c1c5094a1):

1. initializes buf with byte array of length 1. (buf := make([]byte, 1))
2. calls GetFormattedCounterArrayDouble with bytesNeeded set to uint32(cap(buf)), which that GetFormattedCounterArrayDouble would the passed a non-zero value the first time for lpdwBufferSize.
3. reallocates buf based on the resulting bytesNeeded

Unless I am reading this wrong, this seems to be doing exactly what the API docs says should not be done: relying on the returned size to reallocate the buffer, when the input size is greater than zero but less than the required side.

There seems to be an example in internal/pdh/pdh.go which shows the first call to GetFormattedCounterArrayDouble is passed a default bufSize of size zero (0). This seems to be consistent with the linked API documentation.

---

Do you think this could be contributing to the problem?

[rkyser]

We'll try --collector.process.counter-version=1 next if the latest exporter doesn't resolve the issue.

[jkroepke]

Do you think this could be contributing to the problem?

Possible is everything. I can do a snapshot build which fix that issue you mentioned and see, if the issue has been solved.

#2183

[rkyser]

@jkroepke thank you for your help--much appreciated!

While we wait for the snapshot build, we will try the flag you recommended and see what happens.

[jkroepke]

Snapshot Build: https://github.com/prometheus-community/windows_exporter/actions/runs/17047537731/artifacts/3790964665

[jkroepke]

much appreciated!

Dont worry. I'm also interested that this exporter works. Thanks for testing this!

[rkyser]

Exporter deployed. I will report back in the morning.

[rkyser]

It looks like the snapshot might be an improvement.

With the following prom query, I am seeing zero negative deltas for the past 12 hours. Applying the same query to data from two days ago, I see over 300 series of data with negative deltas.

delta(
    windows_process_cpu_time_total{instance="server_name", process!="Idle"}
    [$__rate_interval]
) < 0

@jkroepke Can you think of any other queries I might try? Or other tests?

[jkroepke]

Did you just use the snapshot or did you also set collector.process.counter-version=1 ?