[Bug] Worker Heartbeat: wf_task_latency doesn't always match 1:1 with total_processed_tasks

[yuandrew]

What are you really trying to do?

Worker Heartbeat uses the cording of wf_task_latency to count for total_processed_tasks for the heartbeat metric.

Describe the bug

Maybe not a "bug" but more of a "feature", but in Go, we always record processed, and later in the function conditionally set failure metrics. Java does the same.

But in Core, the two metrics are recorded in different places on different code paths. There is a gap when failed_completion can produce WFTFailedDontReport as its outcome, which maps to WFTReportStatus::DropWft. DropWft is not Reported, so mark_wft_complete skips the latency recording. Failed gets bumped, but processed does not.

We could either separate out the task processing count for core and not piggy back off of the latency metric here, or we could make a breaking change to always record the latency, which would align core with Go.

Additional context

Claude says this can happen Repeated WFT failures (attempt > 1)

let should_report = match &evict_req_outcome {
                EvictionRequestResult::EvictionRequested(Some(attempt), _)
                | EvictionRequestResult::EvictionAlreadyRequested(Some(attempt)) => *attempt <= 1,
                _ => false,
            };

should_report=true → ReportWFTFail → handle_activation_failed → server call → WFTReportStatus::Reported → mark_wft_complete → wf_task_latency()

That, and transient failures on a legacy query