Skip to content

fix: reduce tokio runtime polling overhead - #530

Merged
gnolong merged 11 commits into
mainfrom
fix/reduce-tokio-polling-overhead
Jul 15, 2026
Merged

fix: reduce tokio runtime polling overhead#530
gnolong merged 11 commits into
mainfrom
fix/reduce-tokio-polling-overhead

Conversation

@gnolong

@gnolong gnolong commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fixes [Improvement] Reducing future polling overhead in the tokio runtime #528 by replacing high-frequency Tokio polling with notification-based waiting and cancellation-driven monitor shutdown.
  • Rename the tokio-console feature to tracing, initialize tracing from TaskRunner::start_task, and write plain or JSON summaries to a dedicated runtime trace log.
  • Add traced wrapper to trace .await future and task future.
  • trace log in plain mode is like:
2026-07-14T03:41:41.547090Z | ape-dts Tokio task runtime summary
attributed waker calls: total=410
  count=314 percent_of_all_attributed_calls=76.59% wait_point=yield_now.extractor.wait_task_finish@dt-connector/src/extractor/base_extractor.rs:339
  count=61 percent_of_all_attributed_calls=14.88% wait_point=yield_now.pipeline.run@dt-pipeline/src/base_pipeline.rs:193
  count=29 percent_of_all_attributed_calls=7.07% wait_point=dtqueue.not_full.wait@/Users/wl/etc/clone_1/ape-dts/dt-common/src/meta/dt_queue.rs:92
  count=6 percent_of_all_attributed_calls=1.46% wait_point=dtqueue.not_empty.wait@/Users/wl/etc/clone_1/ape-dts/dt-common/src/meta/dt_queue.rs:132
traced tokio task markers: total=4
  marker=task.extractor_worker@/Users/wl/etc/clone_1/ape-dts/dt-task/src/task_runner.rs:675 task_count=1 poll_count=1084 scheduled_count=1083 busy_ms=144.096 attributed_waker_calls=322
    count=314 percent_of_marker_attributed_calls=97.52% wait_point=yield_now.extractor.wait_task_finish@dt-connector/src/extractor/base_extractor.rs:339
    count=8 percent_of_marker_attributed_calls=2.48% wait_point=dtqueue.not_full.wait@/Users/wl/etc/clone_1/ape-dts/dt-common/src/meta/dt_queue.rs:92
  marker=pg snapshot worker@dt-connector/src/extractor/snapshot_dispatcher.rs:85 task_count=42 poll_count=240 scheduled_count=198 busy_ms=156.181 attributed_waker_calls=21
    count=21 percent_of_marker_attributed_calls=100.00% wait_point=dtqueue.not_full.wait@/Users/wl/etc/clone_1/ape-dts/dt-common/src/meta/dt_queue.rs:92
  marker=task.pipeline_worker@/Users/wl/etc/clone_1/ape-dts/dt-task/src/task_runner.rs:686 task_count=1 poll_count=139 scheduled_count=138 busy_ms=45.392 attributed_waker_calls=67
    count=61 percent_of_marker_attributed_calls=91.04% wait_point=yield_now.pipeline.run@dt-pipeline/src/base_pipeline.rs:193
    count=6 percent_of_marker_attributed_calls=8.96% wait_point=dtqueue.not_empty.wait@/Users/wl/etc/clone_1/ape-dts/dt-common/src/meta/dt_queue.rs:132
  marker=pg snapshot worker@dt-connector/src/extractor/snapshot_dispatcher.rs:68 task_count=1 poll_count=4 scheduled_count=3 busy_ms=3.194 attributed_waker_calls=0

@gnolong
gnolong marked this pull request as ready for review July 14, 2026 05:10
@gnolong
gnolong merged commit 9dc2474 into main Jul 15, 2026
163 of 233 checks passed
@gnolong
gnolong deleted the fix/reduce-tokio-polling-overhead branch July 15, 2026 06:21

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

比较优雅点的应该是#[cfg(feature = "tracing")]直接控制这个类,引用的地方通过not feature来控制要不要with_trace或者dump。现在所有方法都在runtime_trace里实现了一遍not feature和feature的比较臃肿

self.remove_empty_check_logs().await?;
self.upload_check_logs_to_s3().await?;
log_finished!("task finished");
if let Some(summary) = dt_common::runtime_trace::dump_global_summary() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trace功能在finish才log就不适合cdc了,更好的是把trace指标和task_monitor结合起来,可以周期性输出,如果开启了metrics feature还可以通过prometheus style http api fetch

caiq1nyu added a commit that referenced this pull request Jul 15, 2026
* fix: reduce tokio runtime polling overhead (#530)

* fix: reduce tokio runtime polling overhead

* fmt

* fix: remove unused tokio-util deps

* fix: remove unused queue empty wait

* fix: add tokio wake source summary

* fix: rename tracing feature

* fix: add runtime tracing task summary

* fix: collect tokio task runtime metrics

* fix

* chore: code format (#539)

---------

Co-authored-by: wanglong <46259301+gnolong@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Reducing future polling overhead in the tokio runtime

2 participants