fix: Proper network wakers #2086

Gelbpunkt · 2025-11-27T11:03:19Z

This refactors our timer interrupt handling into a separate module that all timer interrupts are configured through and handled by. It allows us to track why a timer interrupt was fired.

We make use of this to now only conditionally wake the network task's waker when necessary. This is either because we sent some network packets, a network device driver received an interrupt, or because a timer interrupt that we configured so we can poll smoltcp in the future was handled.

Closes #1933.
Closes #2126.

github-actions

Benchmark Results

Details

Benchmark	Current: `b4d7d6c`	Previous: `3dff742`	Performance Ratio
startup_benchmark Build Time	`96.11` s	`97.44` s	`0.99` ❗
startup_benchmark File Size	`0.87` MB	`0.86` MB	`1.00` ❗
Startup Time - 1 core	`0.94` s (`±0.03` s)	`0.96` s (`±0.04` s)	`0.98`
Startup Time - 2 cores	`0.96` s (`±0.03` s)	`0.96` s (`±0.03` s)	`1.01`
Startup Time - 4 cores	`0.95` s (`±0.04` s)	`0.97` s (`±0.03` s)	`0.97`
multithreaded_benchmark Build Time	`98.79` s	`96.98` s	`1.02` ❗
multithreaded_benchmark File Size	`0.96` MB	`0.96` MB	`1.00` ❗
Multithreaded Pi Efficiency - 2 Threads	`87.27` % (`±7.50` %)	`87.69` % (`±8.27` %)	`1.00`
Multithreaded Pi Efficiency - 4 Threads	`44.41` % (`±3.52` %)	`43.63` % (`±2.74` %)	`1.02`
Multithreaded Pi Efficiency - 8 Threads	`25.51` % (`±1.86` %)	`25.17` % (`±2.19` %)	`1.01`
micro_benchmarks Build Time	`110.80` s	`105.61` s	`1.05` ❗
micro_benchmarks File Size	`0.97` MB	`0.97` MB	`1.00` ❗
Scheduling time - 1 thread	`69.63` ticks (`±2.80` ticks)	`68.57` ticks (`±3.51` ticks)	`1.02`
Scheduling time - 2 threads	`39.85` ticks (`±5.37` ticks)	`37.47` ticks (`±4.85` ticks)	`1.06`
Micro - Time for syscall (getpid)	`3.82` ticks (`±0.34` ticks)	`3.66` ticks (`±0.37` ticks)	`1.04`
Memcpy speed - (built_in) block size 4096	`63321.94` MByte/s (`±45130.42` MByte/s)	`63614.27` MByte/s (`±45137.75` MByte/s)	`1.00`
Memcpy speed - (built_in) block size 1048576	`30074.38` MByte/s (`±25052.92` MByte/s)	`29708.69` MByte/s (`±24538.84` MByte/s)	`1.01`
Memcpy speed - (built_in) block size 16777216	`24026.60` MByte/s (`±20196.20` MByte/s)	`28251.45` MByte/s (`±23528.23` MByte/s)	`0.85`
Memset speed - (built_in) block size 4096	`64205.46` MByte/s (`±45791.68` MByte/s)	`64610.60` MByte/s (`±45851.70` MByte/s)	`0.99`
Memset speed - (built_in) block size 1048576	`30891.69` MByte/s (`±25520.80` MByte/s)	`30504.90` MByte/s (`±25009.02` MByte/s)	`1.01`
Memset speed - (built_in) block size 16777216	`24802.87` MByte/s (`±20715.82` MByte/s)	`29036.46` MByte/s (`±23978.08` MByte/s)	`0.85`
Memcpy speed - (rust) block size 4096	`58928.23` MByte/s (`±43555.02` MByte/s)	`61996.44` MByte/s (`±45837.23` MByte/s)	`0.95`
Memcpy speed - (rust) block size 1048576	`30083.78` MByte/s (`±25080.02` MByte/s)	`29790.29` MByte/s (`±24658.87` MByte/s)	`1.01`
Memcpy speed - (rust) block size 16777216	`24583.25` MByte/s (`±20692.34` MByte/s)	`28365.42` MByte/s (`±23631.36` MByte/s)	`0.87`
Memset speed - (rust) block size 4096	`59640.76` MByte/s (`±44031.13` MByte/s)	`62841.27` MByte/s (`±46294.21` MByte/s)	`0.95`
Memset speed - (rust) block size 1048576	`30900.28` MByte/s (`±25544.24` MByte/s)	`30590.90` MByte/s (`±25118.10` MByte/s)	`1.01`
Memset speed - (rust) block size 16777216	`25336.48` MByte/s (`±21172.81` MByte/s)	`29150.63` MByte/s (`±24086.62` MByte/s)	`0.87`
alloc_benchmarks Build Time	`110.76` s	`103.51` s	`1.07` ❗
alloc_benchmarks File Size	`0.94` MB	`0.93` MB	`1.00` ❗
Allocations - Allocation success	`100.00` %	`100.00` %	`1`
Allocations - Deallocation success	`100.00` %	`100.00` %	`1`
Allocations - Pre-fail Allocations	`100.00` %	`100.00` %	`1`
Allocations - Average Allocation time	`10526.72` Ticks (`±197.49` Ticks)	`10597.42` Ticks (`±216.79` Ticks)	`0.99`
Allocations - Average Allocation time (no fail)	`10526.72` Ticks (`±197.49` Ticks)	`10597.42` Ticks (`±216.79` Ticks)	`0.99`
Allocations - Average Deallocation time	`1520.93` Ticks (`±707.10` Ticks)	`1080.08` Ticks (`±550.01` Ticks)	`1.41`
mutex_benchmark Build Time	`108.89` s	`110.09` s	`0.99` ❗
mutex_benchmark File Size	`0.97` MB	`0.97` MB	`1.00` ❗
Mutex Stress Test Average Time per Iteration - 1 Threads	`13.24` ns (`±0.68` ns)	`13.06` ns (`±0.68` ns)	`1.01`
Mutex Stress Test Average Time per Iteration - 2 Threads	`18.78` ns (`±8.02` ns)	`14.94` ns (`±1.05` ns)	`1.26`

This comment was automatically generated by workflow using github-action-benchmark.

mkroening

Thanks a lot; this is looking great! :)

src/scheduler/timer_interrupts.rs

src/arch/timer_interrupts.rs

mkroening · 2026-01-20T12:46:40Z

src/scheduler/task/mod.rs

+					create_timer_abs(Source::Scheduler, wt);
 					return;
 				}

 				cursor.move_next();
 			}

-			set_oneshot_timer();
+			create_timer_abs(Source::Scheduler, wt);


The docs say that wakeup_time is relative. This was also done wrongly before, but maybe we should follow up to fix this.

src/arch/timer_interrupts.rs

src/scheduler/timer_interrupts.rs

src/drivers/net/virtio/mod.rs

src/executor/mod.rs

src/executor/network.rs

zyuiop · 2026-02-04T11:02:03Z

Hey, great work!

I think we should evaluate moving the run() call after the first return in block_on, as such:

In a lighttpd/wrk benchmark, I get a x3 performance boost by doing this.

Before:

Running 10s test @ http://localhost:9975
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.44ms    0.98ms  24.22ms   73.61%
    Req/Sec     3.53k   693.42     4.52k    63.00%
  70345 requests in 10.00s, 21.13MB read
Requests/sec:   7034.16
Transfer/sec:      2.11MB

After:

Running 10s test @ http://localhost:9975
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   491.38us  725.83us  22.69ms   98.95%
    Req/Sec    10.76k     2.01k   13.22k    73.76%
  216241 requests in 10.10s, 64.96MB read
Requests/sec:  21410.61

Trying to explain this result:
block_on is invoked in a lot of places, basically most system calls related to IO.
However, not all operations are actually blocking: a lot of times, the polled future returns immediately.
In these cases, which are basically simple functions calls, the background tasks are run at each function call, which is probably too much.

Following discussion with @mkroening, we should keep this patch as-is and discuss this potential change in a separate PR.

mkroening

Thank you so much! I have done a few rough sanity benchmarks for bandwidth:

rev	Receive	Send
Before this PR	3.3 Gbit/s	2.6 Gbit/s
This PR	3.5 Gbit/s	2.5 Gbit/s

mkroening · 2026-02-05T11:42:58Z

As of hermit-os/hermit-rs#914, the CI should be less flaky now.

mkroening · 2026-02-05T12:14:22Z

src/scheduler/timer_interrupts.rs

+	pub fn adjust_by(&mut self, offset: u64) {
+		for timer in self.0.iter_mut() {
+			if timer.wakeup_time != u64::MAX {
+				timer.wakeup_time -= offset;


CI found this reliably overflows:

$ cargo xtask ci rs --arch x86_64 --package mioudp --features hermit/dhcpv4,hermit/virtio-net qemu --devices virtio-net-pci [..] [ 0.450189][0][WARN entropy ] Unable to read entropy! Fallback to a naive implementation! You can connect to the server using `nc`: $ nc -u 127.0.0.1 9975 Anything you type will be echoed back to you. [CI] send "exit" via UDP to 127.0.0.1:9975 [CI] receive: exit [0][PANIC] panicked at src/scheduler/timer_interrupts.rs:75:17: attempt to subtract with overflow [ 5.713705][0][INFO interrupts] Number of interrupts [ 5.715990][0][INFO interrupts] [0][Reschedule]: 3 [ 5.716729][0][INFO interrupts] [0][Timer]: 2 [ 5.717296][0][INFO interrupts] [0][virtio]: 3 exit status 1 [ 5.718251][0][INFO processor ] Shutting down system

The same happens on AArch64. I can reproduce this consistently locally as well.

Good thing we actually test sleeping in our CI now (hermit-os/hermit-rs#914). :D

I cannot reproduce this locally, no matter whether I target x86_64 or aarch64... Quite weird

Anyways, I think saturating_sub is acceptable to use here, let's see whether it passes CI

mkroening · 2026-02-09T18:23:41Z

Note that this PR makes #1680 worse, since not even keyboard inputs work around the problem. That's okay though, since #2231 resolves the issue. After that PR is merged, this PR should get rebased to avoid having commits that are more broken than necessary.

This refactors our timer interrupt handling into a separate module that all timer interrupts are configured through and handled by. It allows us to track why a timer interrupt was fired. We make use of this to now only conditionally wake the network task's waker when necessary. This is either because we sent some network packets, a network device driver received an interrupt, or because a timer interrupt that we configured so we can poll smoltcp in the future was handled.

Gelbpunkt force-pushed the wakers branch from edba06e to c8d52f7 Compare November 27, 2025 11:03

mkroening self-assigned this Nov 27, 2025

Gelbpunkt force-pushed the wakers branch 2 times, most recently from 9c652d1 to c03dc8e Compare November 27, 2025 11:27

github-actions bot reviewed Nov 27, 2025

View reviewed changes

Gelbpunkt force-pushed the wakers branch from c03dc8e to d3a3407 Compare November 27, 2025 11:59

Gelbpunkt force-pushed the wakers branch from d3a3407 to 6108e2f Compare December 13, 2025 20:59

Gelbpunkt force-pushed the wakers branch 8 times, most recently from 390c379 to 466882e Compare December 22, 2025 09:45

mkroening self-requested a review December 22, 2025 10:12

Gelbpunkt marked this pull request as ready for review December 22, 2025 10:13

Gelbpunkt force-pushed the wakers branch 3 times, most recently from f48f443 to 839554d Compare January 8, 2026 10:51

Gelbpunkt mentioned this pull request Jan 8, 2026

Task wakeup timer is overwritten #2126

Open

mkroening requested changes Jan 20, 2026

View reviewed changes

Gelbpunkt force-pushed the wakers branch 2 times, most recently from e1f9653 to d5f9bef Compare January 22, 2026 15:26

Gelbpunkt force-pushed the wakers branch from d5f9bef to 7a4cd75 Compare January 30, 2026 01:46

mkroening requested a review from stlankes February 2, 2026 09:59

stlankes approved these changes Feb 2, 2026

View reviewed changes

mkroening mentioned this pull request Feb 4, 2026

MSI-X support #2101

Open

mkroening force-pushed the wakers branch from 7a4cd75 to adbb324 Compare February 4, 2026 12:38

mkroening approved these changes Feb 4, 2026

View reviewed changes

mkroening force-pushed the wakers branch from 3e9bd14 to 0fd6c6d Compare February 5, 2026 11:41

mkroening requested changes Feb 5, 2026

View reviewed changes

mkroening mentioned this pull request Feb 9, 2026

fix(virtio-fs): de-assert interrupts #2231

Merged

Gelbpunkt force-pushed the wakers branch from 0fd6c6d to b4d7d6c Compare February 10, 2026 21:34

fix: Proper network wakers #2086

Are you sure you want to change the base?

fix: Proper network wakers #2086

Uh oh!

Conversation

Gelbpunkt commented Nov 27, 2025 • edited by mkroening Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Benchmark Results

Uh oh!

mkroening left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mkroening Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zyuiop commented Feb 4, 2026

Uh oh!

mkroening left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkroening commented Feb 5, 2026

Uh oh!

mkroening Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Gelbpunkt Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Gelbpunkt Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

mkroening commented Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Gelbpunkt commented Nov 27, 2025 •

edited by mkroening

Loading

github-actions bot left a comment •

edited

Loading

mkroening left a comment •

edited

Loading