Skip to content

The "wrong" worker wakes up when other workers have designated work. #793

Open
@wks

Description

@wks

After PR #782, only the coordinator can open new buckets.

However, workers may park and then receive designated work packets. In this case, the worker will not wake up. When all workers parked, the coordinator will look if any workers have any designated work. If some workers do, the coordinator will reset the group_sleep state and notifies all workers, hoping workers will wake up and execute their designated work.

However, the first unparked worker can be a "wrong" worker that doesn't have any designated work. It will not find any designated work or any work in any buckets. Then it will set group_sleep = true. That will prevent the "right" workers that actually have designated work from unparking. Then the coordinator will notify all workers again because some still have designated work. This can happen again and again, as shown in the following log.

[2023-04-15T08:22:46Z INFO  mmtk::util::heap::gc_trigger] [POLL] nursery: Triggering collection (128007/128000 pages)
[2023-04-15T08:22:46Z INFO  mmtk::plan::generational::global] Nursery GC
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:46Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Bucket opened
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Bucket opened
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Bucket opened
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Bucket opened
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   Some workers have designated work
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller] Controller received an event! AllParked
[2023-04-15T08:22:47Z WARN  mmtk::scheduler::controller]   ... GC finished.
[2023-04-15T08:22:47Z INFO  mmtk::scheduler::gc_work] End of GC (114450/128000 pages, took 1104 ms)

It will repeat until the "right" worker wakes up first, but the time for this to happen is unbounded, and the problem will get more serious when there are many GC worker threads.

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: BugP-normalPriority: Normal.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions