Special topic chapter for finalizers and weak references #1265

wks · 2025-01-16T12:05:15Z

This PR adds a special topic chapter in the Porting Guide for supporting finalizers and weak references. This topic is frequently asked and somewhat complex, and needs a dedicated chapter.

We also updated the doc comments of the Scanning::process_weak_refs API to add code example of the intended use case, and warn the users about potential pitfalls.

Added a special topic chapter for how to implement finalizers and weak references with MMTk.

wks · 2025-01-23T07:49:49Z

I have finished the chapter, and I invited @qinsoon and @k-sareen to review it. There are some things I haven't done in this PR, but can be added later.

Ephemerons: We currently have not implemented ephemerons in any VMs. We need Ephemeron for V8, but that's not a priority at this moment. We can add a subsection for ephemerons when we implemented it, or when someone needs help implementing ephemerons with MMTk. (Update: I added a section for ephemerons. But since we don't have a VM that actively uses that algorithm, we can't verify if that algorithm is correct.)
Hyperlinks between the doc comments in the code and the Porting Guide chapter. The porting guide needs to be rendered in order to be published to https://docs.mmtk.io/, and that'll happen after this PR is merged. We can add the URL when the new chapter becomes publicly available.

Maybe I also need to define some concepts in that chapter, such as what exactly is a finalizer or a weak reference. I think the current state is clear enough for people with experience with GC and VM development. But please let met know if anything is not clear.

wks · 2025-01-23T08:43:23Z

It is still arguable whether "retain" or "resurrect" is a better word for keeping an unreachable object alive in a GC. "Resurrect" is more intuitive because we also talk about "finalizers can resurrect dead objects". But the word "resurrect" may imply that the object has already died once. It is OK for finalizers because the object has already become unreachable during mutator time, which can be considered as "dead". But for soft references, it is a bit awkward to say we "resurrect" the referent of a soft reference because that implies we consider the referent as "dead" before. It is "dead" in the sense that it is not strongly reachable, but softly reachable objects usually live through GCs, as long as it is not emergency GC. Maybe "retain" is better when describing soft references.

In existing code, we use the word "retain" in many places in ReferenceProcessor. There is only one use of "resurrect" in FinalizerProcessor, and it was added by me.

qinsoon

The doc is written with dfiferent definitions of 'weak ref', and 'resurrect objects'. I think we should refer to a well-accepted literature such as the GC handbook for such definitions, and be consistent with their definitions. In my opinion, we need a pass through the section to fix the definitions and related descriptions. I would like to do a detailed review after that.

qinsoon · 2025-01-23T09:12:13Z

docs/userguide/src/SUMMARY.md

@@ -36,6 +36,8 @@
    - [Performance Tuning](portingguide/perf_tuning/prefix.md)
        - [Link Time Optimization](portingguide/perf_tuning/lto.md)
        - [Optimizing Allocation](portingguide/perf_tuning/alloc.md)
+    - [Special Topics](portingguide/topics/prefix.md)


I suggest using something like "Runtime Features", or "Language Features" as the title. I don't see how this section is 'special' compared to other sections.

What about "VM-Specific Concerns"?

The GC Handbook has a chapter named Language-Specific Concerns, and it discusses only finalizers and weak references. I think that's because finalizer and weak reference semantics is part of the programming language, and is visible to the programmers. Things like conservative stack scanning, object pinning and interior pointers are not all exposed at the language level, but they are more related to VM implementations. So "VM-Specific Concerns" may be a better title.

I don't like calling them "Runtime/Language Features" because "features" are things meant to be used by their users, but things like stack scanning are peculiar aspects of those runtimes/languages that their implementers should care about.

And I think "VM peculiarities" also describes what the chapters in this part are about. But it sounds offensive to the VMs.

I renamed it to "VM-specific concerns".

qinsoon · 2025-01-24T00:05:01Z

docs/userguide/src/portingguide/topics/prefix.md

@@ -0,0 +1,5 @@
+# Special topics
+
+Every VM is special in some way.  Because of this, some VM bindings may use MMTk features not


If we change the title, the paragraph needs to be changed accordingly. I feel it is more reasonable that this section is dedicated to how to implement different runtime features with MMTk.

Yes. I changed this part, too.

qinsoon · 2025-01-24T00:10:09Z

docs/userguide/src/portingguide/topics/weakref.md

+Some VMs support **weak references**.  If an object cannot be reached from roots following only
+strong references, the object will be considered dead.  Weak references to dead objects will be
+cleared, and associated clean-up operations will be executed.  Some VMs also support more complex


The definition for weak references is incorrect.

If an object cannot be reached from roots following only strong references, the object will be considered dead.

If an object cannot be reached from strong references, it is not "strongly reachable". It is not dead, as it can still be "weakly reachable". Sometimes "reachable" implies strongly reachable. That's fine. But this should not be used in a context that may cause confusion.

Weak references to dead objects will be cleared, and associated clean-up operations will be executed.

This is indeed confusing. If we have a weak reference to the object, the object is not dead. It is up to the language semantics or the VM to decide whether to keep the weakly-referenced object alive or not. Only when the VM decides that the referent will not be kept alive, the object/referent is dead, otherwise it is alive.

I will rewrite this part using definitions from the GC handbook, and probably add a "Definitions" section before the "Overview" section.

I didn't use the terms "strongly/weakly reachable" because I thought they were Java-specific. But since the GC Handbook also uses that term in the language-neutral introduction section of weak references, I believe those terms should be acceptable for other VMs in general, too.

I added a "Definitions" section before the "Overview" section, and I used the term "strongly reachable" and "weakly reachable". I also added a Glossary chapter at the top level of the User Guide and linked to it from this chapter. Currently I only added the description of "object graph" in the Glossary.

qinsoon · 2025-01-24T00:18:35Z

docs/userguide/src/portingguide/topics/weakref.md

+-   **Query forwarded address**: If an object is already reached, the VM binding can further query
+    the new address of an object.  This is needed to support copying GC.
+    +   Do this with `ObjectReference::get_forwarded_object()`.
+-   **Resurrect objects**: If an object is not reached, the VM binding can optionally resurrect the


See my argument above for the definition of weak ref. We do not 'resurect' weak reference.

It is arguable whether we should say 'resurrect' finalizable objects. GC needs to keep finalizable objects alive until they are properly finalized -- I don't want to call it 'resurrect', simply it is not dead from the GC's perspective. Also 'resurrecting objects' normally refers to the specific application behavior that an object that is being finalized and should become dead after finalization is at the language/application level kept alive. Using 'resurrect' in the GC may get things confusing.

GC handbook's definition for 'resurrection': "an action performed by a finaliser that caused the previously unreachable object to become reachable."

I consulted @eliotmoss and read the GC handbook, and they both agree with what you said. "Resurrection" refers to the action in the finalizer that makes the object reachable from other parts of the program by, for example, assigning its reference to a global variable. This is not what tracer.trace_object does. Even if we call trace_object(object), it still doesn't guarantee the object will "resurrect" in that sense. If the finalize() function doesn't leak its reference, the object will still be collectable after finalize() returns.

"Retain" should be a better term, and it is already used in JikesRVM as well as our existing ReferenceProcessor in mmtk-core.

I now use "retain" consistently when describing what tracer.trace_object does. I still mentioned "resurrect" briefly when describing the counterintuitive behavior that finalizers make an unreachable object reachable from user threads again.

qinsoon · 2025-01-24T00:23:58Z

docs/userguide/src/portingguide/topics/weakref.md

+descendants) live through the current GC.  The typical use pattern is:
+
+```rust
+impl<VM: VMBinding> Scanning<VM> for VMScanning {


It is generally preferrable to put the code into a test, and include the snippets in the test in the doc. In such cases, the code will be always checked by the tests and the CI. Maybe we cannot run process_weak_refs in our mock tests, but we can still make sure that it can be compiled.

See here for examples:

mmtk-core/docs/userguide/src/portingguide/perf_tuning/alloc.md

Line 36 in 051bc74

{{#include ../../../../../src/vm/tests/mock_tests/mock_test_doc_mutator_storage.rs:mutator_storage_boxed_pointer}}

I now use a mock test to contain the example code. This can ensure it compiles. But we still can't actually run it in mock tests because we can't construct a GCWorker object.

k-sareen · 2025-01-24T00:42:13Z

I would avoid inventing terminology and just use what is in the GC Handbook. The handbook uses resurrect for finalizers and "trace" for soft references. Pg 213 onwards for the First Edition of the GC Handbook

... unless "resurrect" refers to the application-visible phenomenon.

- Slightly rephrased the beginning of the overview section - Always use perfect tense when talking about "reached" and "not reached".

This makes sure the code compiles, and makes sure we notice it if the API of the `process_weak_refs` API is changed.

wks · 2025-02-18T09:31:04Z

The tests on 1.71.1 (our MSRV) failed because the latest version of the "built" crate (v0.7.7) requires Rust 1.74 or later.

k-sareen · 2025-02-18T09:43:11Z

If we want to update our MSRV or pin the crate version it should be a separate PR. Currently my ART port is stuck at Rust v1.74 for reasons and I can't easily update it. So I'd personally like if our MSRV doesn't get bumped too high, but I can always just maintain it in my own branch for backwards compatibility.

wks · 2025-02-18T14:17:16Z

I have made changes since this PR was reviewed the last time. Important changes include:

I renamed "Special Topics" to "VM-specific Concerns"
I now use the word "retain" to refer to the thing trace_object does. Use "resurrect" only to describe the application-visible phenomenon, i.e. an unreachable object becoming reachable again.
I added a "Definitions" section and define finalizers, weak references and strongly/weakly reachable in the way the GC Handbook defines them.
I also added a "Glossary" chapter at the top level and added definition for the "object graph" there. It should make the definition of "strongly/weakly reachable" clearer in the added Finalizer and Weak Reference chapter.
I clarified the behavior of ObjectReference::is_reachable, and emphasize that it may conservatively return true for objects that have not been reached, particularly mature objects during nursery GC.
I now use a mock test case to hold the example code to ensure it is free of compilation error.

Currently there is an error when testing with Rust version 1.71.1 (our current MSRV). I have made another PR for bumping the MSRV to 1.74.1.

k-sareen

LGTM. Thanks! The only main concern I have is to use more general language, i.e. "runtime"/"language runtime" over "VM".

docs/userguide/src/SUMMARY.md

docs/userguide/src/portingguide/concerns/weakref.md

k-sareen · 2025-02-20T03:43:40Z

docs/userguide/src/portingguide/concerns/weakref.md

+            -   clear the referent field,
+            -   remove the `SoftReference` from the list of soft references, and
+            -   optionally enqueue it to the associated `ReferenceQueue` if it has one.
+    -   (This step may expand the transitive closure in emergency GC if any referents are retained.)


Suggested change

- (This step may expand the transitive closure in emergency GC if any referents are retained.)

- (This step may expand the transitive closure in an emergency GC if any referents are retained.)

docs/userguide/src/portingguide/concerns/weakref.md

src/util/address.rs

src/vm/scanning.rs

wks · 2025-02-20T07:27:17Z

LGTM. Thanks! The only main concern I have is to use more general language, i.e. "runtime"/"language runtime" over "VM".

@k-sareen I have made changes according to your comments.

k-sareen

LGTM. Thanks!

commit e787b2d Merge: d56c3b9 0b3ec9b Author: Yi Lin <qinsoon@gmail.com> Date: Tue Mar 25 03:02:42 2025 +0000 Merge branch 'master' into check-fragmentation-immixspace commit 0b3ec9b Author: Kunal Sareen <kunal.sareen@anu.edu.au> Date: Mon Mar 24 17:11:27 2025 +1100 Make work packet buffer size configurable from one location (mmtk#1285) Closes mmtk#1281 commit 129362d Author: Kunshan Wang <wks1986@gmail.com> Date: Thu Mar 20 17:16:35 2025 +0800 Fix lychee command (mmtk#1286) Removed `--base`. All relative URLs in Markdown are relative to the file itself. Increased verbosity to print the checked URLs and redirections. Excluded `dl.acm.org`. It always responses 403 when using Lychee to check links. --------- Co-authored-by: Yi Lin <qinsoon@gmail.com> commit c4f5a02 Author: Yi Lin <qinsoon@gmail.com> Date: Thu Mar 20 16:58:33 2025 +1300 Fix to bytemuck_derive 1.8.1 (mmtk#1288) `bytemuck` uses `bytemuck_derive ^1.4.1`. Recent `bytemuck_derive` versions are not compatible with our MSRV 1.74.1 (1.9.1 requires Rust 1.84+, 1.9.0 requires edition 2024 which is stablized in 1.84). So fix `bytemuck_derive` to the last version before 1.9.0. commit 1a78557 Author: Kunshan Wang <wks1986@gmail.com> Date: Mon Mar 17 17:37:41 2025 +0800 Fixing MSRV-breaking dependencies (mmtk#1284) Some dependencies started to require MSRV above our current MSRV. The build dependency `built` transitively depends on some crates from the ICU4X project that recently started to depend on Rust 1.81, such as `litemap`. The dependency on ICU4X is completely unnecessary, and we removed it from our dependency tree by forcing the use of a particular version of `idna_adapter`. See: https://docs.rs/crate/idna_adapter/1.2.0 The dev dependency `criterion` depends on `ciborium` which depends on the `half` crate. Since v2.5.0, `half` started to depend on Rust 1.81. `ciborium` needs to be fixed because its MSRV is 1.58 and shouldn't depend on a crate that requires Rust 1.81. We lock the version of the `half` to 2.4.1. This kind of problem should be properly addressed with the new MSRV-aware dependency resolver introduced in Rust 1.84. See: https://doc.rust-lang.org/cargo/reference/resolver.html#rust-version commit 8dded8f Author: Kunshan Wang <wks1986@gmail.com> Date: Wed Feb 26 14:51:51 2025 +0800 Fix clippy warning about operator precedence (mmtk#1280) In Rust 1.85, Clippy started to warn about the precedence of `<<` and `|` in one of our use cases, although that lint was added before Rust 1.29. https://rust-lang.github.io/rust-clippy/master/index.html#precedence commit 3832b0d Author: Kunshan Wang <wks1986@gmail.com> Date: Wed Feb 26 13:38:49 2025 +0800 Remove dead trace_object methods. (mmtk#1277) Now that we are using the `#[derive(HasSpaces, PlanTraceObject)]` derive macros to generate the trace_object methods, we can remove the manually-written dead code. commit 84545cc Author: Kunshan Wang <wks1986@gmail.com> Date: Thu Feb 20 16:19:48 2025 +0800 Special topic chapter for finalizers and weak references (mmtk#1265) This PR adds a special topic chapter in the Porting Guide for supporting finalizers and weak references. This topic is frequently asked and somewhat complex, and needs a dedicated chapter. We also updated the doc comments of the `Scanning::process_weak_refs` API to add code example of the intended use case, and warn the users about potential pitfalls. commit df5e0cd Author: Kunshan Wang <wks1986@gmail.com> Date: Thu Feb 20 11:27:05 2025 +0800 Bump MSRV to 1.74.1 (mmtk#1276) The current latest version of the "built" crate (v0.7.7) requires MSRV 1.74. We bump the MSRV to 1.74.1. Since version 0.7.6 of the "built" crate, it generates `static` items instead of `const` items for `PKG_VERSION`, `FEATURES_STR`, etc. Our `build_info.rs` used to define `const` items that take their values. After this change, the Rust compiler now interpret those lines as taking references of `static` items, which is unstable until Rust 1.83. We instead replaced those `const` items in `build_info.rs` with `use` statements that create aliases of the items generated by "built". Bumping MSRV to 1.74.1 also allows us to bump the version of the dependency "criterion" to 0.5 which also requires MSRV 1.74. Previously, we locked the version of "criterion" to 0.4 due to its MSRV requirement. We also updated all dependencies to their latest versions. Among those changes, the "sysinfo" crate renamed several `new` methods to `nothing`. We make changes accordingly. We also use `usize::div_ceil` which was introduced in Rust 1.73. This fixes a clippy warning. commit 4ca8812 Author: Yi Lin <qinsoon@gmail.com> Date: Thu Feb 6 12:30:42 2025 +1300 Fix julia extended tests (mmtk#1270) This PR changes the extended testing workflow for Julia: 1. It now tests with `master` in the upstream Julia repo, and `master` in the binding repo by default. 2. In addition to specifying the binding version, we can also specify the Julia version to run with. This change is necessary, as we no longe record the Julia version in the binding (we record the binding version in Julia instead). commit 054feef Author: tianleq <90177881+tianleq@users.noreply.github.com> Date: Wed Jan 29 14:07:49 2025 +1100 Clear stale line mark state (mmtk#1268) ![image](https://github.com/user-attachments/assets/5ce787c6-03c0-4845-b0a9-ade68bb0f8d6) ![image](https://github.com/user-attachments/assets/605711ba-1140-42b0-9306-849d4e15e244) ![image](https://github.com/user-attachments/assets/2f1ba1d5-9ca8-4678-b75a-45eac5789990) Overall, this fix does not incur significant overhead commit 051bc74 Author: Kunshan Wang <wks1986@gmail.com> Date: Tue Jan 21 13:31:06 2025 +0800 Make GC triggering and heap resizing consistent (mmtk#1266) This PR fixes a bug where the MemBalancer did not increase the heap size enough to accommodate the amount of side metadata needed by the pending allocation. It manifested as looping infinitely between triggering GC and (not actually) resizing the heap size after a GC when the minimum heap size is too small. Now it always includes the side metadata amount when increasing heap size. This PR also refactors the calculation of "shifting right and rounding up" which is used in multiple places. We also replace `alloc_rshift` with `log_data_meta_ratio` for two reasons. (1) The previous implementation would cause unsigned overflow before converting the result to `i32`. (2) `log_data_meta_ratio` has clearer semantics. commit c61e6c8 Author: Kunshan Wang <wks1986@gmail.com> Date: Thu Jan 16 20:05:08 2025 +0800 Force fixed heap size when using NoGC (mmtk#1264) The dynamic heap size trigger needs GC to adjust the heap size, but NoGC can't do GC. So it doesn't make sense to use dynamic heap size with NoGC. Currently, if the user selects the NoGC plan and the dynamic heap size trigger, it will trigger GC at the minimum heap size and then panic immediately. With this change, MMTk will give a warning and use fixed heap size trigger instead, using the maximum heap size specified in the dynamic trigger as the heap size. commit 2f6f078 Author: Kunshan Wang <wks1986@gmail.com> Date: Tue Jan 14 21:03:49 2025 +0800 Fix Clippy warning in 1.84.0 (mmtk#1262) Rust 1.84.0 added a new lint "unnecessary_map_or". We use `Option::is_some_and` (introduced in Rust 1.70.0) as suggested by the lint. commit 541298f Author: Kunshan Wang <wks1986@gmail.com> Date: Tue Jan 14 21:03:46 2025 +0800 Fix a subtraction overflow in get_free_pages. (mmtk#1261) The used pages can also be greater than the total pages for the same reason as those in computing `get_available_pages`, and it can also happen if the VM binding disabled GC, in which case we may over-allocate without triggering GC. When it overflows, `get_free_pages` will cause subtraction overflow, and will panic in debug build. We switch to `saturating_sub` so that it will return 0 if overflow happens. It still makes sense. 0 means there is no free pages because we are over-allocating beyond the current heap size set by the GC trigger. commit 68bf1b6 Author: Kunshan Wang <wks1986@gmail.com> Date: Thu Jan 9 14:40:42 2025 +0800 Unique object enqueuing option (mmtk#1255) Added a constant `VMBinding::UNIQUE_OBJECT_ENQUEUING`. When set to true, MMTk will guarantee that each object is enqueued at most once in each GC. This can be useful for VMs that piggyback on object scanning to visit objects during GC. Implementation-wise, the mark bit is set atomically when `VMBinding::UNIQUE_OBJECT_ENQUEUING` is true. This PR only affects the native MarkSweep space. Other spaces already do this atomically. Fixes: mmtk#1254 commit ec74535 Author: Yi Lin <qinsoon@gmail.com> Date: Tue Jan 7 09:56:50 2025 +1300 Move to Rust 1.83 (mmtk#1253) This PR updates our pinned Rust version to 1.83. This also updates `ci-perf-kit` https://github.com/mmtk/ci-perf-kit/releases/tag/0.8.2 that includes this Rust 1.83 migration as a new epoch. commit c0f9788 Author: Yi Lin <qinsoon@gmail.com> Date: Fri Dec 20 15:28:58 2024 +1300 Bump version to v0.30 (mmtk#1252) commit 2e548e5 Author: Yi Lin <qinsoon@gmail.com> Date: Mon Dec 9 21:23:21 2024 +1300 Allow setting object metadata for VM space objects. Expose VO bit under a feature. (mmtk#1248) This PR changes a few things for vo bit: 1. Add a function to set object metadata for an object in the VM space: `MMTK::initialize_vm_space_object`. 2. Add a feature `vo_bit_access` to expose VO bit and a binding may use it at its own risk. 3. Mark VO bit side metadata base address only avilable for 64 bits. The second is needed for Julia. The Julia binding uses MMTk immortal allocation or VM space for a region of memory, and pop the regions with boot image objects with no clear way to identify each object. The easist workaround is to bulk set VO bit for the region. The problem from this is that MMTk cannot identify valid objects in those regions. However, Julia binding only uses VO bit for pinning objects. Objects in the immortal space or the VM space will not be moved so failing to pinning objects in those regions is benign. Currently the Julia binding duplicates a bunch of side metadata code to bulk set VO bit only using the VO bit side metadata base address. See mmtk/mmtk-julia#200 commit 3c1418a Author: Yi Lin <qinsoon@gmail.com> Date: Mon Dec 9 18:26:42 2024 +1300 Check the option before aggregating live bytes data. Panic if the option is enabled on malloc space. (mmtk#1250) We see failures [here](https://github.com/mmtk/mmtk-core/actions/runs/12193674159/job/34020377988?pr=1248) in OpenJDK tests. ``` [2024-12-06T07:09:55Z INFO mmtk::memory_manager] Initialized MMTk with MarkSweep (FixedHeapSize(54525952)) [2024-12-06T07:09:55Z WARN mmtk::memory_manager] The feature 'extreme_assertions' is enabled. MMTk will run expensive run-time checks. Slow performance should be expected. ===== DaCapo fop starting ===== [2024-12-06T07:10:07Z INFO mmtk::util::heap::gc_trigger] [POLL] MallocSpace: Triggering collection (13313/13312 pages) thread '<unnamed>' panicked at /home/runner/work/mmtk-core/mmtk-core/mmtk-openjdk/repos/mmtk-core/src/policy/marksweepspace/malloc_ms/global.rs:158:9: internal error: entered unreachable code stack backtrace: 0: rust_begin_unwind at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:647:5 1: core::panicking::panic_fmt at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/panicking.rs:72:14 2: core::panicking::panic at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/panicking.rs:144:5 3: <mmtk::policy::marksweepspace::malloc_ms::global::MallocSpace<VM> as mmtk::policy::space::Space<VM>>::common at ./repos/mmtk-core/src/policy/marksweepspace/malloc_ms/global.rs:158:9 4: mmtk::policy::space::Space::get_descriptor at ./repos/mmtk-core/src/policy/space.rs:332:9 5: mmtk::mmtk::MMTK<VM>::aggregate_live_bytes_in_last_gc::{{closure}} at ./repos/mmtk-core/src/mmtk.rs:542:29 6: <mmtk::plan::marksweep::global::MarkSweep<VM> as mmtk::plan::global::HasSpaces>::for_each_space at ./repos/mmtk-core/src/plan/marksweep/global.rs:31:10 7: mmtk::mmtk::MMTK<VM>::aggregate_live_bytes_in_last_gc at ./repos/mmtk-core/src/mmtk.rs:540:9 8: <mmtk::scheduler::gc_work::Release<C> as mmtk::scheduler::work::GCWork<<C as mmtk::scheduler::work::GCWorkContext>::VM>>::do_work at ./repos/mmtk-core/src/scheduler/gc_work.rs:162:13 9: mmtk::scheduler::work::GCWork::do_work_with_stat at ./repos/mmtk-core/src/scheduler/work.rs:45:9 10: mmtk::scheduler::worker::GCWorker<VM>::run at ./repos/mmtk-core/src/scheduler/worker.rs:255:13 11: mmtk::memory_manager::start_worker at ./repos/mmtk-core/src/memory_manager.rs:491:5 12: start_worker at ./mmtk/src/api.rs:214:9 13: _ZN6Thread8call_runEv at ./repos/openjdk/src/hotspot/share/runtime/thread.cpp:402:12 14: thread_native_entry at ./repos/openjdk/src/hotspot/os/linux/os_linux.cpp:826:19 15: <unknown> 16: <unknown> note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. fatal runtime error: failed to initiate panic, error 5 ``` The issue is that `aggregate_live_bytes_in_last_gc` was not guarded by the condition that the option `count_live_bytes_in_gc` is enabled. So it was executed in our tests. The function accesses the space descriptor through `CommonSpace` and `MallocSpace` does not use `CommonSpace`, thus we see the panic. This PR adds a check before calling `aggregate_live_bytes_in_last_gc`. When the option is not enabled, we will not call the function. This PR also adds a panic for `MallocSpace`. If `count_live_bytes` is turned on, we simply panic, as we cannot provide live bytes vs total page stats for `MallocSpace`. commit e8ff7c6 Author: Yi Lin <qinsoon@gmail.com> Date: Thu Dec 5 15:32:12 2024 +1300 Use macos-15 for style check (mmtk#1249) mmtk#1216 updated the test runner image from `macos-12` to `macos-15`, but I forgot to update the image for style checks. This PR updates the runner for style checks as well. commit a753093 Author: Kunshan Wang <wks1986@gmail.com> Date: Tue Dec 3 17:37:46 2024 +0800 Minor changes for debugging. (mmtk#1245) Added `MMTK::debug_print_vm_map` which prints the memory ranges of spaces. `NullableObjectReference` now implements `Clone`, `Copy`, `Display` and `Debug`. This allows the binding to print its value like `Address` and `ObjectReference`, and is useful for logging API functions that involve `NullableObjectReference` parameters. commit 8a398e0 Author: Yi Lin <qinsoon@gmail.com> Date: Tue Dec 3 17:03:34 2024 +1300 Collect live bytes per space, and report by space (mmtk#1238) The current `count_live_bytes_in_gc` feature adds the size of all live objects and compare with the used pages reported by the plan. There are two issues with the feature: 1. VM space is not included in the used pages reported by the plan, but the live objects include objects in the VM space. So the reported fragmentation/utilization is wrong when the VM space is in use. 2. Spaces/policies have very different fragmentation ratio. Reporting the fragmentation for the entire heap is not useful. This PR refactors the current `count_live_bytes_in_gc` feature so we collect live bytes per space, and report by space. commit 3d7bc11 Author: Yi Lin <qinsoon@gmail.com> Date: Mon Dec 2 14:55:37 2024 +1300 Fix warnings for lifetime in MmapAnnotation impl (mmtk#1244) mmtk#1242 fixed most similar issues in the repo, but mmtk#1236 introduced `MmapAnnotation` and introduced a new warning. commit cd2fe83 Author: Kunshan Wang <wks1986@gmail.com> Date: Fri Nov 29 17:17:14 2024 +0800 Annotate mmap ranges using PR_SET_VMA (mmtk#1236) We demand that every invocation of `mmap` within mmtk-core to be accompanied with an "annotation" for the purpose of the mmap. On Linux, we will use `PR_SET_VMA_ANON_NAME` to set the attribute after `mmap` so that it can be seen in `/proc/pid/maps`. This will greatly improve the debugging experience. commit 5bc6ce5 Author: Kunshan Wang <wks1986@gmail.com> Date: Fri Nov 29 13:16:22 2024 +0800 Fix clippy warnings for Rust 1.83 (mmtk#1242) Clippy 1.83 produces some new warnings: - `needless_lifetimes` is extended to suggest eliding `impl` lifetimes. - `empty_line_after_doc_comments` is added to the `suspicious` group. commit 8640ab8 Author: Yi Lin <qinsoon@gmail.com> Date: Fri Nov 8 19:36:42 2024 +1300 Bump version to v0.29 (mmtk#1232) commit 41501a5 Author: Kunal Sareen <kunal.sareen@anu.edu.au> Date: Thu Nov 7 13:35:00 2024 +1100 Fix nightly build and add `inline` attributes to `{un,}likely` (mmtk#1228) commit 753f71c Author: Yi Lin <qinsoon@gmail.com> Date: Thu Nov 7 15:17:58 2024 +1300 Fix auto merge branches (mmtk#1230) This PR changes the auto merge workflow. For each binding, the workflow now allows inputs for base repo and base ref. This change is mostly for the Julia binding which uses `dev` instead of `master` as the default branch. mmtk#1221 only changed for the correctness testing, this PR made corresponding changes for auto merge. commit 59ea62e Author: Kunshan Wang <wks1986@gmail.com> Date: Thu Nov 7 05:31:13 2024 +0800 Use modern syntax for optional dependencies (mmtk#1229) Use the "dep:" prefix to specify optional dependencies in Cargo features. An optional crate dependency implicitly generates a feature of the same name, and can be leaked to the user of the current crate. But if a feature specifies a crate dependency with the "dep:" prefix, it will not implicitly generate the feature, hiding it from the users. The "dep:" prefix was introduced in Rust 1.60. commit 3575521 Author: Kunshan Wang <wks1986@gmail.com> Date: Wed Nov 6 13:49:45 2024 +0800 Make env_logger an optional dependency (mmtk#1226) Now the built-in `env_logger` is guarded behind a Cargo feature "builtin_env_logger". It is a default feature, but can be disabled in Cargo.toml by setting `dependencies.mmtk.default-features = false`. In this way, VM bindings that want to implement its own logger can remove the `env_logger` crate from its dependencies. Fixes: mmtk#744 commit 3830168 Author: Yi Lin <qinsoon@gmail.com> Date: Fri Nov 1 15:02:03 2024 +1300 Change the default testing branch for Julia tests (mmtk#1221) We recently re-organised branches in `mmtk-julia` and `julia`. Namely the previous `master` was renamed to `dev`, and we will use `master` for the version that works with Julia upstream. This PR extracts the default testing repos and branches, and changes the default testing branch for Julia. commit 618fde4 Author: Yi Lin <qinsoon@gmail.com> Date: Mon Oct 21 18:48:08 2024 +1300 Document the policy about performance testing environment and epochs (mmtk#1206) Co-authored-by: Kunshan Wang <wks1986@gmail.com> commit f032697 Author: Kunshan Wang <wks1986@gmail.com> Date: Mon Oct 21 13:46:06 2024 +0800 Performance history canary (mmtk#1209) This PR adds a "canary" build to the performance regression CI of OpenJDK. The "canary" is a chosen revision of mmtk-core and mmtk-openjdk that is tested alongside each merged PR. The performance of the "canary" should not change unless there is an environment change or there is a noise. Spotting a change in the "canary" performance can help us identify environment changes that are unintended or otherwise unnoticed, and also identify the noise level. Currently, we choose a specific release version as the version of the "canary". Using a release version has the advantage of being easy to specify the exact revision of both the mmtk-core and the mmtk-openjdk repository. We may also switch to some methods of automatically select the canary version in the future. There are other minor changes made. - We slightly change the directory structure. We create two directory, namely `latest` and `canary`. In each of the directories, we check out `mmtk-core` and `mmtk-openjdk` of the latest and the canary versions, respectively. - We use the `ci-replace-mmtk-dep.py` script to replace the revision of the `mmtk-core` dependency in `mmtk-openjdk`. As a result, we no longer need to use `sed`, and no longer need to copy the `mmtk-core` directory into `mmtk-openjdk/repos`. - We no longer set the `RUSTUP_TOOLCHAIN` environment variable because 1. The latest and the canary version may not use the same toolchain, and, 2. the right toolchain will be chosen when running the `cargo` command according to the `rust-toolchain` file in the directory. - The scripts in https://github.com/mmtk/ci-perf-kit are changed to take the canary into consideration, too. commit 80b11a0 Author: Patrick LaFontaine <32135464+Pat-Lafon@users.noreply.github.com> Date: Sun Oct 20 20:05:29 2024 -0400 Remove space for nogc link (mmtk#1217) commit 0883898 Author: Yi Lin <qinsoon@gmail.com> Date: Thu Oct 17 20:00:03 2024 +1300 Update CI macos image (mmtk#1216) macos-12 will no longer be supported: actions/runner-images#10721 commit 328deb6 Author: Kunshan Wang <wks1986@gmail.com> Date: Fri Oct 11 16:07:53 2024 +0800 Fix a race between forwarding bits and VO bits. (mmtk#1214) The current code sets the forwarding bits before setting the VO bit when copying an object. If another GC worker is attempting to forward the same object, it may observe the forwarding bits being `FORWARDED` but the VO bit is not set. This violates the semantics of VO bits because VO bits should be set for both from-space and to-space copies. This will affect VM bindings that assert slots always refer to a valid object when scanning objects and may update the same slot multiple times for some reasons. This revision provides a mechanism to ensure that all necessary metadata are set before setting forwarding bits to `FORWARDED`. Currently it affects the VO bits and the mark bits (which are used to update the VO bits in Immix-based plans). It may be used for other metadata introduced in the future. commit 58b3b35 Author: Kunshan Wang <wks1986@gmail.com> Date: Fri Oct 11 14:04:55 2024 +0800 Install cargo-msrv using stable toolchain. (mmtk#1215) When running the CI check "msrv", we install the cargo-msrv command using the stable Rust toolchain because it sometimes requires a higher Rust version than our chosen version in the file `rust-toolchain`. commit c4fdce0 Author: Yi Lin <qinsoon@gmail.com> Date: Fri Sep 27 22:27:47 2024 +1200 Bump version to v0.28 (mmtk#1212) Merge this PR after mmtk#1208 and mmtk#1211. commit de10fa4 Author: Yi Lin <qinsoon@gmail.com> Date: Fri Sep 27 16:41:37 2024 +1200 Update migration guide for mmtk#1205 (mmtk#1211) Co-authored-by: Kunshan Wang <wks1986@gmail.com> commit 7cfebda Author: Yi Lin <qinsoon@gmail.com> Date: Fri Sep 27 15:37:20 2024 +1200 Update ci-perf-kit to plot epoch (mmtk#1208) Use mmtk/ci-perf-kit#46 to plot performance data. commit 5605237 Author: Kunal Sareen <kunal.sareen@anu.edu.au> Date: Fri Sep 20 16:37:13 2024 +1000 Return if a GC ran or not for `handle_user_collection_request` (mmtk#1205) Closes mmtk#1204 --------- Co-authored-by: Yi Lin <qinsoon@gmail.com> commit d56c3b9 Merge: 4cfac97 dd84218 Author: Eduardo Souza <ledusou@gmail.com> Date: Wed Mar 27 00:26:59 2024 +0000 Merge remote-tracking branch 'mmtk/master' into feature/check-fragmentation-immixspace commit 4cfac97 Author: Eduardo Souza <ledusou@gmail.com> Date: Tue Mar 26 23:27:59 2024 +0000 Refactor code; turn on logs; set block size to 16K commit aeb3aeb Author: Eduardo Souza <ledusou@gmail.com> Date: Wed Mar 6 23:12:08 2024 +0000 Adding statistics about number of objects scanned and objects moved in immixspace commit 6f1c924 Author: Eduardo Souza <ledusou@gmail.com> Date: Wed Mar 6 23:02:10 2024 +0000 Change printing info commit f294948 Author: Eduardo Souza <ledusou@gmail.com> Date: Tue Mar 5 01:08:04 2024 +0000 Adding assertion for live lines in immixspace; properly counting live bytes in los commit 184822c Author: Eduardo Souza <ledusou@gmail.com> Date: Tue Mar 5 01:07:19 2024 +0000 Removing dependency on chrono commit 1547428 Author: Eduardo Souza <ledusou@gmail.com> Date: Mon Mar 4 09:52:28 2024 +0000 Print stats for sticky immix as well commit 06284e1 Author: Eduardo Souza <ledusou@gmail.com> Date: Mon Mar 4 09:37:45 2024 +0000 Adding feature to dump memory stats (from los and immixspace) commit 1f39198 Author: Eduardo Souza <ledusou@gmail.com> Date: Mon Mar 4 04:12:19 2024 +0000 Trying to count live blocks and live lines commit bd3305f Author: Eduardo Souza <ledusou@gmail.com> Date: Mon Mar 4 03:26:42 2024 +0000 Zeroing the live bytes right before GC starts commit ee21a9c Author: Eduardo Souza <ledusou@gmail.com> Date: Thu Feb 29 22:59:56 2024 +0000 Moving stats from global state to immixspace commit 530051b Author: Eduardo Souza <ledusou@gmail.com> Date: Thu Feb 29 04:33:27 2024 +0000 Refactor range check commit f081e4f Author: Eduardo Souza <ledusou@gmail.com> Date: Thu Feb 29 04:23:19 2024 +0000 Removing duplicated method commit c2a79ad Author: Eduardo Souza <ledusou@gmail.com> Date: Thu Feb 29 04:13:55 2024 +0000 Adding feature to query the fragmentation of immixspace

wks added 2 commits January 21, 2025 19:33

Special topic chapter for finalizer & weakref

94f192b

Added a special topic chapter for how to implement finalizers and weak references with MMTk.

State machine and deprecated things

be789b2

wks force-pushed the feature/weak-final-guide branch from b2ea9b3 to be789b2 Compare January 21, 2025 11:51

wks added 7 commits January 22, 2025 16:19

Optimization section

a176a41

Update comments

f1c0970

Minor fixes

29d443c

Use the word "resurrect" consistently

79c8398

Rewrite part of the finalizers section

fc946c8

Revise the weakref and optimization sections

160368c

Use en_US spelling consistently

b9c4086

wks changed the title ~~WIP: Special topic chapter for weak references~~ Special topic chapter for finalizers and weak references Jan 23, 2025

wks marked this pull request as ready for review January 23, 2025 06:24

wks requested review from qinsoon and k-sareen January 23, 2025 06:25

Ephemeron

9ec8d5d

qinsoon reviewed Jan 24, 2025

View reviewed changes

wks added 6 commits February 12, 2025 16:06

Rename "Special Topics" to "VM-specific Concerns"

e671ab0

Define finalizer and weak references

2dd16e2

WIP: Use "retain" instead of "resurrect"

e038fc1

Use "retain" instead of "resurrect"

30f5bfc

... unless "resurrect" refers to the application-visible phenomenon.

Minor changes

b07dbd9

Merge branch 'master' into feature/weak-final-guide

1d67c22

wks mentioned this pull request Feb 13, 2025

Rename is_reachable and is_live #1271

Closed

wks added 4 commits February 18, 2025 14:15

Minor changes

abf91d1

- Slightly rephrased the beginning of the overview section - Always use perfect tense when talking about "reached" and "not reached".

Clarify the scope and return value of is_reachable

aa0930d

Use a mock test to host the example code.

8b2b89d

This makes sure the code compiles, and makes sure we notice it if the API of the `process_weak_refs` API is changed.

Fix typo

caefc0c

Link to .md instead of .html

540f477

k-sareen reviewed Feb 20, 2025

View reviewed changes

wks added 4 commits February 20, 2025 12:41

Merge branch 'master' into feature/weak-final-guide

f7df472

Define emergency GC in the glossary

5a9bd5f

Address comments from Kunal

1aa4511

Added more inter-section links

fb47ae5

k-sareen approved these changes Feb 20, 2025

View reviewed changes

wks added this pull request to the merge queue Feb 20, 2025

Merged via the queue into mmtk:master with commit 84545cc Feb 20, 2025
33 checks passed

wks deleted the feature/weak-final-guide branch February 20, 2025 09:21

wks mentioned this pull request Feb 26, 2025

Define "root" and rename methods of RootsWorkFactory #1282

Open

		@@ -0,0 +1,5 @@
		# Special topics

		Every VM is special in some way. Because of this, some VM bindings may use MMTk features not

	- (This step may expand the transitive closure in emergency GC if any referents are retained.)
	- (This step may expand the transitive closure in an emergency GC if any referents are retained.)

Special topic chapter for finalizers and weak references #1265

Special topic chapter for finalizers and weak references #1265

Uh oh!

Conversation

wks commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wks commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wks commented Jan 23, 2025

Uh oh!

qinsoon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-sareen commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wks commented Feb 18, 2025

Uh oh!

k-sareen commented Feb 18, 2025

Uh oh!

wks commented Feb 18, 2025

Uh oh!

k-sareen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wks commented Feb 20, 2025

Uh oh!

k-sareen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wks commented Jan 16, 2025 •

edited

Loading

wks commented Jan 23, 2025 •

edited

Loading

k-sareen commented Jan 24, 2025 •

edited

Loading