Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compact heap before calling C linker #33

Merged
merged 1 commit into from
May 18, 2021

Conversation

trishume
Copy link
Contributor

Reduces peak memory usage, which is important when a build system
tries to run many large links in parallel.

Testing

  • Ran a large link before and after the change while watching htop, verified that before the change ocamlopt had 1.2G of RSS while the C linker was running and after it had 160MB.
  • Ran a large link and linked the compiler while printing the major heap word count and whether it compacted. Saw that the large link was at over 2x the threshold and the compiler links were something like 10x under it.
  • Ran the link with profiling and verified that the compaction only adds around 0.6s to a link that takes 20s when using Gold, but reduces peak memory by around 25% such that Gold and this change has a lower peak memory than GNU ld without this change.

Reduces peak memory usage, which is important when a build system
tries to run many large links in parallel.
@trishume trishume requested a review from mshinwell May 17, 2021 18:47
@mshinwell mshinwell merged commit 7aec5bb into ocaml-flambda:4.11 May 18, 2021
@trishume trishume deleted the link-compact branch May 18, 2021 13:55
poechsel pushed a commit to poechsel/flambda-backend that referenced this pull request May 26, 2021
Reduces peak memory usage, which is important when a build system
tries to run many large links in parallel.
poechsel pushed a commit to poechsel/flambda-backend that referenced this pull request May 27, 2021
Reduces peak memory usage, which is important when a build system
tries to run many large links in parallel.
poechsel pushed a commit that referenced this pull request Jun 28, 2021
Reduces peak memory usage, which is important when a build system
tries to run many large links in parallel.
poechsel pushed a commit that referenced this pull request Jun 29, 2021
Reduces peak memory usage, which is important when a build system
tries to run many large links in parallel.
stedolan added a commit to stedolan/flambda-backend that referenced this pull request Jan 18, 2022
1924269795 Several fixes for partial application and currying
4fee6ae2e8 Pprintast support for new local syntax
8df43e93e5 Quieten Makefile when runtime dep files are not present
88ec84e29e Typecheck x |> f y as (f y x), not ((f y) x)
87a10e3348 Remove autogeneration of @ocaml.curry
c656dc9 Merge flambda-backend changes
11b5424 Avoid printing double spaces in function argument lists
7751faa Restore locations to Typedtree.{pat,let}_bound_idents_full
e450b6c add build_ocaml_compiler.sexp
0403bb3 Revert PR 9895 to continue installing VERSION
b3447db Ensure new local attributes are namespaced properly
7f213fc Allow empty functions again
8f22ad8 Bugfix: ensure local domain state is initialised
80f54dd Bugfix for Selectgen with regions
e8133a1 Fix external-external signature inclusion
9840051 Bootstrap
d879f23 Merge remote-tracking branch 'jane/local-reviewed' into local-merge
94454f5 Use Local_store for the local allocations ref
54a164c Create fewer regions, according to typechecking (ocaml-flambda#59)
1c2479b Merge flambda-backend changes
ce34678 Fix printing of modes in return types
91f2281 Hook mode variable solving into Btype.snapshot/backtrack
54e4b09 Move Alloc_mode and Value_mode to Btype
ff4611e Merge flambda-backend changes
ce62e45 Ensure allocations are initialised, even dead ones
6b6ec5a Fix the alloc.ml test on 32-bit builds
81e9879 Merge flambda-backend changes
40a7f89 Update repo URL for ocaml-jst, and rename script.
0454ee7 Add some new locally-allocating primitives (ocaml-flambda#57)
8acdda1 Reset the local stack pointer in exception handlers (ocaml-flambda#56)
8dafa98 Improve typing for (||) and (&&) (ocaml-flambda#55)
8c64754 Fix make_check_all_arches (ocaml-flambda#54)
b50cd45 Allow arguments to primitives to be local even in tail position (ocaml-flambda#53)
cad125d Fix modes from or-patterns (ocaml-flambda#50)
4efdb72 Fix tailcalls tests with inlining (ocaml-flambda#52)
4a795cb Flambda support (ocaml-flambda#49)
74722cb Add [@ocaml.principal] and [@ocaml.noprincipal] attributes, and use in oo.mli
6d7d3b8 Ensure that functions are evaluated after their arguments (flambda-backend ocaml-flambda#353)
89bda6b Keep Sys.opaque_identity in Cmm and Mach (port upstream PR 9412)
a39126a Fix tailcalls within regions (ocaml-flambda#48)
4ac4cfd Fix stdlib manpages build
3a95f5e Merge flambda-backend changes
efe80c9 Add jane/pull-flambda-patches script
fca94c4 Register allocations for Omitted parameter closures (ocaml-flambda#47)
103b139 Remove various FIXMEs (ocaml-flambda#46)
62ba2c1 Bootstrap
a0062ad Allow local allocations for various primitives (ocaml-flambda#43)
7a2165e Allow primitives to be poly-moded (ocaml-flambda#43)
2af3f55 Fix a flaky test by refactoring TypePairs (ocaml/ocaml#10638)
58dd807 Bootstrap
ee3be10 Fix modes in build_apply for partial applications
fe73656 Tweak for evaluation order of labelled partial applications (#10653)
0527570 Fix caml_modify on local allocations (ocaml-flambda#40)
e657e99 Relax modes for `as` patterns (ocaml-flambda#42)
f815bf2 Add special mode handling for tuples in matches and let bindings (ocaml-flambda#38)
39f1211 Only take the upper bounds of modes associated with allocations (ocaml-flambda#37)
aec6fde Interpret arrow types in "local positions" differently
c4f3319 Bootstrap
ff6fdad Add some missing regions
40d586d Bootstrap
66d8110 Switch to a system with 3 modes for values
f2c5a85 Bugfix for Comballoc with local allocations. (ocaml-flambda#41)
83bcd09 Fix bug with root scanning during compaction (ocaml-flambda#39)
1b5ec83 Track modes in Lambda.lfunction and onwards (ocaml-flambda#33)
f1e2e97 Port ocaml/ocaml#10728
56703cd Port ocaml/ocaml#10081
eb66785 Support local allocations in i386 and fix amd64 bug (ocaml-flambda#31)
c936b19 Disallow local recursive non-functions (ocaml-flambda#30)
c7a193a GC support for local allocations (ocaml-flambda#29)
8dd7270 Nonlocal fields (ocaml-flambda#28)
e19a2f0 Bootstrap
694b9ac Add syntax to the parser for local allocations (ocaml-flambda#26)
f183008 Lower initial stack size
918226f Allow local closure allocations (ocaml-flambda#27)
2552e7d Introduce mode variables (ocaml-flambda#25)
bc41c99 Minor fixes for local allocations (ocaml-flambda#24)
a2a4e60 Runtime and compiler support for more local allocations (ocaml-flambda#23)
d030554 Typechecking for local allocations (ocaml-flambda#21)
9ee2332 Bugfix missing from ocaml-flambda#20
02c4cef Retain block-structured local regions until Mach.
86dbe1c amd64: Move stack realloc calls out-of-line
324d218 More typing modes and locking of environments
a4080b8 Initial version of local allocation (unsafe)

git-subtree-dir: ocaml
git-subtree-split: 1924269795db2450be5c084f7799340e0e003e19
stedolan added a commit that referenced this pull request Feb 1, 2022
173842c Merge flambda-backend changes
ed7eba2 Remove leading space from LINE. (#484)
bd61170 Bump magic numbers (#5)
c50c47d Add CI builds with local allocations enabled
1412792 Move local allocations support behind '-extension local'
6d8e42a Better tail call behaviour in caml_applyN
c7dac3d Typemod: toplevel bindings escape even if no variables are bound
82d6c3e Several fixes for partial application and currying
d05c70c Pprintast support for new local syntax
e0e62fc Typecheck x |> f y as (f y x), not ((f y) x)
d7e34ce Remove autogeneration of @ocaml.curry
b9a0593 Port #493
0a872d9 Code review fixes from #491
6c168bb Remove local allocation counting
3c6e7f0 Code review fixes from #478
bb97207 Rename Lambda.apply_position
a7cb650 Quieten Makefile when runtime dep files are not present
c656dc9 Merge flambda-backend changes
11b5424 Avoid printing double spaces in function argument lists
7751faa Restore locations to Typedtree.{pat,let}_bound_idents_full
e450b6c add build_ocaml_compiler.sexp
0403bb3 Revert PR 9895 to continue installing VERSION
b3447db Ensure new local attributes are namespaced properly
7f213fc Allow empty functions again
8f22ad8 Bugfix: ensure local domain state is initialised
80f54dd Bugfix for Selectgen with regions
e8133a1 Fix external-external signature inclusion
9840051 Bootstrap
d879f23 Merge remote-tracking branch 'jane/local-reviewed' into local-merge
94454f5 Use Local_store for the local allocations ref
54a164c Create fewer regions, according to typechecking (#59)
1c2479b Merge flambda-backend changes
ce34678 Fix printing of modes in return types
91f2281 Hook mode variable solving into Btype.snapshot/backtrack
54e4b09 Move Alloc_mode and Value_mode to Btype
ff4611e Merge flambda-backend changes
ce62e45 Ensure allocations are initialised, even dead ones
6b6ec5a Fix the alloc.ml test on 32-bit builds
81e9879 Merge flambda-backend changes
40a7f89 Update repo URL for ocaml-jst, and rename script.
0454ee7 Add some new locally-allocating primitives (#57)
8acdda1 Reset the local stack pointer in exception handlers (#56)
8dafa98 Improve typing for (||) and (&&) (#55)
8c64754 Fix make_check_all_arches (#54)
b50cd45 Allow arguments to primitives to be local even in tail position (#53)
cad125d Fix modes from or-patterns (#50)
4efdb72 Fix tailcalls tests with inlining (#52)
4a795cb Flambda support (#49)
74722cb Add [@ocaml.principal] and [@ocaml.noprincipal] attributes, and use in oo.mli
6d7d3b8 Ensure that functions are evaluated after their arguments (flambda-backend #353)
89bda6b Keep Sys.opaque_identity in Cmm and Mach (port upstream PR 9412)
a39126a Fix tailcalls within regions (#48)
4ac4cfd Fix stdlib manpages build
3a95f5e Merge flambda-backend changes
efe80c9 Add jane/pull-flambda-patches script
fca94c4 Register allocations for Omitted parameter closures (#47)
103b139 Remove various FIXMEs (#46)
62ba2c1 Bootstrap
a0062ad Allow local allocations for various primitives (#43)
7a2165e Allow primitives to be poly-moded (#43)
2af3f55 Fix a flaky test by refactoring TypePairs (ocaml/ocaml#10638)
58dd807 Bootstrap
ee3be10 Fix modes in build_apply for partial applications
fe73656 Tweak for evaluation order of labelled partial applications (#10653)
0527570 Fix caml_modify on local allocations (#40)
e657e99 Relax modes for `as` patterns (#42)
f815bf2 Add special mode handling for tuples in matches and let bindings (#38)
39f1211 Only take the upper bounds of modes associated with allocations (#37)
aec6fde Interpret arrow types in "local positions" differently
c4f3319 Bootstrap
ff6fdad Add some missing regions
40d586d Bootstrap
66d8110 Switch to a system with 3 modes for values
f2c5a85 Bugfix for Comballoc with local allocations. (#41)
83bcd09 Fix bug with root scanning during compaction (#39)
1b5ec83 Track modes in Lambda.lfunction and onwards (#33)
f1e2e97 Port ocaml/ocaml#10728
56703cd Port ocaml/ocaml#10081
eb66785 Support local allocations in i386 and fix amd64 bug (#31)
c936b19 Disallow local recursive non-functions (#30)
c7a193a GC support for local allocations (#29)
8dd7270 Nonlocal fields (#28)
e19a2f0 Bootstrap
694b9ac Add syntax to the parser for local allocations (#26)
f183008 Lower initial stack size
918226f Allow local closure allocations (#27)
2552e7d Introduce mode variables (#25)
bc41c99 Minor fixes for local allocations (#24)
a2a4e60 Runtime and compiler support for more local allocations (#23)
d030554 Typechecking for local allocations (#21)
9ee2332 Bugfix missing from #20
02c4cef Retain block-structured local regions until Mach.
86dbe1c amd64: Move stack realloc calls out-of-line
324d218 More typing modes and locking of environments
a4080b8 Initial version of local allocation (unsafe)

git-subtree-dir: ocaml
git-subtree-split: 173842c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants