-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
alloc profiler: get stacks for pool allocs via wrapper function #17
alloc profiler: get stacks for pool allocs via wrapper function #17
Conversation
Prevents loading of `libjulia-codegen` into `opt`. Co-authored-by: Jameson Nash <vtjnash@gmail.com>
JuliaLang#43663)" This reverts commit 85a6990.
Regression introduced by ca11c2f (when tests fail or are interrupted)
Fix llvmpasses test
See comment in diff for explanation. This fixes test/stacktraces.jl on aarch64 macOS 12, and according to an OpenJDK issue where they ran into the same problem, https://git.openjdk.java.net/jdk/pull/6193, probably also x86_64 macOS 12.
Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
In a similar spirit to JuliaLang#43587, this commit introduces error check for `setfield!`. We can bail out from inference if we can prove either of: - the object is not mutable type - the object is `Module` object - the value being assigned is incompatible with the declared type of object field This commit also adds the throwness check for `setfield!` (i.e. `setfield!_nothrow`). This throwness check won't be used in the current native compilation pipeline since `setfield!` call can't be eliminated even if we can prove that it never throws. But this throwness check would be used by EscapeAnalysis.jl integration and so I'd like to include it in Base.
…JuliaLang#43587) Obvious errors are usually caught in higher places like `convert`, but it's better to have error checks at each builtin level in order to enable early bail out from errorneous code compilation (when somehow it does not rely on common abstraction). These checks are also useful for third consumers like JET.
…uliaLang#43684) Previously, we had the machinery to load per-UUID, but some tooling needs to be able to load all preferences (without loading `Preferences.jl`), so it's better to support being able to load everything from just within Base.
…FO` (for e.g. when Git is not available) (JuliaLang#43717) * `base/version_git.sh`: fix the default value for `Base.GIT_VERSION_INFO` (for e.g. when Git is not available) * Use single-quotes to avoid the need to backslash-escape the double quotes
4c45f29 had reintroduced a use of getNumArgOperands(), which was already slated for removal in (at least) LLVM 12.
This shouldn't matter on any platform with a sensible ABI; just happened across this when tracking down some other issue.
Also makes it clear to the reader that signals-mach.c uses a different approach entirely (which almost tripped me up the first time I was reading through this part of the codebase).
Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
4c4aaa7
to
aae4442
Compare
2081749
to
85d4797
Compare
…ng#43426) We could also consider adding `defalg(v::AbstractArray{<:Number}) = DEFAULT_UNSTABLE`, but it is unlikely that someone will want to do `Union` of `<:Number` and something other than `Missing` that still would support comparison. Relevant for JuliaStrings/InlineStrings.jl#21 (and other custom types that will want to add `defalg` support allowing for `Union` with `Missing`).
…3830) This was open-coded inside inlining, but the effect-free modeling code should be with the other builtins (in preparation of calling it from more places).
85d4797
to
5c6ae25
Compare
JL_DLLEXPORT jl_value_t *jl_gc_pool_alloc_outer(jl_ptls_t ptls, int pool_offset, | ||
int osize) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My main comment is about trying to find a more informative name for this, like maybe:
jl_gc_pool_alloc_from_gen
jl_gc_pool_alloc_from_llvm_gen
jl_gc_pool_alloc_from_generated
...jl_gc_pool_alloc_that_will_be_called_from_generated_code
... i dunno...
Blech naming is hard
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome. So, I think this looks great, totally solves the problem, and makes sense! 👍
I do think it'd be good to get a second opinion. Thanks for all the reviews of late, @vchuravy -- would you be able to have a look here too? The PR description explains very well what we did here. ❤️
@@ -1196,6 +1196,16 @@ static NOINLINE jl_taggedvalue_t *add_page(jl_gc_pool_t *p) JL_NOTSAFEPOINT | |||
return fl; | |||
} | |||
|
|||
// called into by LLVM-generated code | |||
JL_DLLEXPORT jl_value_t *jl_gc_pool_alloc_outer(jl_ptls_t ptls, int pool_offset, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer keeping the same name here, and renaming the inner internal
implementation, which then also shouldn't be exported (and ideally inlined into this function).
Maybe call that one then jl_gc_pool_alloc_impl
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We talked about inlining it too, but we weren't sure about how to inline it only into this function..
If we mark that one inline, it'll end up inlining into the normal callers of it, though.. So the C functions (indicated in Pete's graph, above) that calls jl_gc_pool_alloc
would now call jl_gc_pool_alloc_impl
instead, and then it would be inlined.
So it'd become inlined into jl_gc_alloc_
and jl_gc_alloc_string
. Maybe that's okay?
Or would you want us to create yet a third separate wrapper function that gets called by those? So we would have three functions:
jl_gc_pool_alloc_from_generated
: called from generated codejl_gc_pool_alloc
: called fromjl_gc_alloc_
andjl_gc_alloc_string
jl_gc_pool_alloc_impl
: called fromjl_gc_pool_alloc
? Or is that too weird? And the extra inlining is actually fine?
* Define `extrema` using `mapreduce`; support `init` * Fix tests for SparseArrays * API clean and export `extrema!` * Re-implement `reducedim_init` for extrema * Merge `master` to pull in JuliaSparse/SparseArrays.jl#63 * Mark `BigInt` tests as broken Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr> Co-authored-by: Simeon Schaub <simeondavidschaub99@gmail.com> Co-authored-by: Takafumi Arakaki <aka.tkf@gmail.com> Co-authored-by: Tim Holy <tim.holy@gmail.com>
…#43812) When `tmerge` is applied to `Const`/`PartialStruct` and a field is `Const` in one of the types and `Union{}` (i.e. undef) in the other, the resulting field type must not be `Const` (as the resursively called `tmerge` produces). Fixes JuliaLang#43784.
## Overview Record the type and stack of every allocation (or only at a given sample interval), and return as Julia objects. Alternate approach to existing alloc profiler PR: JuliaLang#33467 Complementary to garbage profiler PR: JuliaLang#42658 (maybe there's some nice way to meld them) This may be reinventing the wheel from JuliaLang#33467, but I'm not sure why that one needs stuff like LLVM passes. I mimicked some stuff from it, but this was my attempt to get something up and running. Could easily be missing stuff. ## Usage: ```julia using Profile.Allocs res = Allocs.@Profile sample_rate=0.001 my_func() prof = Allocs.fetch() # do something with `prof` ``` See also: JuliaPerf/PProf.jl#46 for support for visualizing these. Co-authored-by: Nathan Daly <nhdaly@gmail.com>
6bed933
to
d0f5ce0
Compare
Closed in favor of JuliaLang#43868 cuz I couldn't change the base to point at JuliaLang/julia |
When calling `jl_error()` or `jl_errorf()`, we must check to see if we are so early in the bringup process that it is dangerous to attempt to construct a backtrace because the data structures used to provide line information are not properly setup. This can be easily triggered by running: ``` julia -C invalid ``` On an `i686-linux-gnu` build, this will hit the "Invalid CPU Name" branch in `jitlayers.cpp`, which calls `jl_errorf()`. This in turn calls `jl_throw()`, which will eventually call `jl_DI_for_fptr` as part of the backtrace printing process, which fails as the object maps are not fully initialized. See the below `gdb` stacktrace for details: ``` $ gdb -batch -ex 'r' -ex 'bt' --args ./julia -C invalid ... fatal: error thrown and no exception handler available. ErrorException("Invalid CPU name "invalid".") Thread 1 "julia" received signal SIGSEGV, Segmentation fault. 0xf75bd665 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo>, std::_Select1st<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> >, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__k=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h:1277 1277 /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h: No such file or directory. #0 0xf75bd665 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo>, std::_Select1st<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> >, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__k=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h:1277 #1 std::map<unsigned int, JITDebugInfoRegistry::ObjectInfo, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__x=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_map.h:1258 #2 jl_DI_for_fptr (fptr=4155049385, symsize=symsize@entry=0xffffcfa8, slide=slide@entry=0xffffcfa0, Section=Section@entry=0xffffcfb8, context=context@entry=0xffffcf94) at /cache/build/default-amdci5-4/julialang/julia-master/src/debuginfo.cpp:1181 #3 0xf75c056a in jl_getFunctionInfo_impl (frames_out=0xffffd03c, pointer=4155049385, skipC=0, noInline=0) at /cache/build/default-amdci5-4/julialang/julia-master/src/debuginfo.cpp:1210 #4 0xf7a6ca98 in jl_print_native_codeloc (ip=4155049385) at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:636 #5 0xf7a6cd54 in jl_print_bt_entry_codeloc (bt_entry=0xf0798018) at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:657 #6 jlbacktrace () at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:1090 #7 0xf7a3cd2b in ijl_no_exc_handler (e=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:605 #8 0xf7a3d10a in throw_internal (ct=ct@entry=0xf070c010, exception=<optimized out>, exception@entry=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:638 #9 0xf7a3d330 in ijl_throw (e=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:654 #10 0xf7a905aa in ijl_errorf (fmt=fmt@entry=0xf7647cd4 "Invalid CPU name \"%s\".") at /cache/build/default-amdci5-4/julialang/julia-master/src/rtutils.c:77 #11 0xf75a4b22 in (anonymous namespace)::createTargetMachine () at /cache/build/default-amdci5-4/julialang/julia-master/src/jitlayers.cpp:823 #12 JuliaOJIT::JuliaOJIT (this=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/jitlayers.cpp:1044 #13 0xf7531793 in jl_init_llvm () at /cache/build/default-amdci5-4/julialang/julia-master/src/codegen.cpp:8585 #14 0xf75318a8 in jl_init_codegen_impl () at /cache/build/default-amdci5-4/julialang/julia-master/src/codegen.cpp:8648 #15 0xf7a51a52 in jl_restore_system_image_from_stream (f=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2131 #16 0xf7a55c03 in ijl_restore_system_image_data (buf=0xe859c1c0 <jl_system_image_data> "8'\031\003", len=125161105) at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2184 #17 0xf7a55cf9 in jl_load_sysimg_so () at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:424 #18 ijl_restore_system_image (fname=0x80a0900 "/build/bk_download/julia-d78fdad601/lib/julia/sys.so") at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2157 #19 0xf7a3bdfc in _finish_julia_init (rel=rel@entry=JL_IMAGE_JULIA_HOME, ct=<optimized out>, ptls=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/init.c:741 #20 0xf7a3c8ac in julia_init (rel=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/init.c:728 #21 0xf7a7f61d in jl_repl_entrypoint (argc=<optimized out>, argv=0xffffddf4) at /cache/build/default-amdci5-4/julialang/julia-master/src/jlapi.c:705 #22 0x080490a7 in main (argc=3, argv=0xffffddf4) at /cache/build/default-amdci5-4/julialang/julia-master/cli/loader_exe.c:59 ``` To prevent this, we simply avoid calling `jl_errorf` this early in the process, punting the problem to a later PR that can update guard conditions within `jl_error*`.
When calling `jl_error()` or `jl_errorf()`, we must check to see if we are so early in the bringup process that it is dangerous to attempt to construct a backtrace because the data structures used to provide line information are not properly setup. This can be easily triggered by running: ``` julia -C invalid ``` On an `i686-linux-gnu` build, this will hit the "Invalid CPU Name" branch in `jitlayers.cpp`, which calls `jl_errorf()`. This in turn calls `jl_throw()`, which will eventually call `jl_DI_for_fptr` as part of the backtrace printing process, which fails as the object maps are not fully initialized. See the below `gdb` stacktrace for details: ``` $ gdb -batch -ex 'r' -ex 'bt' --args ./julia -C invalid ... fatal: error thrown and no exception handler available. ErrorException("Invalid CPU name "invalid".") Thread 1 "julia" received signal SIGSEGV, Segmentation fault. 0xf75bd665 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo>, std::_Select1st<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> >, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__k=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h:1277 1277 /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h: No such file or directory. #0 0xf75bd665 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo>, std::_Select1st<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> >, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__k=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h:1277 #1 std::map<unsigned int, JITDebugInfoRegistry::ObjectInfo, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__x=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_map.h:1258 #2 jl_DI_for_fptr (fptr=4155049385, symsize=symsize@entry=0xffffcfa8, slide=slide@entry=0xffffcfa0, Section=Section@entry=0xffffcfb8, context=context@entry=0xffffcf94) at /cache/build/default-amdci5-4/julialang/julia-master/src/debuginfo.cpp:1181 #3 0xf75c056a in jl_getFunctionInfo_impl (frames_out=0xffffd03c, pointer=4155049385, skipC=0, noInline=0) at /cache/build/default-amdci5-4/julialang/julia-master/src/debuginfo.cpp:1210 #4 0xf7a6ca98 in jl_print_native_codeloc (ip=4155049385) at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:636 #5 0xf7a6cd54 in jl_print_bt_entry_codeloc (bt_entry=0xf0798018) at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:657 #6 jlbacktrace () at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:1090 #7 0xf7a3cd2b in ijl_no_exc_handler (e=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:605 #8 0xf7a3d10a in throw_internal (ct=ct@entry=0xf070c010, exception=<optimized out>, exception@entry=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:638 #9 0xf7a3d330 in ijl_throw (e=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:654 #10 0xf7a905aa in ijl_errorf (fmt=fmt@entry=0xf7647cd4 "Invalid CPU name \"%s\".") at /cache/build/default-amdci5-4/julialang/julia-master/src/rtutils.c:77 #11 0xf75a4b22 in (anonymous namespace)::createTargetMachine () at /cache/build/default-amdci5-4/julialang/julia-master/src/jitlayers.cpp:823 #12 JuliaOJIT::JuliaOJIT (this=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/jitlayers.cpp:1044 #13 0xf7531793 in jl_init_llvm () at /cache/build/default-amdci5-4/julialang/julia-master/src/codegen.cpp:8585 #14 0xf75318a8 in jl_init_codegen_impl () at /cache/build/default-amdci5-4/julialang/julia-master/src/codegen.cpp:8648 #15 0xf7a51a52 in jl_restore_system_image_from_stream (f=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2131 #16 0xf7a55c03 in ijl_restore_system_image_data (buf=0xe859c1c0 <jl_system_image_data> "8'\031\003", len=125161105) at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2184 #17 0xf7a55cf9 in jl_load_sysimg_so () at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:424 #18 ijl_restore_system_image (fname=0x80a0900 "/build/bk_download/julia-d78fdad601/lib/julia/sys.so") at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2157 #19 0xf7a3bdfc in _finish_julia_init (rel=rel@entry=JL_IMAGE_JULIA_HOME, ct=<optimized out>, ptls=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/init.c:741 #20 0xf7a3c8ac in julia_init (rel=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/init.c:728 #21 0xf7a7f61d in jl_repl_entrypoint (argc=<optimized out>, argv=0xffffddf4) at /cache/build/default-amdci5-4/julialang/julia-master/src/jlapi.c:705 #22 0x080490a7 in main (argc=3, argv=0xffffddf4) at /cache/build/default-amdci5-4/julialang/julia-master/cli/loader_exe.c:59 ``` To prevent this, we simply avoid calling `jl_errorf` this early in the process, punting the problem to a later PR that can update guard conditions within `jl_error*`. (cherry picked from commit 21ab24e)
merges into JuliaLang#42768
Add a wrapper around jl_gc_pool_alloc to allow us to instrument it.
managed_malloc
taken care of in #20Before
*: instrumented; red: currently missed
graphviz
After
*: instrumented; red: currently missed
graphviz
```graphviz digraph G { node [shape=box];generated_code [label=""];
new_array;
jl_gc_alloc_string [label="jl_gc_alloc_string *"];
jl_gc_alloc [label="jl_gc_alloc *"];
jl_managed_malloc [label="jl_managed_malloc *"]; // #20
jl_gc_pool_alloc;
jl_gc_pool_alloc_outer [fillcolor=lightgreen style=filled label="jl_gc_pool_alloc_outer *"];
jl_gc_big_alloc;
generated_code -> jl_gc_alloc_string;
generated_code -> new_array;
generated_code -> jl_gc_alloc;
generated_code -> jl_gc_big_alloc [color=red];
generated_code -> jl_gc_pool_alloc_outer;
jl_gc_pool_alloc_outer -> jl_gc_pool_alloc;
jl_gc_alloc -> jl_gc_pool_alloc;
jl_gc_alloc -> jl_gc_big_alloc;
new_array -> jl_gc_alloc;
new_array -> jl_managed_malloc;
jl_gc_alloc_string -> jl_gc_pool_alloc;
jl_gc_alloc_string -> jl_gc_big_alloc;
}