-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Profile: Faster data dict lookup #43805
Profile: Faster data dict lookup #43805
Conversation
cebc2ac
to
2cf728a
Compare
# Base.update_stackframes_callback[] | ||
stn = map(x->(x, 1), st) | ||
try Base.invokelatest(Base.update_stackframes_callback[], stn) catch end | ||
dict[UInt64(ip)] = map(first, stn) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I always forget if Dict
is threadsafe. I don't think it is?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought it was if it's preallocated like it is here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At least experimentally they seem to be?
julia> dict = Dict{Int,Vector{Int}}();
julia> foreach(i -> dict[i] = Int[], 1:10_000)
julia> @sync for i in keys(dict)
Threads.@spawn begin
dict[i] = collect(1:i)
end
end
julia> for i in keys(dict)
sum(dict[i]) == sum(1:i) || error()
end
julia>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yes as long as we are not reallocating...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, this is a very bad idea
f999b7a
to
2cf728a
Compare
2cf728a
to
28396c9
Compare
try Base.invokelatest(Base.update_stackframes_callback[], stn) catch end | ||
dict[UInt64(ip)] = map(first, stn) | ||
# we don't want metadata here as we're just looking up ips | ||
unique_data_itr = Iterators.unique(has_meta(data) ? strip_meta(data) : data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since unique
uses hashing which is much of what Dicts do, why is this a speed improvement?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That just facilitates identifying the unique ips before going off to parallelize. It's the parallel part that helps most
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
…aLang#43816)" This reverts commit e6fa3ec.
The ip -> stackframe lookup that happens when viewing profiling data dominates TTFPP and TTSPP
Master
which is basically all spent in
getdict!
withinretrieve
This PR
so
And when julia only has one thread it takes the same amount of time as master.