-
Notifications
You must be signed in to change notification settings - Fork 6k
8346123: [REDO] NMT should not use ThreadCritical #22745
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This reverts commit c3df050.
👋 Welcome back roberttoyonaga! A progress list of the required criteria for merging this PR into |
@roberttoyonaga This change now passes all automated pre-integration checks. ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details. After integration, the commit message for the final commit will be:
You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed. At the time when this comment was updated there had been 264 new commits pushed to the
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details. As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@kimbarrett, @dholmes-ora, @coleenp, @tstuefe) but any other Committer may sponsor as well. ➡️ To flag this PR as ready for integration with the above commit message, type |
@roberttoyonaga The following label will be automatically applied to this pull request:
When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command. |
/label serviceability |
@roberttoyonaga |
21dd479
to
e817cf1
Compare
Webrevs
|
src/hotspot/share/nmt/memTracker.hpp
Outdated
@@ -269,6 +276,15 @@ class MemTracker : AllStatic { | |||
// and return true; false if not found. | |||
static bool print_containing_region(const void* p, outputStream* out); | |||
|
|||
// Same as MutexLocker but can be used during VM init while single threaded and before mutexes are ready or current thread has been assigned. | |||
// Performs no action during VM init. | |||
class NmtVirtualMemoryLocker: public ConditionalMutexLocker { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should not derive from ConditionalMutexLocker - it's not designed to be a base class.
Either use has-a relationship, or just derive directly from MutexLockerImpl, since we don't need
the assert in ConditionalMutexLocker anyway. And I'm surprised there isn't a constructor taking
the current thread, like all the other locker variants.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should be able to use a ConditionalMutexLocker directly to handle this situation - it is one of the usecases that caused CML to be introduced.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CML could perhaps be used has-a (though that might be messy), but should not
be used is-a. One of the basic rules for base classes is to have either a
public virtual destructor (don't do that here) or a non-public non-virtual
destructor, to prevent accidental slicing. MutexLockerImpl has that - it was
designed to be a base class. CML doesn't - it was not designed to be a base
class. (There is a common (though not universal) opinion that classes should
either be base or concrete classes, not both.) I know we violate this rule all
over the place; that doesn't mean we should do so here.
But really, there's no benefit to using CML here. Deriving from
MutexLockerImpl is about the same number of characters, and seems clearer to
me because it's more direct.
class NmtVirtualMemoryLocker : public MutexLockerImpl {
public:
NmtVirtualMemoryLocker()
: MutexLockerImpl(_done_bootstrap ? NmtVirtualMemory_lock : nullptr,
Mutex::_no_safepoint_check_flag)
{}
}
Related, I just noticed that MutexLockerImpl is copyable, but shouldn't be.
(I remember a discussion a long time ago about whether StackObj should be
noncopyable, which foundered on a combination of some significant number of
existing uses that violated that restriction, along with disagreement about
the meaning of deriving from StackObj. So we're not getting noncopyable from
there.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MutexLockerImpl
was not intended for external subclassing, but as the internal base class for MutexLocker
, and ConditionalMutexLocker
. CML was provided for exactly this kind of situation: conditionally locking the mutex.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok I see. I will stop using ConditionalMutexLocker
as a base class. Instead, I'll just get rid of NmtVirtualMemoryLocker
and use ConditionalMutexLocker
directly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding use of MutexLockerImpl
, perhaps, though I see nothing in the JBS
issue or the PR that says that. If it's the name that drives that position
then I think it can easily be argued that it's misnamed, and should be
something like MutexLockerBase. It is nicely designed for use as a base
class, with protected ctors/dtor, making it a convient basis for lockers for
specific use cases. But see my other response about how to has-a use CML (and
MutexLocker) for that purpose.
The GHA test
I don't think this is related to the changes in this PR. The same problem seems to have been reported a week ago here: JDK-8345895 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The direct use of CML is functionally correct but requires leaking knowledge to all the use-sites that should not need to know about the actual condition for locking. This really needs to be encapsulated which means we need to be able to subclass CML.
@kbarrett what is needed to make ConditionalMutexLocker
subclassable?
os::print_memory_mappings((char*)start, bytes, tty); | ||
fileStream fs(stdout); | ||
os::print_memory_mappings((char*)start, bytes, &fs); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why was this change made? tty could be stdout or stderr depending on VM flag settings.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change was decided on in the original PR after it was discovered that calling os::print_memory_mappings from the windows implementation of os::pd_release_memory
causes a rank conflict between tty_lock
and NmtVirtualMemory_lock
. This is getting called from os::release_memory
after we've already acquired the lock for NMT.
Original discussion here #20852 (comment).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay ... still not sure this shouldn't be dynamically determining whether stdout or stderr is the right target.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay ... still not sure this shouldn't be dynamically determining whether stdout or stderr is the right target.
We should really have a not-blocking variant of tty
since er use tty for things like debug output when being deep down the stack. In most cases, printing to a small stringStream and then printing the stream content in one go is what I do to get uninterrupted printing. Works well.
@roberttoyonaga pragmatic proposal would be to just remove this debug output, so the whole ASSERT section. I think I added this way back when we had problems investigating Windows reservations writing over each other, but that problem has been fixed and this assert has not been firing for ages.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok I see. I've changed it now to use defaultStream::output_stream()
which should choose either stderr or stdout based on the VM flag -XX:+DisplayVMOutputToStderr
. After the change, running the test release_bad_ranges
on Windows still passes (when using tty, this test fails due to lock rank conflict).
@@ -526,6 +526,9 @@ jint Threads::create_vm(JavaVMInitArgs* args, bool* canTryAgain) { | |||
// Initialize global data structures and create system classes in heap | |||
vm_init_globals(); | |||
|
|||
// Once mutexes are ready, we can use NMT locks. | |||
MemTracker::set_done_bootstrap(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be done after the main thread has attached and set the current thread, otherwise if we hit any NMT code that needs the lock it would crash trying to acquire it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok I've moved it until after initialize_thread_current
.
Co-authored-by: David Holmes <62092539+dholmes-ora@users.noreply.github.com>
@roberttoyonaga this pull request can not be integrated into git checkout 8346123
git fetch https://git.openjdk.org/jdk.git master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push |
I agree that directly using ConditionalMutexLocker isn't pleasant here, and
You don't change ConditionalMutexLocker at all. Just change how it's used: class NmtVirtualMemoryLocker : StackObj { public: MutexLocker can be has-a used similarly for convenience lockers that don't Note that making MutexLockerImpl noncopyable (as it should be, but that ought Note that I was previously wrong when I suggested it might be messy to has-a And not that it really matters, but deriving from StackObj with a CML member |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry if I'm repeating other comments, but it looks like a good change but could be a bit less repetition. Thanks for removing this instance of ThreadCritical.
MUTEX_DEFN(DCmdFactory_lock , PaddedMutex , nosafepoint); | ||
MUTEX_DEFN(NMTQuery_lock , PaddedMutex , safepoint); | ||
MUTEX_DEFN(NMTCompilationCostHistory_lock , PaddedMutex , nosafepoint); | ||
MUTEX_DEFN(NmtVirtualMemory_lock , PaddedMutex , service-4); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this service-4? Does it depend on being a rank lower than another lock? If so, this and the SharedDecoder_lock should be declared below as MUTEX_DEFL and call out that lock.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it must have a lower rank than "G1Mapper_lock" which is in G1RegionsSmallerThanCommitSizeMapper
, not in mutexLocker.cpp
. I've moved SharedDecoder_lock
down below as MUTEX_DEFL
and I've left a comment beside MUTEX_DEFN(NmtVirtualMemory_lock, PaddedMutex , service-4);
to explain the rank. Would this be enough?
src/hotspot/share/runtime/os.cpp
Outdated
@@ -2310,7 +2310,7 @@ char* os::map_memory(int fd, const char* file_name, size_t file_offset, | |||
bool os::unmap_memory(char *addr, size_t bytes) { | |||
bool result; | |||
if (MemTracker::enabled()) { | |||
ThreadCritical tc; | |||
ConditionalMutexLocker cml(NmtVirtualMemory_lock, MemTracker::is_done_bootstrap(), Mutex::_no_safepoint_check_flag); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This pattern is all over. Can you create a class in nmtSomeHeader like:
class NMTLocker {
ConditionalMutexLocker cml;
NMTLocker();
};
in cpp file:
NmtLocker() : _cml(NmtVirtualMemory_lock, MemTracker::is_done_bootstrap(), Mutex::_no_safepoint_check_flag) {}
Or something like that, rather than this be repeated everywhere. I only skimmed the other comments, so maybe David and Kim said the same thing.
Oh yes, now I see, sorry I did repeat Kim's suggestion. |
Doh! @kimbarrett don't know what I was thinking. Thanks |
…t for SharedDecoder_lock
Yes, good point. For JavaThreads: For NonJavaThreads: Maybe a workaround is to delay |
You can't attach to a VM until a certain point deep into initialization has been reached.
Yes it is a problem that the created thread adds itself to the list. So a simple solution would be to just use |
Do we still think using hotspot Mutexes is a good goal? As opposed to Roberts's original attempt of using a non-asserting platform mutex? What do people think? I myself am not so sure anymore. We get deadlock prevention but also get brittle at initialization. |
Being brittle at initialization is unfortunate, and there is also the point @kimbarrett mentions about this only working with VM threads. I am not exactly sure what the best choice is here, but I do think deadlock detection is very nice since the NMT locking is quite broad in scope. I believe, in most cases, we lock around both the NMT accounting and the actual memory operation. |
Hi @dholmes-ora, I think it would still be problem that NonJavaThreads (such as Both JavaThreads and NonJavaThreads can use NMT before being detectable by |
Sorry yes of course that is how we started on this current round - the WatcherThread is created before the main thread is "attached". I need to rewind a few steps and see exactly what it is we need to be initialized before this lock can be used ... |
Before Hmmm I'm not sure anymore if introducing |
I agree But I'd prefer to see |
Okay I've gone back to the old approach of using a marker. I've replaced the name Yes, I agree that this marker should live in NMT instead of Mutex or Threads because we are not only dependent on mutexes being ready or threads being ready, we're dependent on both. And this dependency is specific to NMT (at least it seems for now) since NMT is used so early. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nothing further from me. Thanks for your patience and perseverance on this one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good.
But after the last attempt I would feel safer if we could run this through some more extensive testing. I ran a bunch of tests myself, and will ping the SAP guys. @dholmes-ora could you schedule some tests at Oracle? IIRC the ZGC tests were the ones that broke the most.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm glad that this could use Mutex in the end and not use the PlatforMutex. If all else fails though, the way this is structured could use PlatformMutex as the underlying implementation and that would be okay. Removing the use of ThreadCritical is the motivating part of this change.
I've downloaded the patch and am running tier1-7 on it.
I reran our internal tests tier1-7 on this change and they all passed. |
Thank you everyone for your feedback and time reviewing this! |
@roberttoyonaga |
@roberttoyonaga Hold off, I still wait for the SAP people to report back with their tests. |
Okay, SAP gave green light too. /sponsor |
Going to push as commit 3804082.
Your commit was automatically rebased without conflicts. |
@tstuefe @roberttoyonaga Pushed as commit 3804082. 💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored. |
This is a redo of JDK-8304824 which was backed out by JDK-8343726 due to problems documented in JDK-8343244.
The problem was that
NmtVirtualMemoryLocker
was not locking when the current thread is not attached by checkingThread::current_or_null_safe() != nullptr
. This is necessary during VM init, but should not be allowed afterward. NMT may be used inattach_current_thread
before the current thread is set. The lock was not being acquired in that case, which intermittently caused NMT accounting to become corrupted, triggering various assertions when future NMT operations are done. To fix this, I've adopted Thomas' suggestion to reverse the order ofin
attach_current_thread
. This allowsNmtVirtualMemoryLocker
to be locked after current thread is set.To allow for
NmtVirtualMemoryLocker
to be used during VM init, I've replaced theConditionalMutexLocker
checkThread::current_or_null_safe() != nullptr
with a new flag:_done_bootstrap
. This flag prevents the lock from being used during VM init, at which point we are single threaded anyway. This avoids errors due to Hotspot mutexes and current thread not yet being ready.I also added new asserts in
virtualMemoryTracker.cpp
to catch future bugs like this where the lock is not held when it should be. I updated the appropriate VMT tests to also lock (there were a few cases where locking was being bypassed) so they can pass the new asserts.I also removed the unused
_query_lock
variable inMemTracker
.Testing:
java/lang/Thread/jni/AttachCurrentThread/AttachTest.java
. The test consistently passes with the new changes in this PR.Progress
Issue
Reviewers
Reviewing
Using
git
Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/22745/head:pull/22745
$ git checkout pull/22745
Update a local copy of the PR:
$ git checkout pull/22745
$ git pull https://git.openjdk.org/jdk.git pull/22745/head
Using Skara CLI tools
Checkout this PR locally:
$ git pr checkout 22745
View PR using the GUI difftool:
$ git pr show -t 22745
Using diff file
Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/22745.diff
Using Webrev
Link to Webrev Comment