Prevent compilation requests to be re-issued when code cache is exhausted

[mpirvu]

When the code cache becomes full, the JIT uses
getCompilationInfo()->getPersistentInfo()->setDisableFurtherCompilation(true)
to prevent future compilation requests from being queued. The rejection happens in

compileOnSeparateThread()
   {
...
   if ((getNumCompThreadsActive() == 0
#if defined(J9VM_OPT_CRIU_SUPPORT)
        || getCRRuntime()->shouldSuspendThreadsForCheckpoint()
#endif
        || getPersistentInfo()->getDisableFurtherCompilation())
       && !details.isJitDumpMethod())
      {
      bool shouldReturn = true;

      // Fail the compilation, unless this is on the queue (maybe even in progress)
      if (!requestExistsInCompilationQueue(details, fe))
         {
         startPC = compilationEnd(vmThread, details, _jitConfig, 0, oldStartPC);
         }
...
   }

Typically, once a method fails compilation, the JIT calls compilationEnd() which will call jitMethodFailedTranslation(vmThread, method), which writes J9_JIT_NEVER_TRANSLATE into j9method->extra so that the interpreter doesn't continue to send this method for compilation. The code looks like this:

   else if (!oldStartPC)
      {
      // Tell the VM that a non-compiled method failed translation
      //
      if (vmThread && entry && !entry->isOutOfProcessCompReq())
         jitMethodFailedTranslation(vmThread, method);

Note that calling jitMethodFailedTranslation(vmThread, method); is gated by a test on entry. When a compilation is rejected early due to the getPersistentInfo()->getDisableFurtherCompilation() test, the entry does not exist because the compilation request has not yet been added to the compilation queue. Thus, j9method->extra is not set to J9_JIT_NEVER_TRANSLATE and the interpreter will send the method over and over again to the JIT compiler.

This issue proposes to eliminate the test on entry in compilationEnd() , provided it is safe to do so.

[github-actions]

Issue Number: 20448
Status: Open
Recommended Components: comp:vm, comp:gc, comp:test
Recommended Assignees: babsingh, thallium, tajila