Monitor cache is never hit in JIT generated code

[Akira1Saitoh]

For monitorenter/monitorexit, JIT generates the code for searching a monitor from the cache in J9VMThread if the object does not have a lockword.
Monitors are cached in objectMonitorLookupCache field, which is an array of j9objectmonitor_t.
When running in compressed refs mode in mixed refs build, all existing platform seem to generate incorrect code to access to the entry of this array. I think it does not cause any functional problem but the cache is never hit. I suppose it is an oversight when OpenJ9 switched to mixed refs builds.

In mixed refs build, both OMR_GC_COMPRESSED_POINTERS and OMR_GC_FULL_POINTERS are defined as in #8847 (comment).
Thus, objectMonitorLookupCache in J9VMThread is an array of UDATA as j9objectmonitor_t is UDATA when OMR_GC_FULL_POINTERS is defined.

openj9/runtime/oti/j9accessbarrier.h

Lines 39 to 46 in c332ee2

	/*
	* Define type used to represent the lock in the 'monitor' slot of an object's header
	*/
	#if defined(OMR_GC_FULL_POINTERS)
	typedef UDATA j9objectmonitor_t;
	#else /* OMR_GC_FULL_POINTERS */
	typedef U_32 j9objectmonitor_t;
	#endif /* OMR_GC_FULL_POINTERS */

openj9/runtime/oti/j9nonbuilder.h

Line 4986 in c332ee2

j9objectmonitor_t objectMonitorLookupCache[J9VM_OBJECT_MONITOR_CACHE_SIZE];

However, p/x/z codegen calculate the offset of the entry in the array assuming the element size is TR::Compiler->om.sizeofReferenceField(), which is not the same as UDATA when running in compressed refs mode. Thus, the calculated offset is wrong and the cache is never hit (except for the first slot whose offset is 0).

openj9/runtime/compiler/p/codegen/J9TreeEvaluator.cpp

Line 7447 in c332ee2

generateShiftLeftImmediateLong(cg, node, lookupOffsetReg, lookupOffsetReg, trailingZeroes((int32_t) TR::Compiler->om.sizeofReferenceField()));

openj9/runtime/compiler/x/codegen/J9TreeEvaluator.cpp

Line 965 in c332ee2

generateRegImmInstruction(TR::InstOpCode::SHLRegImm1(), node, lookupOffsetReg, trailingZeroes(TR::Compiler->om.sizeofReferenceField()), cg);

openj9/runtime/compiler/z/codegen/J9TreeEvaluator.cpp

Line 8445 in c332ee2

generateRSInstruction(cg, TR::InstOpCode::SLLG, node, lookupOffsetReg, lookupOffsetReg, trailingZeroes((int32_t) TR::Compiler->om.sizeofReferenceField()));

I noticed this problem when implementing #9240 for AArch64. When debugging with gdb, I observed the generated code failing to find the existing entry in the cache array if using TR::Compiler->om.sizeofReferenceField() as the element size.

FYI @0xdaryl @knn-k