[llvm] Add support for llvm IR atomicrmw fminimum/fmaximum instructions #136759
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tructions This patch adds support for atomicrmw `fmaximum` and `fminimum` instructions. These mirror the `llvm.maximum.*` and `llvm.minimum.*` instructions, but are atomic and use IEEE754 2019 handling for NaNs, which is different to `fmax` and `fmin`. See https://llvm.org/docs/LangRef.html#llvm-minimum-intrinsic for more details. Future changes will allow this LLVM IR to be lowered to specialised assembler instruction on suitable targets, such as AArch64.
llvmbot
added
backend:AArch64
llvm:globalisel
llvm:instcombine
llvm:support
llvm:SelectionDAG
|
@llvm/pr-subscribers-backend-amdgpu @llvm/pr-subscribers-backend-aarch64 Author: Jonathan Thackray (jthackray) ChangesThis patch adds support for atomicrmw These mirror the Future changes will allow this LLVM IR to be lowered to specialised assembler instruction on suitable targets, such as AArch64. Patch is 33.46 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/136759.diff 35 Files Affected:
diff --git a/llvm/docs/GlobalISel/GenericOpcode.rst b/llvm/docs/GlobalISel/GenericOpcode.rst
index 5291b42d80870..987d19e2f6ce1 100644
--- a/llvm/docs/GlobalISel/GenericOpcode.rst
+++ b/llvm/docs/GlobalISel/GenericOpcode.rst
@@ -922,7 +922,8 @@ operands.
G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX,
G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD,
G_ATOMICRMW_FSUB, G_ATOMICRMW_FMAX,
- G_ATOMICRMW_FMIN, G_ATOMICRMW_UINC_WRAP,
+ G_ATOMICRMW_FMIN, G_ATOMICRMW_FMAXIMUM,
+ G_ATOMICRMW_FMINIMUM, G_ATOMICRMW_UINC_WRAP,
G_ATOMICRMW_UDEC_WRAP, G_ATOMICRMW_USUB_COND,
G_ATOMICRMW_USUB_SAT
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 9e78f317bd9ea..d43625a0759cc 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -11598,6 +11598,8 @@ operation. The operation must be one of the following keywords:
- fsub
- fmax
- fmin
+- fmaximum
+- fminimum
- uinc_wrap
- udec_wrap
- usub_cond
@@ -11607,7 +11609,7 @@ For most of these operations, the type of '<value>' must be an integer
type whose bit width is a power of two greater than or equal to eight
and less than or equal to a target-specific size limit. For xchg, this
may also be a floating point or a pointer type with the same size constraints
-as integers. For fadd/fsub/fmax/fmin, this must be a floating-point
+as integers. For fadd/fsub/fmax/fmin/fmaximum/fminimum, this must be a floating-point
or fixed vector of floating-point type. The type of the '``<pointer>``'
operand must be a pointer to that type. If the ``atomicrmw`` is marked
as ``volatile``, then the optimizer is not allowed to modify the
@@ -11648,8 +11650,10 @@ operation argument:
- umin: ``*ptr = *ptr < val ? *ptr : val`` (using an unsigned comparison)
- fadd: ``*ptr = *ptr + val`` (using floating point arithmetic)
- fsub: ``*ptr = *ptr - val`` (using floating point arithmetic)
-- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*`` intrinsic)
-- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*`` intrinsic)
+- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*` intrinsic)
+- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*` intrinsic)
+- fmaximum: ``*ptr = maximum(*ptr, val)`` (match the `llvm.maximum.*` intrinsic)
+- fminimum: ``*ptr = minimum(*ptr, val)`` (match the `llvm.minimum.*` intrinsic)
- uinc_wrap: ``*ptr = (*ptr u>= val) ? 0 : (*ptr + 1)`` (increment value with wraparound to zero when incremented above input value)
- udec_wrap: ``*ptr = ((*ptr == 0) || (*ptr u> val)) ? val : (*ptr - 1)`` (decrement with wraparound to input value when decremented below zero).
- usub_cond: ``*ptr = (*ptr u>= val) ? *ptr - val : *ptr`` (subtract only if no unsigned overflow).
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index cc4fd38944ff6..9739a033f1260 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -72,6 +72,9 @@ Changes to LLVM infrastructure
themselves (i.e., the `TargetIntrinsicInfo` class).
* Fix Microsoft demangling of string literals to be stricter
(#GH129970))
+* Added the support for ``fmaximum`` and ``fminimum`` in ``atomicrmw`` instruction. The
+ comparison is expected to match the behavior of ``llvm.maximum.*`` and
+ ``llvm.minimum.*`` respectively.
Changes to building LLVM
------------------------
diff --git a/llvm/include/llvm-c/Core.h b/llvm/include/llvm-c/Core.h
index 871e5eb8457d7..bdd6490e7dd7d 100644
--- a/llvm/include/llvm-c/Core.h
+++ b/llvm/include/llvm-c/Core.h
@@ -386,6 +386,12 @@ typedef enum {
LLVMAtomicRMWBinOpFMin, /**< Sets the value if it's smaller than the
original using an floating point comparison and
return the old one */
+ LLVMAtomicRMWBinOpFMaximum, /**< Sets the value if it's greater than the
+ original using an floating point comparison and
+ return the old one */
+ LLVMAtomicRMWBinOpFMinimum, /**< Sets the value if it's smaller than the
+ original using an floating point comparison and
+ return the old one */
LLVMAtomicRMWBinOpUIncWrap, /**< Increments the value, wrapping back to zero
when incremented above input value */
LLVMAtomicRMWBinOpUDecWrap, /**< Decrements the value, wrapping back to
diff --git a/llvm/include/llvm/AsmParser/LLToken.h b/llvm/include/llvm/AsmParser/LLToken.h
index a8f9c71781701..c7e4bdf3ff811 100644
--- a/llvm/include/llvm/AsmParser/LLToken.h
+++ b/llvm/include/llvm/AsmParser/LLToken.h
@@ -276,6 +276,8 @@ enum Kind {
kw_umin,
kw_fmax,
kw_fmin,
+ kw_fmaximum,
+ kw_fminimum,
kw_uinc_wrap,
kw_udec_wrap,
kw_usub_cond,
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 92b6e68d9d0a7..9317e6aa793de 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -504,7 +504,9 @@ enum RMWOperations {
RMW_UINC_WRAP = 15,
RMW_UDEC_WRAP = 16,
RMW_USUB_COND = 17,
- RMW_USUB_SAT = 18
+ RMW_USUB_SAT = 18,
+ RMW_FMAXIMUM = 19,
+ RMW_FMINIMUM = 20,
};
/// OverflowingBinaryOperatorOptionalFlags - Flags for serializing
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
index f9dcbeb370bef..d5f621a74b6e7 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
@@ -1663,6 +1663,42 @@ class MachineIRBuilder {
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
MachineMemOperand &MMO);
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMAXIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point maximum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMINIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point minimum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
/// Build and insert `OldValRes<def> = G_ATOMICRMW_USUB_COND Addr, Val, MMO`.
///
/// Atomically replace the value at \p Addr with the original value minus \p
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index ad8a95a353b56..80ef32aff62ae 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -1367,6 +1367,8 @@ enum NodeType {
ATOMIC_LOAD_FSUB,
ATOMIC_LOAD_FMAX,
ATOMIC_LOAD_FMIN,
+ ATOMIC_LOAD_FMAXIMUM,
+ ATOMIC_LOAD_FMINIMUM,
ATOMIC_LOAD_UINC_WRAP,
ATOMIC_LOAD_UDEC_WRAP,
ATOMIC_LOAD_USUB_COND,
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
index b279ca90be9e8..568c5e8155314 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
@@ -1517,6 +1517,8 @@ class MemSDNode : public SDNode {
case ISD::ATOMIC_LOAD_FSUB:
case ISD::ATOMIC_LOAD_FMAX:
case ISD::ATOMIC_LOAD_FMIN:
+ case ISD::ATOMIC_LOAD_FMAXIMUM:
+ case ISD::ATOMIC_LOAD_FMINIMUM:
case ISD::ATOMIC_LOAD_UINC_WRAP:
case ISD::ATOMIC_LOAD_UDEC_WRAP:
case ISD::ATOMIC_LOAD_USUB_COND:
@@ -1603,6 +1605,8 @@ class AtomicSDNode : public MemSDNode {
N->getOpcode() == ISD::ATOMIC_LOAD_FSUB ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMAX ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMIN ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMAXIMUM ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMINIMUM ||
N->getOpcode() == ISD::ATOMIC_LOAD_UINC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_UDEC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/include/llvm/IR/Instructions.h b/llvm/include/llvm/IR/Instructions.h
index 95f0ef875fc07..c164f76eb335b 100644
--- a/llvm/include/llvm/IR/Instructions.h
+++ b/llvm/include/llvm/IR/Instructions.h
@@ -751,6 +751,14 @@ class AtomicRMWInst : public Instruction {
/// \p minnum matches the behavior of \p llvm.minnum.*.
FMin,
+ /// *p = maximum(old, v)
+ /// \p maximum matches the behavior of \p llvm.maximum.*.
+ FMaximum,
+
+ /// *p = minimum(old, v)
+ /// \p minimum matches the behavior of \p llvm.minimum.*.
+ FMinimum,
+
/// Increment one up to a maximum value.
/// *p = (old u>= v) ? 0 : (old + 1)
UIncWrap,
@@ -812,6 +820,8 @@ class AtomicRMWInst : public Instruction {
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
return true;
default:
return false;
diff --git a/llvm/include/llvm/Support/TargetOpcodes.def b/llvm/include/llvm/Support/TargetOpcodes.def
index 5ef3707b81fe9..771c318da817d 100644
--- a/llvm/include/llvm/Support/TargetOpcodes.def
+++ b/llvm/include/llvm/Support/TargetOpcodes.def
@@ -426,6 +426,8 @@ HANDLE_TARGET_OPCODE(G_ATOMICRMW_FADD)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FSUB)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAX)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMIN)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAXIMUM)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMINIMUM)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UINC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UDEC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_USUB_COND)
diff --git a/llvm/include/llvm/Target/GenericOpcodes.td b/llvm/include/llvm/Target/GenericOpcodes.td
index e134bab61bf63..c041c3cdfca5b 100644
--- a/llvm/include/llvm/Target/GenericOpcodes.td
+++ b/llvm/include/llvm/Target/GenericOpcodes.td
@@ -1351,6 +1351,8 @@ def G_ATOMICRMW_FADD : G_ATOMICRMW_OP;
def G_ATOMICRMW_FSUB : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMAX : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMIN : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMAXIMUM : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMINIMUM : G_ATOMICRMW_OP;
def G_ATOMICRMW_UINC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_UDEC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_USUB_COND : G_ATOMICRMW_OP;
diff --git a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
index c8c0eeb57099a..cb55a6fb112c5 100644
--- a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
+++ b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
@@ -266,6 +266,8 @@ def : GINodeEquiv<G_ATOMICRMW_FADD, atomic_load_fadd>;
def : GINodeEquiv<G_ATOMICRMW_FSUB, atomic_load_fsub>;
def : GINodeEquiv<G_ATOMICRMW_FMAX, atomic_load_fmax>;
def : GINodeEquiv<G_ATOMICRMW_FMIN, atomic_load_fmin>;
+def : GINodeEquiv<G_ATOMICRMW_FMAXIMUM, atomic_load_fmaximum>;
+def : GINodeEquiv<G_ATOMICRMW_FMINIMUM, atomic_load_fminimum>;
def : GINodeEquiv<G_ATOMICRMW_UINC_WRAP, atomic_load_uinc_wrap>;
def : GINodeEquiv<G_ATOMICRMW_UDEC_WRAP, atomic_load_udec_wrap>;
def : GINodeEquiv<G_ATOMICRMW_USUB_COND, atomic_load_usub_cond>;
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 9c241b6c4df0f..dda4f1e91f702 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -772,6 +772,10 @@ def atomic_load_fmax : SDNode<"ISD::ATOMIC_LOAD_FMAX", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_fmin : SDNode<"ISD::ATOMIC_LOAD_FMIN", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fmaximum : SDNode<"ISD::ATOMIC_LOAD_FMAXIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fminimum : SDNode<"ISD::ATOMIC_LOAD_FMINIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_uinc_wrap : SDNode<"ISD::ATOMIC_LOAD_UINC_WRAP", SDTAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_udec_wrap : SDNode<"ISD::ATOMIC_LOAD_UDEC_WRAP", SDTAtomic2,
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index 4d25b12c9ab06..ce813e1d7b1c4 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -749,6 +749,8 @@ lltok::Kind LLLexer::LexIdentifier() {
KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
KEYWORD(umin); KEYWORD(fmax); KEYWORD(fmin);
+ KEYWORD(fmaximum);
+ KEYWORD(fminimum);
KEYWORD(uinc_wrap);
KEYWORD(udec_wrap);
KEYWORD(usub_cond);
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index af0422f09bc4f..d2ea21cbd328c 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -8626,6 +8626,14 @@ int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
Operation = AtomicRMWInst::FMin;
IsFP = true;
break;
+ case lltok::kw_fmaximum:
+ Operation = AtomicRMWInst::FMaximum;
+ IsFP = true;
+ break;
+ case lltok::kw_fminimum:
+ Operation = AtomicRMWInst::FMinimum;
+ IsFP = true;
+ break;
}
Lex.Lex(); // Eat the operation.
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index b152adfbd93d8..d283650e7500f 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -1356,6 +1356,10 @@ static AtomicRMWInst::BinOp getDecodedRMWOperation(unsigned Val) {
case bitc::RMW_FSUB: return AtomicRMWInst::FSub;
case bitc::RMW_FMAX: return AtomicRMWInst::FMax;
case bitc::RMW_FMIN: return AtomicRMWInst::FMin;
+ case bitc::RMW_FMAXIMUM:
+ return AtomicRMWInst::FMaximum;
+ case bitc::RMW_FMINIMUM:
+ return AtomicRMWInst::FMinimum;
case bitc::RMW_UINC_WRAP:
return AtomicRMWInst::UIncWrap;
case bitc::RMW_UDEC_WRAP:
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 73bed85c65b3d..0893cf7aec8c0 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -678,6 +678,10 @@ static unsigned getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::FSub: return bitc::RMW_FSUB;
case AtomicRMWInst::FMax: return bitc::RMW_FMAX;
case AtomicRMWInst::FMin: return bitc::RMW_FMIN;
+ case AtomicRMWInst::FMaximum:
+ return bitc::RMW_FMAXIMUM;
+ case AtomicRMWInst::FMinimum:
+ return bitc::RMW_FMINIMUM;
case AtomicRMWInst::UIncWrap:
return bitc::RMW_UINC_WRAP;
case AtomicRMWInst::UDecWrap:
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index a3e9700fa3089..c376de877ac7d 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -931,6 +931,8 @@ static Value *performMaskedAtomicOp(AtomicRMWInst::BinOp Op,
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMin:
case AtomicRMWInst::FMax:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::UIncWrap:
case AtomicRMWInst::UDecWrap:
case AtomicRMWInst::USubCond:
@@ -1819,6 +1821,8 @@ static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::UMin:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::FAdd:
case AtomicRMWInst::FSub:
case AtomicRMWInst::UIncWrap:
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index 931e4fe19e69a..c9fda0cc3f924 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -3435,6 +3435,12 @@ bool IRTranslator::translateAtomicRMW(const User &U,
case AtomicRMWInst::FMin:
Opcode = TargetOpcode::G_ATOMICRMW_FMIN;
break;
+ case AtomicRMWInst::FMaximum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
Opcode = TargetOpcode::G_ATOMICRMW_UINC_WRAP;
break;
diff --git a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
index 359677027f52f..055963d10948f 100644
--- a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
@@ -1155,6 +1155,22 @@ MachineIRBuilder::buildAtomicRMWFMin(const DstOp &OldValRes, const SrcOp &Addr,
MMO);
}
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMAXIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMINIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
MachineInstrBuilder
MachineIRBuilder::buildFence(unsigned Ordering, unsigned Scope) {
return buildInstr(TargetOpcode::G_FENCE)
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index a20651b65c2a9..da94e1181bd2b 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -9039,6 +9039,8 @@ SDValue SelectionDAG::getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT,
Opcode == ISD::ATOMIC_LOAD_UMAX || Opcode == ISD::ATOMIC_LOAD_FADD ||
Opcode == ISD::ATOMIC_LOAD_FSUB || Opcode == ISD::ATOMIC_LOAD_FMAX ||
Opcode == ISD::ATOMIC_LOAD_FMIN ||
+ Opcode == ISD::ATOMIC_LOAD_FMINIMUM ||
+ Opcode == ISD::ATOMIC_LOAD_FMAXIMUM ||
Opcode == ISD::ATOMIC_LOAD_UINC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_UDEC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 66bd78af2939c..1d7ed8a87e567 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5081,6 +5081,12 @@ void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
case AtomicRMWInst::FSub: NT = ISD::ATOMIC_LOAD_FSUB; break;
case AtomicRMWInst::FMax: NT = ISD::ATOMIC_LOAD_FMAX; break;
case AtomicRMWInst::FMin: NT = ISD::ATOMIC_LOAD_FMIN; break;
+ case AtomicRMWInst::FMaximum:
+...
[truncated]
|
|
@llvm/pr-subscribers-llvm-globalisel Author: Jonathan Thackray (jthackray) ChangesThis patch adds support for atomicrmw These mirror the Future changes will allow this LLVM IR to be lowered to specialised assembler instruction on suitable targets, such as AArch64. Patch is 33.46 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/136759.diff 35 Files Affected:
diff --git a/llvm/docs/GlobalISel/GenericOpcode.rst b/llvm/docs/GlobalISel/GenericOpcode.rst
index 5291b42d80870..987d19e2f6ce1 100644
--- a/llvm/docs/GlobalISel/GenericOpcode.rst
+++ b/llvm/docs/GlobalISel/GenericOpcode.rst
@@ -922,7 +922,8 @@ operands.
G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX,
G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD,
G_ATOMICRMW_FSUB, G_ATOMICRMW_FMAX,
- G_ATOMICRMW_FMIN, G_ATOMICRMW_UINC_WRAP,
+ G_ATOMICRMW_FMIN, G_ATOMICRMW_FMAXIMUM,
+ G_ATOMICRMW_FMINIMUM, G_ATOMICRMW_UINC_WRAP,
G_ATOMICRMW_UDEC_WRAP, G_ATOMICRMW_USUB_COND,
G_ATOMICRMW_USUB_SAT
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 9e78f317bd9ea..d43625a0759cc 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -11598,6 +11598,8 @@ operation. The operation must be one of the following keywords:
- fsub
- fmax
- fmin
+- fmaximum
+- fminimum
- uinc_wrap
- udec_wrap
- usub_cond
@@ -11607,7 +11609,7 @@ For most of these operations, the type of '<value>' must be an integer
type whose bit width is a power of two greater than or equal to eight
and less than or equal to a target-specific size limit. For xchg, this
may also be a floating point or a pointer type with the same size constraints
-as integers. For fadd/fsub/fmax/fmin, this must be a floating-point
+as integers. For fadd/fsub/fmax/fmin/fmaximum/fminimum, this must be a floating-point
or fixed vector of floating-point type. The type of the '``<pointer>``'
operand must be a pointer to that type. If the ``atomicrmw`` is marked
as ``volatile``, then the optimizer is not allowed to modify the
@@ -11648,8 +11650,10 @@ operation argument:
- umin: ``*ptr = *ptr < val ? *ptr : val`` (using an unsigned comparison)
- fadd: ``*ptr = *ptr + val`` (using floating point arithmetic)
- fsub: ``*ptr = *ptr - val`` (using floating point arithmetic)
-- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*`` intrinsic)
-- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*`` intrinsic)
+- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*` intrinsic)
+- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*` intrinsic)
+- fmaximum: ``*ptr = maximum(*ptr, val)`` (match the `llvm.maximum.*` intrinsic)
+- fminimum: ``*ptr = minimum(*ptr, val)`` (match the `llvm.minimum.*` intrinsic)
- uinc_wrap: ``*ptr = (*ptr u>= val) ? 0 : (*ptr + 1)`` (increment value with wraparound to zero when incremented above input value)
- udec_wrap: ``*ptr = ((*ptr == 0) || (*ptr u> val)) ? val : (*ptr - 1)`` (decrement with wraparound to input value when decremented below zero).
- usub_cond: ``*ptr = (*ptr u>= val) ? *ptr - val : *ptr`` (subtract only if no unsigned overflow).
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index cc4fd38944ff6..9739a033f1260 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -72,6 +72,9 @@ Changes to LLVM infrastructure
themselves (i.e., the `TargetIntrinsicInfo` class).
* Fix Microsoft demangling of string literals to be stricter
(#GH129970))
+* Added the support for ``fmaximum`` and ``fminimum`` in ``atomicrmw`` instruction. The
+ comparison is expected to match the behavior of ``llvm.maximum.*`` and
+ ``llvm.minimum.*`` respectively.
Changes to building LLVM
------------------------
diff --git a/llvm/include/llvm-c/Core.h b/llvm/include/llvm-c/Core.h
index 871e5eb8457d7..bdd6490e7dd7d 100644
--- a/llvm/include/llvm-c/Core.h
+++ b/llvm/include/llvm-c/Core.h
@@ -386,6 +386,12 @@ typedef enum {
LLVMAtomicRMWBinOpFMin, /**< Sets the value if it's smaller than the
original using an floating point comparison and
return the old one */
+ LLVMAtomicRMWBinOpFMaximum, /**< Sets the value if it's greater than the
+ original using an floating point comparison and
+ return the old one */
+ LLVMAtomicRMWBinOpFMinimum, /**< Sets the value if it's smaller than the
+ original using an floating point comparison and
+ return the old one */
LLVMAtomicRMWBinOpUIncWrap, /**< Increments the value, wrapping back to zero
when incremented above input value */
LLVMAtomicRMWBinOpUDecWrap, /**< Decrements the value, wrapping back to
diff --git a/llvm/include/llvm/AsmParser/LLToken.h b/llvm/include/llvm/AsmParser/LLToken.h
index a8f9c71781701..c7e4bdf3ff811 100644
--- a/llvm/include/llvm/AsmParser/LLToken.h
+++ b/llvm/include/llvm/AsmParser/LLToken.h
@@ -276,6 +276,8 @@ enum Kind {
kw_umin,
kw_fmax,
kw_fmin,
+ kw_fmaximum,
+ kw_fminimum,
kw_uinc_wrap,
kw_udec_wrap,
kw_usub_cond,
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 92b6e68d9d0a7..9317e6aa793de 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -504,7 +504,9 @@ enum RMWOperations {
RMW_UINC_WRAP = 15,
RMW_UDEC_WRAP = 16,
RMW_USUB_COND = 17,
- RMW_USUB_SAT = 18
+ RMW_USUB_SAT = 18,
+ RMW_FMAXIMUM = 19,
+ RMW_FMINIMUM = 20,
};
/// OverflowingBinaryOperatorOptionalFlags - Flags for serializing
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
index f9dcbeb370bef..d5f621a74b6e7 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
@@ -1663,6 +1663,42 @@ class MachineIRBuilder {
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
MachineMemOperand &MMO);
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMAXIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point maximum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMINIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point minimum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
/// Build and insert `OldValRes<def> = G_ATOMICRMW_USUB_COND Addr, Val, MMO`.
///
/// Atomically replace the value at \p Addr with the original value minus \p
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index ad8a95a353b56..80ef32aff62ae 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -1367,6 +1367,8 @@ enum NodeType {
ATOMIC_LOAD_FSUB,
ATOMIC_LOAD_FMAX,
ATOMIC_LOAD_FMIN,
+ ATOMIC_LOAD_FMAXIMUM,
+ ATOMIC_LOAD_FMINIMUM,
ATOMIC_LOAD_UINC_WRAP,
ATOMIC_LOAD_UDEC_WRAP,
ATOMIC_LOAD_USUB_COND,
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
index b279ca90be9e8..568c5e8155314 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
@@ -1517,6 +1517,8 @@ class MemSDNode : public SDNode {
case ISD::ATOMIC_LOAD_FSUB:
case ISD::ATOMIC_LOAD_FMAX:
case ISD::ATOMIC_LOAD_FMIN:
+ case ISD::ATOMIC_LOAD_FMAXIMUM:
+ case ISD::ATOMIC_LOAD_FMINIMUM:
case ISD::ATOMIC_LOAD_UINC_WRAP:
case ISD::ATOMIC_LOAD_UDEC_WRAP:
case ISD::ATOMIC_LOAD_USUB_COND:
@@ -1603,6 +1605,8 @@ class AtomicSDNode : public MemSDNode {
N->getOpcode() == ISD::ATOMIC_LOAD_FSUB ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMAX ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMIN ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMAXIMUM ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMINIMUM ||
N->getOpcode() == ISD::ATOMIC_LOAD_UINC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_UDEC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/include/llvm/IR/Instructions.h b/llvm/include/llvm/IR/Instructions.h
index 95f0ef875fc07..c164f76eb335b 100644
--- a/llvm/include/llvm/IR/Instructions.h
+++ b/llvm/include/llvm/IR/Instructions.h
@@ -751,6 +751,14 @@ class AtomicRMWInst : public Instruction {
/// \p minnum matches the behavior of \p llvm.minnum.*.
FMin,
+ /// *p = maximum(old, v)
+ /// \p maximum matches the behavior of \p llvm.maximum.*.
+ FMaximum,
+
+ /// *p = minimum(old, v)
+ /// \p minimum matches the behavior of \p llvm.minimum.*.
+ FMinimum,
+
/// Increment one up to a maximum value.
/// *p = (old u>= v) ? 0 : (old + 1)
UIncWrap,
@@ -812,6 +820,8 @@ class AtomicRMWInst : public Instruction {
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
return true;
default:
return false;
diff --git a/llvm/include/llvm/Support/TargetOpcodes.def b/llvm/include/llvm/Support/TargetOpcodes.def
index 5ef3707b81fe9..771c318da817d 100644
--- a/llvm/include/llvm/Support/TargetOpcodes.def
+++ b/llvm/include/llvm/Support/TargetOpcodes.def
@@ -426,6 +426,8 @@ HANDLE_TARGET_OPCODE(G_ATOMICRMW_FADD)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FSUB)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAX)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMIN)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAXIMUM)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMINIMUM)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UINC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UDEC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_USUB_COND)
diff --git a/llvm/include/llvm/Target/GenericOpcodes.td b/llvm/include/llvm/Target/GenericOpcodes.td
index e134bab61bf63..c041c3cdfca5b 100644
--- a/llvm/include/llvm/Target/GenericOpcodes.td
+++ b/llvm/include/llvm/Target/GenericOpcodes.td
@@ -1351,6 +1351,8 @@ def G_ATOMICRMW_FADD : G_ATOMICRMW_OP;
def G_ATOMICRMW_FSUB : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMAX : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMIN : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMAXIMUM : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMINIMUM : G_ATOMICRMW_OP;
def G_ATOMICRMW_UINC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_UDEC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_USUB_COND : G_ATOMICRMW_OP;
diff --git a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
index c8c0eeb57099a..cb55a6fb112c5 100644
--- a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
+++ b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
@@ -266,6 +266,8 @@ def : GINodeEquiv<G_ATOMICRMW_FADD, atomic_load_fadd>;
def : GINodeEquiv<G_ATOMICRMW_FSUB, atomic_load_fsub>;
def : GINodeEquiv<G_ATOMICRMW_FMAX, atomic_load_fmax>;
def : GINodeEquiv<G_ATOMICRMW_FMIN, atomic_load_fmin>;
+def : GINodeEquiv<G_ATOMICRMW_FMAXIMUM, atomic_load_fmaximum>;
+def : GINodeEquiv<G_ATOMICRMW_FMINIMUM, atomic_load_fminimum>;
def : GINodeEquiv<G_ATOMICRMW_UINC_WRAP, atomic_load_uinc_wrap>;
def : GINodeEquiv<G_ATOMICRMW_UDEC_WRAP, atomic_load_udec_wrap>;
def : GINodeEquiv<G_ATOMICRMW_USUB_COND, atomic_load_usub_cond>;
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 9c241b6c4df0f..dda4f1e91f702 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -772,6 +772,10 @@ def atomic_load_fmax : SDNode<"ISD::ATOMIC_LOAD_FMAX", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_fmin : SDNode<"ISD::ATOMIC_LOAD_FMIN", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fmaximum : SDNode<"ISD::ATOMIC_LOAD_FMAXIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fminimum : SDNode<"ISD::ATOMIC_LOAD_FMINIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_uinc_wrap : SDNode<"ISD::ATOMIC_LOAD_UINC_WRAP", SDTAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_udec_wrap : SDNode<"ISD::ATOMIC_LOAD_UDEC_WRAP", SDTAtomic2,
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index 4d25b12c9ab06..ce813e1d7b1c4 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -749,6 +749,8 @@ lltok::Kind LLLexer::LexIdentifier() {
KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
KEYWORD(umin); KEYWORD(fmax); KEYWORD(fmin);
+ KEYWORD(fmaximum);
+ KEYWORD(fminimum);
KEYWORD(uinc_wrap);
KEYWORD(udec_wrap);
KEYWORD(usub_cond);
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index af0422f09bc4f..d2ea21cbd328c 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -8626,6 +8626,14 @@ int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
Operation = AtomicRMWInst::FMin;
IsFP = true;
break;
+ case lltok::kw_fmaximum:
+ Operation = AtomicRMWInst::FMaximum;
+ IsFP = true;
+ break;
+ case lltok::kw_fminimum:
+ Operation = AtomicRMWInst::FMinimum;
+ IsFP = true;
+ break;
}
Lex.Lex(); // Eat the operation.
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index b152adfbd93d8..d283650e7500f 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -1356,6 +1356,10 @@ static AtomicRMWInst::BinOp getDecodedRMWOperation(unsigned Val) {
case bitc::RMW_FSUB: return AtomicRMWInst::FSub;
case bitc::RMW_FMAX: return AtomicRMWInst::FMax;
case bitc::RMW_FMIN: return AtomicRMWInst::FMin;
+ case bitc::RMW_FMAXIMUM:
+ return AtomicRMWInst::FMaximum;
+ case bitc::RMW_FMINIMUM:
+ return AtomicRMWInst::FMinimum;
case bitc::RMW_UINC_WRAP:
return AtomicRMWInst::UIncWrap;
case bitc::RMW_UDEC_WRAP:
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 73bed85c65b3d..0893cf7aec8c0 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -678,6 +678,10 @@ static unsigned getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::FSub: return bitc::RMW_FSUB;
case AtomicRMWInst::FMax: return bitc::RMW_FMAX;
case AtomicRMWInst::FMin: return bitc::RMW_FMIN;
+ case AtomicRMWInst::FMaximum:
+ return bitc::RMW_FMAXIMUM;
+ case AtomicRMWInst::FMinimum:
+ return bitc::RMW_FMINIMUM;
case AtomicRMWInst::UIncWrap:
return bitc::RMW_UINC_WRAP;
case AtomicRMWInst::UDecWrap:
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index a3e9700fa3089..c376de877ac7d 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -931,6 +931,8 @@ static Value *performMaskedAtomicOp(AtomicRMWInst::BinOp Op,
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMin:
case AtomicRMWInst::FMax:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::UIncWrap:
case AtomicRMWInst::UDecWrap:
case AtomicRMWInst::USubCond:
@@ -1819,6 +1821,8 @@ static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::UMin:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::FAdd:
case AtomicRMWInst::FSub:
case AtomicRMWInst::UIncWrap:
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index 931e4fe19e69a..c9fda0cc3f924 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -3435,6 +3435,12 @@ bool IRTranslator::translateAtomicRMW(const User &U,
case AtomicRMWInst::FMin:
Opcode = TargetOpcode::G_ATOMICRMW_FMIN;
break;
+ case AtomicRMWInst::FMaximum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
Opcode = TargetOpcode::G_ATOMICRMW_UINC_WRAP;
break;
diff --git a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
index 359677027f52f..055963d10948f 100644
--- a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
@@ -1155,6 +1155,22 @@ MachineIRBuilder::buildAtomicRMWFMin(const DstOp &OldValRes, const SrcOp &Addr,
MMO);
}
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMAXIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMINIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
MachineInstrBuilder
MachineIRBuilder::buildFence(unsigned Ordering, unsigned Scope) {
return buildInstr(TargetOpcode::G_FENCE)
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index a20651b65c2a9..da94e1181bd2b 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -9039,6 +9039,8 @@ SDValue SelectionDAG::getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT,
Opcode == ISD::ATOMIC_LOAD_UMAX || Opcode == ISD::ATOMIC_LOAD_FADD ||
Opcode == ISD::ATOMIC_LOAD_FSUB || Opcode == ISD::ATOMIC_LOAD_FMAX ||
Opcode == ISD::ATOMIC_LOAD_FMIN ||
+ Opcode == ISD::ATOMIC_LOAD_FMINIMUM ||
+ Opcode == ISD::ATOMIC_LOAD_FMAXIMUM ||
Opcode == ISD::ATOMIC_LOAD_UINC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_UDEC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 66bd78af2939c..1d7ed8a87e567 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5081,6 +5081,12 @@ void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
case AtomicRMWInst::FSub: NT = ISD::ATOMIC_LOAD_FSUB; break;
case AtomicRMWInst::FMax: NT = ISD::ATOMIC_LOAD_FMAX; break;
case AtomicRMWInst::FMin: NT = ISD::ATOMIC_LOAD_FMIN; break;
+ case AtomicRMWInst::FMaximum:
+...
[truncated]
|
|
@llvm/pr-subscribers-flang-openmp Author: Jonathan Thackray (jthackray) ChangesThis patch adds support for atomicrmw These mirror the Future changes will allow this LLVM IR to be lowered to specialised assembler instruction on suitable targets, such as AArch64. Patch is 33.46 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/136759.diff 35 Files Affected:
diff --git a/llvm/docs/GlobalISel/GenericOpcode.rst b/llvm/docs/GlobalISel/GenericOpcode.rst
index 5291b42d80870..987d19e2f6ce1 100644
--- a/llvm/docs/GlobalISel/GenericOpcode.rst
+++ b/llvm/docs/GlobalISel/GenericOpcode.rst
@@ -922,7 +922,8 @@ operands.
G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX,
G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD,
G_ATOMICRMW_FSUB, G_ATOMICRMW_FMAX,
- G_ATOMICRMW_FMIN, G_ATOMICRMW_UINC_WRAP,
+ G_ATOMICRMW_FMIN, G_ATOMICRMW_FMAXIMUM,
+ G_ATOMICRMW_FMINIMUM, G_ATOMICRMW_UINC_WRAP,
G_ATOMICRMW_UDEC_WRAP, G_ATOMICRMW_USUB_COND,
G_ATOMICRMW_USUB_SAT
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 9e78f317bd9ea..d43625a0759cc 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -11598,6 +11598,8 @@ operation. The operation must be one of the following keywords:
- fsub
- fmax
- fmin
+- fmaximum
+- fminimum
- uinc_wrap
- udec_wrap
- usub_cond
@@ -11607,7 +11609,7 @@ For most of these operations, the type of '<value>' must be an integer
type whose bit width is a power of two greater than or equal to eight
and less than or equal to a target-specific size limit. For xchg, this
may also be a floating point or a pointer type with the same size constraints
-as integers. For fadd/fsub/fmax/fmin, this must be a floating-point
+as integers. For fadd/fsub/fmax/fmin/fmaximum/fminimum, this must be a floating-point
or fixed vector of floating-point type. The type of the '``<pointer>``'
operand must be a pointer to that type. If the ``atomicrmw`` is marked
as ``volatile``, then the optimizer is not allowed to modify the
@@ -11648,8 +11650,10 @@ operation argument:
- umin: ``*ptr = *ptr < val ? *ptr : val`` (using an unsigned comparison)
- fadd: ``*ptr = *ptr + val`` (using floating point arithmetic)
- fsub: ``*ptr = *ptr - val`` (using floating point arithmetic)
-- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*`` intrinsic)
-- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*`` intrinsic)
+- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*` intrinsic)
+- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*` intrinsic)
+- fmaximum: ``*ptr = maximum(*ptr, val)`` (match the `llvm.maximum.*` intrinsic)
+- fminimum: ``*ptr = minimum(*ptr, val)`` (match the `llvm.minimum.*` intrinsic)
- uinc_wrap: ``*ptr = (*ptr u>= val) ? 0 : (*ptr + 1)`` (increment value with wraparound to zero when incremented above input value)
- udec_wrap: ``*ptr = ((*ptr == 0) || (*ptr u> val)) ? val : (*ptr - 1)`` (decrement with wraparound to input value when decremented below zero).
- usub_cond: ``*ptr = (*ptr u>= val) ? *ptr - val : *ptr`` (subtract only if no unsigned overflow).
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index cc4fd38944ff6..9739a033f1260 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -72,6 +72,9 @@ Changes to LLVM infrastructure
themselves (i.e., the `TargetIntrinsicInfo` class).
* Fix Microsoft demangling of string literals to be stricter
(#GH129970))
+* Added the support for ``fmaximum`` and ``fminimum`` in ``atomicrmw`` instruction. The
+ comparison is expected to match the behavior of ``llvm.maximum.*`` and
+ ``llvm.minimum.*`` respectively.
Changes to building LLVM
------------------------
diff --git a/llvm/include/llvm-c/Core.h b/llvm/include/llvm-c/Core.h
index 871e5eb8457d7..bdd6490e7dd7d 100644
--- a/llvm/include/llvm-c/Core.h
+++ b/llvm/include/llvm-c/Core.h
@@ -386,6 +386,12 @@ typedef enum {
LLVMAtomicRMWBinOpFMin, /**< Sets the value if it's smaller than the
original using an floating point comparison and
return the old one */
+ LLVMAtomicRMWBinOpFMaximum, /**< Sets the value if it's greater than the
+ original using an floating point comparison and
+ return the old one */
+ LLVMAtomicRMWBinOpFMinimum, /**< Sets the value if it's smaller than the
+ original using an floating point comparison and
+ return the old one */
LLVMAtomicRMWBinOpUIncWrap, /**< Increments the value, wrapping back to zero
when incremented above input value */
LLVMAtomicRMWBinOpUDecWrap, /**< Decrements the value, wrapping back to
diff --git a/llvm/include/llvm/AsmParser/LLToken.h b/llvm/include/llvm/AsmParser/LLToken.h
index a8f9c71781701..c7e4bdf3ff811 100644
--- a/llvm/include/llvm/AsmParser/LLToken.h
+++ b/llvm/include/llvm/AsmParser/LLToken.h
@@ -276,6 +276,8 @@ enum Kind {
kw_umin,
kw_fmax,
kw_fmin,
+ kw_fmaximum,
+ kw_fminimum,
kw_uinc_wrap,
kw_udec_wrap,
kw_usub_cond,
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 92b6e68d9d0a7..9317e6aa793de 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -504,7 +504,9 @@ enum RMWOperations {
RMW_UINC_WRAP = 15,
RMW_UDEC_WRAP = 16,
RMW_USUB_COND = 17,
- RMW_USUB_SAT = 18
+ RMW_USUB_SAT = 18,
+ RMW_FMAXIMUM = 19,
+ RMW_FMINIMUM = 20,
};
/// OverflowingBinaryOperatorOptionalFlags - Flags for serializing
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
index f9dcbeb370bef..d5f621a74b6e7 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
@@ -1663,6 +1663,42 @@ class MachineIRBuilder {
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
MachineMemOperand &MMO);
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMAXIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point maximum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMINIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point minimum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
/// Build and insert `OldValRes<def> = G_ATOMICRMW_USUB_COND Addr, Val, MMO`.
///
/// Atomically replace the value at \p Addr with the original value minus \p
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index ad8a95a353b56..80ef32aff62ae 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -1367,6 +1367,8 @@ enum NodeType {
ATOMIC_LOAD_FSUB,
ATOMIC_LOAD_FMAX,
ATOMIC_LOAD_FMIN,
+ ATOMIC_LOAD_FMAXIMUM,
+ ATOMIC_LOAD_FMINIMUM,
ATOMIC_LOAD_UINC_WRAP,
ATOMIC_LOAD_UDEC_WRAP,
ATOMIC_LOAD_USUB_COND,
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
index b279ca90be9e8..568c5e8155314 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
@@ -1517,6 +1517,8 @@ class MemSDNode : public SDNode {
case ISD::ATOMIC_LOAD_FSUB:
case ISD::ATOMIC_LOAD_FMAX:
case ISD::ATOMIC_LOAD_FMIN:
+ case ISD::ATOMIC_LOAD_FMAXIMUM:
+ case ISD::ATOMIC_LOAD_FMINIMUM:
case ISD::ATOMIC_LOAD_UINC_WRAP:
case ISD::ATOMIC_LOAD_UDEC_WRAP:
case ISD::ATOMIC_LOAD_USUB_COND:
@@ -1603,6 +1605,8 @@ class AtomicSDNode : public MemSDNode {
N->getOpcode() == ISD::ATOMIC_LOAD_FSUB ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMAX ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMIN ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMAXIMUM ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMINIMUM ||
N->getOpcode() == ISD::ATOMIC_LOAD_UINC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_UDEC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/include/llvm/IR/Instructions.h b/llvm/include/llvm/IR/Instructions.h
index 95f0ef875fc07..c164f76eb335b 100644
--- a/llvm/include/llvm/IR/Instructions.h
+++ b/llvm/include/llvm/IR/Instructions.h
@@ -751,6 +751,14 @@ class AtomicRMWInst : public Instruction {
/// \p minnum matches the behavior of \p llvm.minnum.*.
FMin,
+ /// *p = maximum(old, v)
+ /// \p maximum matches the behavior of \p llvm.maximum.*.
+ FMaximum,
+
+ /// *p = minimum(old, v)
+ /// \p minimum matches the behavior of \p llvm.minimum.*.
+ FMinimum,
+
/// Increment one up to a maximum value.
/// *p = (old u>= v) ? 0 : (old + 1)
UIncWrap,
@@ -812,6 +820,8 @@ class AtomicRMWInst : public Instruction {
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
return true;
default:
return false;
diff --git a/llvm/include/llvm/Support/TargetOpcodes.def b/llvm/include/llvm/Support/TargetOpcodes.def
index 5ef3707b81fe9..771c318da817d 100644
--- a/llvm/include/llvm/Support/TargetOpcodes.def
+++ b/llvm/include/llvm/Support/TargetOpcodes.def
@@ -426,6 +426,8 @@ HANDLE_TARGET_OPCODE(G_ATOMICRMW_FADD)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FSUB)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAX)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMIN)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAXIMUM)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMINIMUM)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UINC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UDEC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_USUB_COND)
diff --git a/llvm/include/llvm/Target/GenericOpcodes.td b/llvm/include/llvm/Target/GenericOpcodes.td
index e134bab61bf63..c041c3cdfca5b 100644
--- a/llvm/include/llvm/Target/GenericOpcodes.td
+++ b/llvm/include/llvm/Target/GenericOpcodes.td
@@ -1351,6 +1351,8 @@ def G_ATOMICRMW_FADD : G_ATOMICRMW_OP;
def G_ATOMICRMW_FSUB : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMAX : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMIN : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMAXIMUM : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMINIMUM : G_ATOMICRMW_OP;
def G_ATOMICRMW_UINC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_UDEC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_USUB_COND : G_ATOMICRMW_OP;
diff --git a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
index c8c0eeb57099a..cb55a6fb112c5 100644
--- a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
+++ b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
@@ -266,6 +266,8 @@ def : GINodeEquiv<G_ATOMICRMW_FADD, atomic_load_fadd>;
def : GINodeEquiv<G_ATOMICRMW_FSUB, atomic_load_fsub>;
def : GINodeEquiv<G_ATOMICRMW_FMAX, atomic_load_fmax>;
def : GINodeEquiv<G_ATOMICRMW_FMIN, atomic_load_fmin>;
+def : GINodeEquiv<G_ATOMICRMW_FMAXIMUM, atomic_load_fmaximum>;
+def : GINodeEquiv<G_ATOMICRMW_FMINIMUM, atomic_load_fminimum>;
def : GINodeEquiv<G_ATOMICRMW_UINC_WRAP, atomic_load_uinc_wrap>;
def : GINodeEquiv<G_ATOMICRMW_UDEC_WRAP, atomic_load_udec_wrap>;
def : GINodeEquiv<G_ATOMICRMW_USUB_COND, atomic_load_usub_cond>;
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 9c241b6c4df0f..dda4f1e91f702 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -772,6 +772,10 @@ def atomic_load_fmax : SDNode<"ISD::ATOMIC_LOAD_FMAX", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_fmin : SDNode<"ISD::ATOMIC_LOAD_FMIN", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fmaximum : SDNode<"ISD::ATOMIC_LOAD_FMAXIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fminimum : SDNode<"ISD::ATOMIC_LOAD_FMINIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_uinc_wrap : SDNode<"ISD::ATOMIC_LOAD_UINC_WRAP", SDTAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_udec_wrap : SDNode<"ISD::ATOMIC_LOAD_UDEC_WRAP", SDTAtomic2,
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index 4d25b12c9ab06..ce813e1d7b1c4 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -749,6 +749,8 @@ lltok::Kind LLLexer::LexIdentifier() {
KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
KEYWORD(umin); KEYWORD(fmax); KEYWORD(fmin);
+ KEYWORD(fmaximum);
+ KEYWORD(fminimum);
KEYWORD(uinc_wrap);
KEYWORD(udec_wrap);
KEYWORD(usub_cond);
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index af0422f09bc4f..d2ea21cbd328c 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -8626,6 +8626,14 @@ int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
Operation = AtomicRMWInst::FMin;
IsFP = true;
break;
+ case lltok::kw_fmaximum:
+ Operation = AtomicRMWInst::FMaximum;
+ IsFP = true;
+ break;
+ case lltok::kw_fminimum:
+ Operation = AtomicRMWInst::FMinimum;
+ IsFP = true;
+ break;
}
Lex.Lex(); // Eat the operation.
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index b152adfbd93d8..d283650e7500f 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -1356,6 +1356,10 @@ static AtomicRMWInst::BinOp getDecodedRMWOperation(unsigned Val) {
case bitc::RMW_FSUB: return AtomicRMWInst::FSub;
case bitc::RMW_FMAX: return AtomicRMWInst::FMax;
case bitc::RMW_FMIN: return AtomicRMWInst::FMin;
+ case bitc::RMW_FMAXIMUM:
+ return AtomicRMWInst::FMaximum;
+ case bitc::RMW_FMINIMUM:
+ return AtomicRMWInst::FMinimum;
case bitc::RMW_UINC_WRAP:
return AtomicRMWInst::UIncWrap;
case bitc::RMW_UDEC_WRAP:
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 73bed85c65b3d..0893cf7aec8c0 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -678,6 +678,10 @@ static unsigned getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::FSub: return bitc::RMW_FSUB;
case AtomicRMWInst::FMax: return bitc::RMW_FMAX;
case AtomicRMWInst::FMin: return bitc::RMW_FMIN;
+ case AtomicRMWInst::FMaximum:
+ return bitc::RMW_FMAXIMUM;
+ case AtomicRMWInst::FMinimum:
+ return bitc::RMW_FMINIMUM;
case AtomicRMWInst::UIncWrap:
return bitc::RMW_UINC_WRAP;
case AtomicRMWInst::UDecWrap:
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index a3e9700fa3089..c376de877ac7d 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -931,6 +931,8 @@ static Value *performMaskedAtomicOp(AtomicRMWInst::BinOp Op,
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMin:
case AtomicRMWInst::FMax:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::UIncWrap:
case AtomicRMWInst::UDecWrap:
case AtomicRMWInst::USubCond:
@@ -1819,6 +1821,8 @@ static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::UMin:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::FAdd:
case AtomicRMWInst::FSub:
case AtomicRMWInst::UIncWrap:
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index 931e4fe19e69a..c9fda0cc3f924 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -3435,6 +3435,12 @@ bool IRTranslator::translateAtomicRMW(const User &U,
case AtomicRMWInst::FMin:
Opcode = TargetOpcode::G_ATOMICRMW_FMIN;
break;
+ case AtomicRMWInst::FMaximum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
Opcode = TargetOpcode::G_ATOMICRMW_UINC_WRAP;
break;
diff --git a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
index 359677027f52f..055963d10948f 100644
--- a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
@@ -1155,6 +1155,22 @@ MachineIRBuilder::buildAtomicRMWFMin(const DstOp &OldValRes, const SrcOp &Addr,
MMO);
}
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMAXIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMINIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
MachineInstrBuilder
MachineIRBuilder::buildFence(unsigned Ordering, unsigned Scope) {
return buildInstr(TargetOpcode::G_FENCE)
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index a20651b65c2a9..da94e1181bd2b 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -9039,6 +9039,8 @@ SDValue SelectionDAG::getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT,
Opcode == ISD::ATOMIC_LOAD_UMAX || Opcode == ISD::ATOMIC_LOAD_FADD ||
Opcode == ISD::ATOMIC_LOAD_FSUB || Opcode == ISD::ATOMIC_LOAD_FMAX ||
Opcode == ISD::ATOMIC_LOAD_FMIN ||
+ Opcode == ISD::ATOMIC_LOAD_FMINIMUM ||
+ Opcode == ISD::ATOMIC_LOAD_FMAXIMUM ||
Opcode == ISD::ATOMIC_LOAD_UINC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_UDEC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 66bd78af2939c..1d7ed8a87e567 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5081,6 +5081,12 @@ void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
case AtomicRMWInst::FSub: NT = ISD::ATOMIC_LOAD_FSUB; break;
case AtomicRMWInst::FMax: NT = ISD::ATOMIC_LOAD_FMAX; break;
case AtomicRMWInst::FMin: NT = ISD::ATOMIC_LOAD_FMIN; break;
+ case AtomicRMWInst::FMaximum:
+...
[truncated]
|
nikic
reviewed
Apr 23, 2025
llvm/include/llvm-c/Core.h
Outdated
| return the old one */ | ||
| LLVMAtomicRMWBinOpFMinimum, /**< Sets the value if it's smaller than the | ||
| original using an floating point comparison and | ||
| return the old one */ |
There was a problem hiding this comment.
This needs to be added to the end of the enum to prevent a C ABI break.
There was a problem hiding this comment.
Thanks, now moved in latest patch.
| return !CF->isNegative() && CF->isInfinity(); | ||
| case AtomicRMWInst::FMinimum: | ||
| // minimum(x, -inf) -> +inf | ||
| return CF->isNegative() && CF->isInfinity(); |
There was a problem hiding this comment.
I don't think these are correct? Inf saturates maxnum, NaN saturates maximum.
There was a problem hiding this comment.
Thanks. I've removed these, as @arsenm says below they should be done in a later patch.
arsenm
reviewed
Apr 23, 2025
| @@ -68,6 +68,12 @@ bool isSaturating(AtomicRMWInst& RMWI) { | |||
| case AtomicRMWInst::FMin: | |||
| // minnum(x, -inf) -> +inf | |||
| return CF->isNegative() && CF->isInfinity(); | |||
| case AtomicRMWInst::FMaximum: | |||
There was a problem hiding this comment.
This is an extra optimization that should be done in a follow up, not the initial base patch
There was a problem hiding this comment.
Thanks, removed for now.
Comment on lines
+934
to
+935
| case AtomicRMWInst::FMaximum: | ||
| case AtomicRMWInst::FMinimum: |
There was a problem hiding this comment.
Missing test coverage for this, should copy the fmin/fmax tests in test/Transforms/AtomicExpand. This is not the same as LowerAtomic which you did add the test for
There was a problem hiding this comment.
Thanks, now added. Actually, fmin/fmax were missing, so I've added those to llvm/test/Transforms/AtomicExpand/AArch64/atomicrmw-fp.ll. Should I add fminimum/fmaximum for all target architectures?
| ; CHECK-NEXT: store | ||
| ret float %j | ||
| ; CHECK: ret float [[INST]] | ||
| } |
There was a problem hiding this comment.
Test a vector case too, e.g <2 x half>
There was a problem hiding this comment.
Thanks for the suggestion, done.
jthackray
requested review from
CarolineConcatto,
kmclaughlin-arm,
Lukacma and
rgwott
arsenm
reviewed
Apr 23, 2025
| ; | ||
| %res = atomicrmw fmaximum ptr %ptr, double %value seq_cst | ||
| ret double %res | ||
| } |
There was a problem hiding this comment.
test a vector case like <2 x half> and bfloat
There was a problem hiding this comment.
Thanks, now added.
| ret float %res | ||
| } | ||
|
|
||
| define float @atomicrmw_fmaximum_float(ptr %ptr, float %value) { |
There was a problem hiding this comment.
Thanks, now added.
arsenm
approved these changes
Apr 24, 2025
Comment on lines
+1768
to
+1777
| case AtomicRMWInst::FMaximum: { | ||
| report_fatal_error("atomic floating point fmaximum not supported for " | ||
| "buffer resources and should've been expanded away"); | ||
| break; | ||
| } | ||
| case AtomicRMWInst::FMinimum: { | ||
| report_fatal_error("atomic floating point fminimum not supported for " | ||
| "buffer resources and should've been expanded away"); | ||
| break; | ||
| } |
There was a problem hiding this comment.
we don't really need a unique error message for each of these cases
Comment on lines
+8779
to
+8780
| case AtomicRMWInst::FMaximum: | ||
| case AtomicRMWInst::FMinimum: |
There was a problem hiding this comment.
This code really should be deleted. I don't know why OMPIRBuilder has anything like this, but it's also a straight copy of the lower atomic utility
There was a problem hiding this comment.
Thanks. I added it to avoid a compiler warning about unhandled case.
Thanks. Yes, I've got a couple of follow-on patches planned. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds support for atomicrmw
fmaximumandfminimuminstructions.These mirror the
llvm.maximum.*andllvm.minimum.*instructions, but are atomic and use IEEE754 2019 handling for NaNs, which is different tofmaxandfmin. See https://llvm.org/docs/LangRef.html#llvm-minimum-intrinsic for more details.Future changes will allow this LLVM IR to be lowered to specialised assembler instruction on suitable targets, such as AArch64.