[LV] Add support for cmp reductions with decreasing IVs. #140451
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
llvmbot
added
vectorizers
llvm:analysis
|
@llvm/pr-subscribers-llvm-analysis @llvm/pr-subscribers-llvm-transforms Author: Florian Hahn (fhahn) ChangesSimilar to FindLastIV, add FindFirstIV to support select (icmp(), x, y) reductions where one of x or y is a decreasing induction. This is done via a new recurrence kind FindFirstIV, which selects the first value from the reduction vector using umax instead of the last value (FindLastIV). It uses unsigned max as sentinel value. Alternatively we could consolidate FindFirstIV/FindLastIV in a single FindIV kind, and add a field indicating if whether we want to select the first or last value. Patch is 70.46 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/140451.diff 12 Files Affected:
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 140edff13a67f..64127df9ccbd0 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -57,11 +57,14 @@ enum class RecurKind {
IFindLastIV, ///< FindLast reduction with select(icmp(),x,y) where one of
///< (x,y) is increasing loop induction, and both x and y are
///< integer type.
- FFindLastIV ///< FindLast reduction with select(fcmp(),x,y) where one of (x,y)
- ///< is increasing loop induction, and both x and y are integer
- ///< type.
- // TODO: Any_of and FindLast reduction need not be restricted to integer type
- // only.
+ FFindLastIV, ///< FindLast reduction with select(fcmp(),x,y) where one of (x,y)
+ ///< is increasing loop induction, and both x and y are integer
+ ///< type.
+ FindFirstIV /// FindLast reduction with select(icmp(),x,y) where one of
+ ///< (x,y) is a decreasing loop induction, and both x and y are
+ ///< integer type.
+ // TODO: Any_of, FindLast and FindFirst reduction need not be restricted to
+ // integer type only.
};
/// The RecurrenceDescriptor is used to identify recurrences variables in a
@@ -163,12 +166,13 @@ class RecurrenceDescriptor {
/// Returns a struct describing whether the instruction is either a
/// Select(ICmp(A, B), X, Y), or
/// Select(FCmp(A, B), X, Y)
- /// where one of (X, Y) is an increasing loop induction variable, and the
- /// other is a PHI value.
+ /// where one of (X, Y) is an increasing (FindLast) or decreasing (FindFirst)
+ /// loop induction variable, and the other is a PHI value.
// TODO: Support non-monotonic variable. FindLast does not need be restricted
// to increasing loop induction variables.
- static InstDesc isFindLastIVPattern(Loop *TheLoop, PHINode *OrigPhi,
- Instruction *I, ScalarEvolution &SE);
+ static InstDesc isFindIVPattern(RecurKind Kind, Loop *TheLoop,
+ PHINode *OrigPhi, Instruction *I,
+ ScalarEvolution &SE);
/// Returns a struct describing if the instruction is a
/// Select(FCmp(X, Y), (Z = X op PHINode), PHINode) instruction pattern.
@@ -262,17 +266,29 @@ class RecurrenceDescriptor {
return Kind == RecurKind::IFindLastIV || Kind == RecurKind::FFindLastIV;
}
+ /// Returns true if the recurrence kind is of the form
+ /// select(cmp(),x,y) where one of (x,y) is an increasing or decreasing loop
+ /// induction.
+ static bool isFindIVRecurrenceKind(RecurKind Kind) {
+ return Kind == RecurKind::IFindLastIV || Kind == RecurKind::FFindLastIV ||
+ Kind == RecurKind::FindFirstIV;
+ }
+
/// Returns the type of the recurrence. This type can be narrower than the
/// actual type of the Phi if the recurrence has been type-promoted.
Type *getRecurrenceType() const { return RecurrenceType; }
- /// Returns the sentinel value for FindLastIV recurrences to replace the start
+ /// Returns the sentinel value for FindFirstIV &FindLastIV recurrences to replace the start
/// value.
Value *getSentinelValue() const {
- assert(isFindLastIVRecurrenceKind(Kind) && "Unexpected recurrence kind");
Type *Ty = StartValue->getType();
- return ConstantInt::get(Ty,
- APInt::getSignedMinValue(Ty->getIntegerBitWidth()));
+ if (isFindLastIVRecurrenceKind(Kind))
+ return ConstantInt::get(
+ Ty, APInt::getSignedMinValue(Ty->getIntegerBitWidth()));
+ else {
+ assert(Kind == RecurKind::FindFirstIV);
+ return ConstantInt::get(Ty, APInt::getMaxValue(Ty->getIntegerBitWidth()));
+ }
}
/// Returns a reference to the instructions used for type-promoting the
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index a216b0347b9fa..098ee88334bee 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -53,6 +53,7 @@ bool RecurrenceDescriptor::isIntegerRecurrenceKind(RecurKind Kind) {
case RecurKind::FAnyOf:
case RecurKind::IFindLastIV:
case RecurKind::FFindLastIV:
+ case RecurKind::FindFirstIV:
return true;
}
return false;
@@ -688,8 +689,9 @@ RecurrenceDescriptor::isAnyOfPattern(Loop *Loop, PHINode *OrigPhi,
// value of the data type or a non-constant value by using mask and multiple
// reduction operations.
RecurrenceDescriptor::InstDesc
-RecurrenceDescriptor::isFindLastIVPattern(Loop *TheLoop, PHINode *OrigPhi,
- Instruction *I, ScalarEvolution &SE) {
+RecurrenceDescriptor::isFindIVPattern(RecurKind Kind, Loop *TheLoop,
+ PHINode *OrigPhi, Instruction *I,
+ ScalarEvolution &SE) {
// TODO: Support the vectorization of FindLastIV when the reduction phi is
// used by more than one select instruction. This vectorization is only
// performed when the SCEV of each increasing induction variable used by the
@@ -705,7 +707,7 @@ RecurrenceDescriptor::isFindLastIVPattern(Loop *TheLoop, PHINode *OrigPhi,
m_Value(NonRdxPhi)))))
return InstDesc(false, I);
- auto IsIncreasingLoopInduction = [&](Value *V) {
+ auto IsSupportedLoopInduction = [&](Value *V) {
Type *Ty = V->getType();
if (!SE.isSCEVable(Ty))
return false;
@@ -715,20 +717,26 @@ RecurrenceDescriptor::isFindLastIVPattern(Loop *TheLoop, PHINode *OrigPhi,
return false;
const SCEV *Step = AR->getStepRecurrence(SE);
- if (!SE.isKnownPositive(Step))
+ if (Kind == RecurKind::FindFirstIV) {
+ if (!SE.isKnownNegative(Step))
+ return false;
+ } else if (!SE.isKnownPositive(Step))
return false;
const ConstantRange IVRange = SE.getSignedRange(AR);
unsigned NumBits = Ty->getIntegerBitWidth();
- // Keep the minimum value of the recurrence type as the sentinel value.
- // The maximum acceptable range for the increasing induction variable,
- // called the valid range, will be defined as
+ // Keep the minimum (FindLast) or maximum (FindFirst) value of the
+ // recurrence type as the sentinel value. The maximum acceptable range for
+ // the induction variable, called the valid range, will be defined as
// [<sentinel value> + 1, <sentinel value>)
- // where <sentinel value> is SignedMin(<recurrence type>)
+ // where <sentinel value> is SignedMin(<recurrence type>) for FindLast or
+ // UnsignedMax(<recurrence type>) for FindFirst.
// TODO: This range restriction can be lifted by adding an additional
// virtual OR reduction.
- const APInt Sentinel = APInt::getSignedMinValue(NumBits);
- const ConstantRange ValidRange =
+ const APInt Sentinel = Kind == RecurKind::FindFirstIV
+ ? APInt::getMaxValue(NumBits)
+ : APInt::getSignedMinValue(NumBits);
+ ConstantRange ValidRange =
ConstantRange::getNonEmpty(Sentinel + 1, Sentinel);
LLVM_DEBUG(dbgs() << "LV: FindLastIV valid range is " << ValidRange
<< ", and the signed range of " << *AR << " is "
@@ -741,10 +749,11 @@ RecurrenceDescriptor::isFindLastIVPattern(Loop *TheLoop, PHINode *OrigPhi,
// We are looking for selects of the form:
// select(cmp(), phi, increasing_loop_induction) or
// select(cmp(), increasing_loop_induction, phi)
- // TODO: Support for monotonically decreasing induction variable
- if (!IsIncreasingLoopInduction(NonRdxPhi))
+ if (!IsSupportedLoopInduction(NonRdxPhi))
return InstDesc(false, I);
+ if (Kind == RecurKind::FindFirstIV)
+ return InstDesc(I, Kind);
return InstDesc(I, isa<ICmpInst>(I->getOperand(0)) ? RecurKind::IFindLastIV
: RecurKind::FFindLastIV);
}
@@ -883,8 +892,8 @@ RecurrenceDescriptor::InstDesc RecurrenceDescriptor::isRecurrenceInstr(
if (Kind == RecurKind::FAdd || Kind == RecurKind::FMul ||
Kind == RecurKind::Add || Kind == RecurKind::Mul)
return isConditionalRdxPattern(Kind, I);
- if (isFindLastIVRecurrenceKind(Kind) && SE)
- return isFindLastIVPattern(L, OrigPhi, I, *SE);
+ if (isFindIVRecurrenceKind(Kind) && SE)
+ return isFindIVPattern(Kind, L, OrigPhi, I, *SE);
[[fallthrough]];
case Instruction::FCmp:
case Instruction::ICmp:
@@ -1002,6 +1011,12 @@ bool RecurrenceDescriptor::isReductionPHI(PHINode *Phi, Loop *TheLoop,
<< "FindLastIV reduction PHI." << *Phi << "\n");
return true;
}
+ if (AddReductionVar(Phi, RecurKind::FindFirstIV, TheLoop, FMF, RedDes, DB, AC,
+ DT, SE)) {
+ LLVM_DEBUG(dbgs() << "Found a FindFirstV reduction PHI." << *Phi << "\n");
+ return true;
+ }
+
if (AddReductionVar(Phi, RecurKind::FMul, TheLoop, FMF, RedDes, DB, AC, DT,
SE)) {
LLVM_DEBUG(dbgs() << "Found an FMult reduction PHI." << *Phi << "\n");
@@ -1171,6 +1186,7 @@ unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
case RecurKind::UMin:
case RecurKind::IAnyOf:
case RecurKind::IFindLastIV:
+ case RecurKind::FindFirstIV:
return Instruction::ICmp;
case RecurKind::FMax:
case RecurKind::FMin:
diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index 2fff9521017ff..cc1774dcca524 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -1244,12 +1244,14 @@ Value *llvm::createAnyOfReduction(IRBuilderBase &Builder, Value *Src,
Value *llvm::createFindLastIVReduction(IRBuilderBase &Builder, Value *Src,
Value *Start,
const RecurrenceDescriptor &Desc) {
- assert(RecurrenceDescriptor::isFindLastIVRecurrenceKind(
- Desc.getRecurrenceKind()) &&
- "Unexpected reduction kind");
+ assert(
+ RecurrenceDescriptor::isFindIVRecurrenceKind(Desc.getRecurrenceKind()) &&
+ "Unexpected reduction kind");
Value *Sentinel = Desc.getSentinelValue();
Value *MaxRdx = Src->getType()->isVectorTy()
- ? Builder.CreateIntMaxReduce(Src, true)
+ ? (Desc.getRecurrenceKind() == RecurKind::FindFirstIV
+ ? Builder.CreateIntMinReduce(Src, false)
+ : Builder.CreateIntMaxReduce(Src, true))
: Src;
// Correct the final reduction result back to the start value if the maximum
// reduction is sentinel value.
@@ -1345,8 +1347,8 @@ Value *llvm::createSimpleReduction(IRBuilderBase &Builder, Value *Src,
Value *llvm::createSimpleReduction(VectorBuilder &VBuilder, Value *Src,
RecurKind Kind) {
assert(!RecurrenceDescriptor::isAnyOfRecurrenceKind(Kind) &&
- !RecurrenceDescriptor::isFindLastIVRecurrenceKind(Kind) &&
- "AnyOf or FindLastIV reductions are not supported.");
+ !RecurrenceDescriptor::isFindIVRecurrenceKind(Kind) &&
+ "AnyOf, FindFirstIV and FindLastIV reductions are not supported.");
Intrinsic::ID Id = getReductionIntrinsicID(Kind);
auto *SrcTy = cast<VectorType>(Src->getType());
Type *SrcEltTy = SrcTy->getElementType();
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 148b187f4f25d..ee4ab5765a5f1 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -5173,7 +5173,7 @@ LoopVectorizationCostModel::selectInterleaveCount(VPlan &Plan, ElementCount VF,
const RecurrenceDescriptor &RdxDesc = Reduction.second;
RecurKind RK = RdxDesc.getRecurrenceKind();
return RecurrenceDescriptor::isAnyOfRecurrenceKind(RK) ||
- RecurrenceDescriptor::isFindLastIVRecurrenceKind(RK);
+ RecurrenceDescriptor::isFindIVRecurrenceKind(RK);
});
if (HasSelectCmpReductions) {
LLVM_DEBUG(dbgs() << "LV: Not interleaving select-cmp reductions.\n");
@@ -7726,7 +7726,7 @@ static void fixReductionScalarResumeWhenVectorizingEpilog(
auto *EpiRedResult = dyn_cast<VPInstruction>(R);
if (!EpiRedResult ||
(EpiRedResult->getOpcode() != VPInstruction::ComputeReductionResult &&
- EpiRedResult->getOpcode() != VPInstruction::ComputeFindLastIVResult))
+ EpiRedResult->getOpcode() != VPInstruction::ComputeFindIVResult))
return;
auto *EpiRedHeaderPhi =
@@ -7744,7 +7744,7 @@ static void fixReductionScalarResumeWhenVectorizingEpilog(
"AnyOf expected to start by comparing main resume value to original "
"start value");
MainResumeValue = Cmp->getOperand(0);
- } else if (RecurrenceDescriptor::isFindLastIVRecurrenceKind(
+ } else if (RecurrenceDescriptor::isFindIVRecurrenceKind(
RdxDesc.getRecurrenceKind())) {
using namespace llvm::PatternMatch;
Value *Cmp, *OrigResumeV, *CmpOp;
@@ -9760,7 +9760,7 @@ void LoopVectorizationPlanner::adjustRecipesForReductions(
RecurKind Kind = RdxDesc.getRecurrenceKind();
assert(
!RecurrenceDescriptor::isAnyOfRecurrenceKind(Kind) &&
- !RecurrenceDescriptor::isFindLastIVRecurrenceKind(Kind) &&
+ !RecurrenceDescriptor::isFindIVRecurrenceKind(Kind) &&
"AnyOf and FindLast reductions are not allowed for in-loop reductions");
// Collect the chain of "link" recipes for the reduction starting at PhiR.
@@ -9917,7 +9917,7 @@ void LoopVectorizationPlanner::adjustRecipesForReductions(
(cast<VPInstruction>(&U)->getOpcode() ==
VPInstruction::ComputeReductionResult ||
cast<VPInstruction>(&U)->getOpcode() ==
- VPInstruction::ComputeFindLastIVResult);
+ VPInstruction::ComputeFindIVResult);
});
if (CM.usePredicatedReductionSelect())
PhiR->setOperand(1, NewExitingVPV);
@@ -9962,11 +9962,11 @@ void LoopVectorizationPlanner::adjustRecipesForReductions(
VPInstruction *FinalReductionResult;
VPBuilder::InsertPointGuard Guard(Builder);
Builder.setInsertPoint(MiddleVPBB, IP);
- if (RecurrenceDescriptor::isFindLastIVRecurrenceKind(
+ if (RecurrenceDescriptor::isFindIVRecurrenceKind(
RdxDesc.getRecurrenceKind())) {
VPValue *Start = PhiR->getStartValue();
FinalReductionResult =
- Builder.createNaryOp(VPInstruction::ComputeFindLastIVResult,
+ Builder.createNaryOp(VPInstruction::ComputeFindIVResult,
{PhiR, Start, NewExitingVPV}, ExitDL);
} else {
FinalReductionResult = Builder.createNaryOp(
@@ -10013,11 +10013,11 @@ void LoopVectorizationPlanner::adjustRecipesForReductions(
continue;
}
- if (RecurrenceDescriptor::isFindLastIVRecurrenceKind(
+ if (RecurrenceDescriptor::isFindIVRecurrenceKind(
RdxDesc.getRecurrenceKind())) {
- // Adjust the start value for FindLastIV recurrences to use the sentinel
- // value after generating the ResumePhi recipe, which uses the original
- // start value.
+ // Adjust the start value for FindFirstIV/FindLastIV recurrences to use
+ // the sentinel value after generating the ResumePhi recipe, which uses
+ // the original start value.
PhiR->setOperand(0, Plan->getOrAddLiveIn(RdxDesc.getSentinelValue()));
}
}
@@ -10385,7 +10385,7 @@ static void preparePlanForMainVectorLoop(VPlan &MainPlan, VPlan &EpiPlan) {
VPlanTransforms::runPass(VPlanTransforms::removeDeadRecipes, MainPlan);
using namespace VPlanPatternMatch;
- // When vectorizing the epilogue, FindLastIV reductions can introduce multiple
+ // When vectorizing the epilogue, FindFirstIV & FindLastIV reductions can introduce multiple
// uses of undef/poison. If the reduction start value may be undef or poison
// it needs to be frozen and the frozen start has to be used when computing
// the reduction result. We also need to use the frozen value in the resume
@@ -10396,7 +10396,7 @@ static void preparePlanForMainVectorLoop(VPlan &MainPlan, VPlan &EpiPlan) {
VPBuilder Builder(Plan.getEntry());
for (VPRecipeBase &R : *Plan.getMiddleBlock()) {
auto *VPI = dyn_cast<VPInstruction>(&R);
- if (!VPI || VPI->getOpcode() != VPInstruction::ComputeFindLastIVResult)
+ if (!VPI || VPI->getOpcode() != VPInstruction::ComputeFindIVResult)
continue;
VPValue *OrigStart = VPI->getOperand(1);
if (isGuaranteedNotToBeUndefOrPoison(OrigStart->getLiveInIRValue()))
@@ -10507,17 +10507,17 @@ preparePlanForEpilogueVectorLoop(VPlan &Plan, Loop *L,
IRBuilder<> Builder(PBB, PBB->getFirstNonPHIIt());
ResumeV =
Builder.CreateICmpNE(ResumeV, RdxDesc.getRecurrenceStartValue());
- } else if (RecurrenceDescriptor::isFindLastIVRecurrenceKind(RK)) {
+ } else if (RecurrenceDescriptor::isFindIVRecurrenceKind(RK)) {
ToFrozen[RdxDesc.getRecurrenceStartValue()] =
cast<PHINode>(ResumeV)->getIncomingValueForBlock(
EPI.MainLoopIterationCountCheck);
- // VPReductionPHIRecipe for FindLastIV reductions requires an adjustment
- // to the resume value. The resume value is adjusted to the sentinel
- // value when the final value from the main vector loop equals the start
- // value. This ensures correctness when the start value might not be
- // less than the minimum value of a monotonically increasing induction
- // variable.
+ // VPReductionPHIRecipe for FindFirstIV/FindLastIV reductions requires
+ // an adjustment to the resume value. The resume value is adjusted to
+ // the sentinel value when the final value from the main vector loop
+ // equals the start value. This ensures correctness when the start value
+ // might not be less than the minimum value of a monotonically
+ // increasing induction variable.
BasicBlock *ResumeBB = cast<Instruction>(ResumeV)->getParent();
IRBuilder<> Builder(ResumeBB, ResumeBB->getFirstNonPHIIt());
Value *Cmp = Builder.CreateICmpEQ(
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index eb339282fdae8..7c4eae01e056a 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -23250,6 +23250,7 @@ class HorizontalReduction {
case RecurKind::FAnyOf:
case RecurKind::IFindLastIV:
case RecurKind::FFindLastIV:
+ case RecurKind::FindFirstIV:
case RecurKind::FMaximumNum:
case RecurKind::FMinimumNum:
case RecurKind::None:
@@ -23386,6 +23387,7 @@ class HorizontalReduction {
case RecurKind::FAnyOf:
case RecurKind::IFindLastIV:
case RecurKind::FFindLastIV:
+ case RecurKind::FindFirstIV:
case RecurKind::FMaximumNum:
case RecurKind::FMinimumNum:
case RecurKind::None:
@@ -23487,6 +23489,7 @@ class HorizontalReduction {
case RecurKind::FAnyOf:
case RecurKind::IFindLastIV:
case RecurKind::FFindLastIV:
+ case RecurKind::FindFirstIV:
case RecurKind::FMaximumNum:
case RecurKind::FMinimumNum:
case RecurKind::None:
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index e634de1e17c69..ca9f350528203 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -898,7 +898,7 @@ class VPInstruction : public VPRecipeWithIRFlags,
BranchOnCount,
BranchOnCond,
Broadcast,
- ComputeFindLastIVResult,
+ ComputeFindIVResult,
ComputeReductionResult,
// Extracts the last lane from its operand if it is a vector, or the last
// part if scalar. In the latter case, the recipe will be removed during
diff --git a/llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp b/llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp
index ac0f30cb4693c..7ff54fdf0e924 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp
@@ -87,7 +87,7 @@ Type *VPTypeAnalysis::inferScalarTypeForRecipe(const...
[truncated]
|
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
artagnon
reviewed
May 27, 2025
artagnon
reviewed
May 28, 2025
artagnon
reviewed
May 28, 2025
llvm/lib/Analysis/IVDescriptors.cpp
Outdated
| Kind == RecurKind::FindFirstIVSMin) { | ||
| if (!SE.isKnownNegative(Step)) | ||
| return false; | ||
| } else if (!SE.isKnownPositive(Step)) | ||
| return false; | ||
|
|
||
| const ConstantRange IVRange = SE.getSignedRange(AR); |
There was a problem hiding this comment.
Shouldn't this be getUnsignedRange in some cases?
| if (isFindLastIVRecurrenceKind(Kind)) { | ||
| return ConstantInt::get( | ||
| Ty, APInt::getSignedMinValue(Ty->getIntegerBitWidth())); | ||
| } else if (Kind == RecurKind::FindFirstIVSMin) { |
| } else if (Kind == RecurKind::FindFirstIVSMin) { | ||
| return ConstantInt::get( | ||
| Ty, APInt::getSignedMaxValue(Ty->getIntegerBitWidth())); | ||
| } else { |
llvm/lib/Analysis/IVDescriptors.cpp
Outdated
Comment on lines
739
to
743
| if (Kind == RecurKind::FindFirstIVSMin) | ||
| ValidRange = | ||
| ConstantRange::getNonEmpty(APInt::getSignedMinValue(NumBits), | ||
| APInt::getSignedMaxValue(NumBits) - 1); | ||
| else { |
There was a problem hiding this comment.
The code here completely changed after rebase.
Comment on lines
624
to
633
| if (RK == RecurKind::FindLastIV) | ||
| ReducedPartRdx = | ||
| createMinMaxOp(Builder, RecurKind::SMax, ReducedPartRdx, | ||
| State.get(getOperand(2 + Part))); | ||
| else | ||
| ReducedPartRdx = | ||
| createMinMaxOp(Builder, | ||
| RK == RecurKind::FindFirstIVSMin ? RecurKind::SMin | ||
| : RecurKind::UMin, | ||
| ReducedPartRdx, State.get(getOperand(2 + Part))); |
There was a problem hiding this comment.
Lots of similar code, better just to preselect RecurKind
There was a problem hiding this comment.
Done in the latest version, thanks
Mel-Chen
reviewed
May 28, 2025
Comment on lines
283
to
310
| /// Returns the sentinel value for FindFirstIV &FindLastIV recurrences to | ||
| /// replace the start value. | ||
| Value *getSentinelValue() const { | ||
| assert(isFindLastIVRecurrenceKind(Kind) && "Unexpected recurrence kind"); | ||
| Type *Ty = StartValue->getType(); | ||
| return ConstantInt::get(Ty, | ||
| APInt::getSignedMinValue(Ty->getIntegerBitWidth())); | ||
| if (isFindLastIVRecurrenceKind(Kind)) { | ||
| return ConstantInt::get( | ||
| Ty, APInt::getSignedMinValue(Ty->getIntegerBitWidth())); | ||
| } else if (Kind == RecurKind::FindFirstIVSMin) { | ||
| return ConstantInt::get( | ||
| Ty, APInt::getSignedMaxValue(Ty->getIntegerBitWidth())); | ||
| } else { | ||
| assert(Kind == RecurKind::FindFirstIVUMin); | ||
| return ConstantInt::get(Ty, APInt::getMaxValue(Ty->getIntegerBitWidth())); | ||
| } | ||
| } |
There was a problem hiding this comment.
#141788 Try to change the sentinel value to be selected dynamically instead of based on RecurKind statically.
If I'm not mistaken, the values for signed max, unsigned max, signed min, and unsigned min are all distinct.
So making the sentinel value non-static could help with supporting the unsigned version, I think.
There was a problem hiding this comment.
Yep that would also be a good extension.
Comment on lines
59
to
64
| FindFirstIVUMin, /// FindFirst reduction with select(icmp(),x,y) where one of | ||
| ///< (x,y) is a decreasing loop induction, and both x and y | ||
| ///< are integer type. | ||
| FindFirstIVSMin /// FindFirst reduction with select(icmp(),x,y) where one of | ||
| ///< (x,y) is a decreasing loop induction, and both x and y | ||
| ///< are integer type. |
There was a problem hiding this comment.
This patch introduces too many new idioms at once.
To keep the patch reviewable and focused, I suggest we pick one direction to extend first:
either support unsigned FindLastIV, or add signed FindFirstIV support.
There was a problem hiding this comment.
Thanks, updated to just add FindFirstIVSMin (the original version didn't try to add FindLastIV IIRC)
artagnon
reviewed
Jun 27, 2025
llvm/lib/Analysis/IVDescriptors.cpp
Outdated
| Kind == RecurKind::FindFirstIVSMin) { | ||
| if (!SE.isKnownNegative(Step)) | ||
| return false; | ||
| } else if (!SE.isKnownPositive(Step)) | ||
| return false; | ||
|
|
||
| const ConstantRange IVRange = SE.getSignedRange(AR); |
| /// Returns the type of the recurrence. This type can be narrower than the | ||
| /// actual type of the Phi if the recurrence has been type-promoted. | ||
| Type *getRecurrenceType() const { return RecurrenceType; } | ||
|
|
||
| /// Returns the sentinel value for FindLastIV recurrences to replace the start | ||
| /// value. | ||
| /// Returns the sentinel value for FindFirstIV &FindLastIV recurrences to |
There was a problem hiding this comment.
Suggested change
| /// Returns the sentinel value for FindFirstIV &FindLastIV recurrences to | |
| /// Returns the sentinel value for FindFirstIV & FindLastIV recurrences to |
| MinMaxKind = IsSigned ? RecurKind::SMax : RecurKind::UMax; | ||
| } else { | ||
| assert(RecurrenceDescriptor::isFindFirstIVRecurrenceKind(RK) && | ||
| "Kind must either be a FindLastIV or FindFirstIV"); |
There was a problem hiding this comment.
Suggested change
| "Kind must either be a FindLastIV or FindFirstIV"); | |
| "Kind must either be FindLastIV or FindFirstIV"); |
| assert(RecurrenceDescriptor::isFindFirstIVRecurrenceKind(RK) && | ||
| "Kind must either be a FindLastIV or FindFirstIV"); | ||
| assert(IsSigned && | ||
| "only FindFirstIV with SMax is supported at the moment"); |
There was a problem hiding this comment.
Suggested change
| "only FindFirstIV with SMax is supported at the moment"); | |
| "Only FindFirstIV with SMax is currently supported"); |
Comment on lines
2
to
4
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=1 -force-vector-width=4 -S < %s | FileCheck --check-prefix=IC1VF4 %s | ||
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=4 -force-vector-width=4 -S < %s | FileCheck --check-prefix=IC4VF4 %s | ||
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=4 -force-vector-width=1 -S < %s | FileCheck --check-prefix=IC4VF1 %s |
There was a problem hiding this comment.
Suggested change
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=1 -force-vector-width=4 -S < %s | FileCheck --check-prefix=IC1VF4 %s | |
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=4 -force-vector-width=4 -S < %s | FileCheck --check-prefix=IC4VF4 %s | |
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=4 -force-vector-width=1 -S < %s | FileCheck --check-prefix=IC4VF1 %s | |
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=1 -force-vector-width=4 -S < %s | FileCheck --check-prefix=CHECK,IC1VF4 %s | |
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=4 -force-vector-width=4 -S < %s | FileCheck --check-prefix=CHECK,IC4VF4 %s | |
| ; RUN: opt -passes=loop-vectorize -force-vector-interleave=4 -force-vector-width=1 -S < %s | FileCheck --check-prefix=CHECK,IC4VF1 %s |
There was a problem hiding this comment.
Ah yes, keep forgetting this is working now, thanks!
artagnon
approved these changes
Jun 27, 2025
llvm/lib/Analysis/IVDescriptors.cpp
Outdated
Comment on lines
719
to
720
| if (isFindFirstIVRecurrenceKind(Kind)) { | ||
| if (!SE.isKnownNegative(Step)) | ||
| return std::nullopt; | ||
| } else if (!SE.isKnownPositive(Step)) | ||
| return std::nullopt; |
There was a problem hiding this comment.
Suggested change
| if (isFindFirstIVRecurrenceKind(Kind)) { | |
| if (!SE.isKnownNegative(Step)) | |
| return std::nullopt; | |
| } else if (!SE.isKnownPositive(Step)) | |
| return std::nullopt; | |
| if ((isFindFirstIVRecurrenceKind(Kind) && !SE.isKnownNegative(Step)) || | |
| (isFindLastIVRecurrenceKind(Kind) && !SE.isKnownPositive(Step))) | |
| return std::nullopt; |
llvm/lib/Analysis/IVDescriptors.cpp
Outdated
Comment on lines
747
to
748
| assert(isFindFirstIVRecurrenceKind(Kind) && | ||
| "Kind must either be FindLastIV or FindFirstIV"); |
There was a problem hiding this comment.
Suggested change
| assert(isFindFirstIVRecurrenceKind(Kind) && | |
| "Kind must either be FindLastIV or FindFirstIV"); |
There's already an assert on line 773.
There was a problem hiding this comment.
Yeah, thought it might be good for extra safety, but dropped it for now, thanks
Comment on lines
1230
to
1235
| Value *MaxRdx = | ||
| Src->getType()->isVectorTy() | ||
| ? (RecurrenceDescriptor::isFindLastIVRecurrenceKind(RdxKind) | ||
| ? Builder.CreateIntMaxReduce(Src, IsSigned) | ||
| : Builder.CreateIntMinReduce(Src, IsSigned)) | ||
| : Src; |
There was a problem hiding this comment.
Suggested change
| Value *MaxRdx = | |
| Src->getType()->isVectorTy() | |
| ? (RecurrenceDescriptor::isFindLastIVRecurrenceKind(RdxKind) | |
| ? Builder.CreateIntMaxReduce(Src, IsSigned) | |
| : Builder.CreateIntMinReduce(Src, IsSigned)) | |
| : Src; | |
| bool IsMaxRdx = RecurrenceDescriptor::isFindLastIVRecurrenceKind(RdxKind); | |
| Value *MinMaxRdx = | |
| Src->getType()->isVectorTy() | |
| ? (IsMaxRdx | |
| ? Builder.CreateIntMaxReduce(Src, IsSigned) | |
| : Builder.CreateIntMinReduce(Src, IsSigned)) | |
| : Src; |
| !RecurrenceDescriptor::isFindLastIVRecurrenceKind(Kind) && | ||
| "AnyOf or FindLastIV reductions are not supported."); | ||
| !RecurrenceDescriptor::isFindIVRecurrenceKind(Kind) && | ||
| "AnyOf, FindFirstIV and FindLastIV reductions are not supported."); |
There was a problem hiding this comment.
Suggested change
| "AnyOf, FindFirstIV and FindLastIV reductions are not supported."); | |
| "AnyOf and FindIV reductions are not supported."); |
Similar to FindLastIV, add FindFirstIV to support select (icmp(), x, y) reductions where one of x or y is a decreasing induction. This is done via a new recurrence kind FindFirstIVSMin, which selects the first value from the reduction vector using smin instead of the last value (FindLastIV). It uses signed max as sentinel value. The
llvm-sync bot
pushed a commit
to arm/arm-toolchain
that referenced
this pull request
Jun 29, 2025
…#140451) Similar to FindLastIV, add FindFirstIVSMin to support select (icmp(), x, y) reductions where one of x or y is a decreasing induction, producing a SMin reduction. It uses signed max as sentinel value. PR: llvm/llvm-project#140451
|
Just chiming into say this PR gives a a 78% speedup on RISC-V for MultiSource/Benchmarks/TSVC/Searching-flt/Searching-flt: https://lnt.lukelau.me/db_default/v4/nts/profile/492/702/699 |
rlavaee
pushed a commit
to rlavaee/llvm-project
that referenced
this pull request
Jul 1, 2025
Similar to FindLastIV, add FindFirstIVSMin to support select (icmp(), x, y) reductions where one of x or y is a decreasing induction, producing a SMin reduction. It uses signed max as sentinel value. PR: llvm#140451
rlavaee
pushed a commit
to rlavaee/llvm-project
that referenced
this pull request
Jul 1, 2025
Similar to FindLastIV, add FindFirstIVSMin to support select (icmp(), x, y) reductions where one of x or y is a decreasing induction, producing a SMin reduction. It uses signed max as sentinel value. PR: llvm#140451
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Similar to FindLastIV, add FindFirstIV to support select (icmp(), x, y)
reductions where one of x or y is a decreasing induction. This is done
via two new recurrence kinds FindFirstIV(UMin|SMin), which selects the first
value from the reduction vector using umin/smin instead of the last value
(FindLastIV). It uses unsigned max/signed max as sentinel value. The
SMin version is used if the IV range does not include signed max.
Otherwise UMin is used if the IV range does not include signed max.
Alternatively we could consolidate FindFirstIV/FindLastIV in a single
FindIV kind, and add a field indicating if whether we want to select the
first or last value.