Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lambda inlining optimization #353

Open
wants to merge 87 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
274fd19
Added lambda inlining
TimothyGeerkensGuardsquare Aug 3, 2023
043be3a
Fixes from inlining lambdas in the klox compiler
MaartenS11 Aug 4, 2023
c098ad8
Added test for retracer + addressed Oberon's comment from index 1 to 9
TimothyGeerkensGuardsquare Aug 4, 2023
7922f11
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
MaartenS11 Aug 7, 2023
bb66672
Fixed regression from merge
MaartenS11 Aug 7, 2023
573995e
Removed some unused fields from the RecursiveInliner
MaartenS11 Aug 7, 2023
e4ce1e0
Removed import .* in LambdaLocator
TimothyGeerkensGuardsquare Aug 7, 2023
70e5966
Simplified the RecursiveInliner a bit
MaartenS11 Aug 7, 2023
aabc5bd
Cleaned up RecursiveInliner by moving some code into a new CalledMeth…
MaartenS11 Aug 7, 2023
f4a22fa
Move some code into IndyLambdaImplVisitor class
MaartenS11 Aug 7, 2023
7b2146e
Remove argument from RecursiveInliner because it isn't really needed …
MaartenS11 Aug 7, 2023
3573c19
Reduce line length a bit by putting index calculation in a variable
MaartenS11 Aug 7, 2023
025571c
Remove wildcard imports from RecursiveInliner
MaartenS11 Aug 7, 2023
845d044
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
MaartenS11 Aug 7, 2023
13b38d5
Removed visitor staircase in LambdaLocator
TimothyGeerkensGuardsquare Aug 7, 2023
fb9fe7d
Removed wildcards in import
TimothyGeerkensGuardsquare Aug 7, 2023
cf74760
Cleaned up the while loop in the LambdaUsageFinder by using the new I…
MaartenS11 Aug 7, 2023
2090514
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
MaartenS11 Aug 7, 2023
fb5eedb
Merge pull request #1 from MaartenS11/klox-fixes
MaartenS11 Aug 7, 2023
6ae5fc3
Removed unused class and method
TimothyGeerkensGuardsquare Aug 7, 2023
e90c5c9
merge
TimothyGeerkensGuardsquare Aug 7, 2023
33aa83d
Add more comments to the RecursiveInliner
MaartenS11 Aug 7, 2023
1d49b49
removed print and added logging
TimothyGeerkensGuardsquare Aug 7, 2023
cb60c13
refactored test and added performance tests
TimothyGeerkensGuardsquare Aug 7, 2023
3d3bf8f
Only stop inlining if we actually have the current lambda in one of t…
MaartenS11 Aug 7, 2023
8c354ab
refactored util
TimothyGeerkensGuardsquare Aug 8, 2023
813be34
Use argument index from LambdaUsageFinder instead of searching in the…
MaartenS11 Aug 8, 2023
f155107
RecursiveInliner should also use the argument index from the lambda u…
MaartenS11 Aug 8, 2023
7d82d2d
Remove findFirstLambdaParameter method since it is no longer needed
MaartenS11 Aug 8, 2023
047719c
WIP commit (please amend)
MaartenS11 Aug 8, 2023
5cc6748
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
MaartenS11 Aug 8, 2023
1f5c85b
Merge pull request #2 from MaartenS11/recursive-inliner-adjustments
MaartenS11 Aug 8, 2023
54233a5
Put stop condition that was duplicated into a method
MaartenS11 Aug 8, 2023
ef37241
Don't hardcode the MethodInliner max resulting code size
MaartenS11 Aug 8, 2023
beeadd5
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
MaartenS11 Aug 8, 2023
343df22
Fixed cast removal so it works in all cases
MaartenS11 Aug 8, 2023
71213fe
Reverted some minor accidental changes
MaartenS11 Aug 8, 2023
42493c6
Cleaned up a bit
MaartenS11 Aug 8, 2023
8cfd505
Merge pull request #3 from MaartenS11/more-fixes
MaartenS11 Aug 8, 2023
cff6565
Conditionally inline lambdas
MaartenS11 Aug 9, 2023
5644a9d
Added logging when a lambda has been inlined and only print method le…
MaartenS11 Aug 9, 2023
48c52e5
Log the consuming method and invoke method so the use knows which met…
MaartenS11 Aug 9, 2023
badf492
Merge pull request #4 from MaartenS11/inline-conditionally
MaartenS11 Aug 9, 2023
61e1e98
Fixed typo in comment
MaartenS11 Aug 9, 2023
00664b2
Removed some unnecessary comments from the tests
MaartenS11 Aug 9, 2023
fca2cde
Fixed typo in comment
MaartenS11 Aug 9, 2023
f8f1155
Rename package to lambdainline instead of lambdaInline
MaartenS11 Aug 9, 2023
bcab91e
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
MaartenS11 Aug 9, 2023
2710ff3
Rename lambda_locator package to lambdalocator
MaartenS11 Aug 9, 2023
49bd302
Added newline to Lambda class and removed some unneeded methods
MaartenS11 Aug 9, 2023
55b4f0e
Use result of lambdaUsageHandler.handle() directly instead of storing…
MaartenS11 Aug 9, 2023
7613999
This doesn't have to be a nested if
MaartenS11 Aug 9, 2023
ee3e3ab
Added more javadoc comments to the IterativeInstructionVisitor
MaartenS11 Aug 9, 2023
f1ba773
Added more javadoc strings and fixed a typo
MaartenS11 Aug 9, 2023
7d602c9
Renamed isSourceInstruction to isNotSourceInstruction
TimothyGeerkensGuardsquare Aug 9, 2023
4d18370
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
TimothyGeerkensGuardsquare Aug 9, 2023
2648338
Added more javadoc comments
MaartenS11 Aug 9, 2023
b431cb7
Apply suggestions from code review
MaartenS11 Aug 9, 2023
11f21f1
Use Optional in MethodLengthFinder so the programmer is made aware th…
MaartenS11 Aug 9, 2023
028e8c4
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
MaartenS11 Aug 9, 2023
0cc53cd
Return ProgramMethod instead of just Method
MaartenS11 Aug 9, 2023
a440b45
Use the name kotlin lambdas instead of static lambdas in the LambdaLo…
MaartenS11 Aug 9, 2023
7e652b3
Renamed constantInstruction and offset to be more precise also added …
MaartenS11 Aug 9, 2023
607bf86
Added more info about what "a lambda" really means
MaartenS11 Aug 10, 2023
f69fd14
Change some things to the FixedPointCodeAttributeVisitor API
MaartenS11 Aug 10, 2023
95b23ab
invokespecial is now considered a potential source for the sourcetracer
TimothyGeerkensGuardsquare Aug 10, 2023
c5bca4a
merge
TimothyGeerkensGuardsquare Aug 10, 2023
23d4efe
Use library and program class pools instead of passing AppView
MaartenS11 Aug 10, 2023
7cdac26
Merge branch 'lambda-inlining' of github.com:MaartenS11/proguard into…
MaartenS11 Aug 10, 2023
03ae5c6
Made one of the debug log messages info again because it was accident…
MaartenS11 Aug 10, 2023
ad8b5ff
Add large javadoc comment explaining all the steps taken when using t…
MaartenS11 Aug 10, 2023
26d9850
Added comment explaining why descriptor is changed to use unboxed types
MaartenS11 Aug 10, 2023
a5ebb5e
Move a lot of code from the BaseLambdaInliner into a private inner cl…
MaartenS11 Aug 10, 2023
71b63a4
Fixed incorrect comment mentioning method that no longer exists
MaartenS11 Aug 10, 2023
7da8e01
Disable replacing usages with aconst_null
MaartenS11 Aug 10, 2023
751cc74
Add -inlinekotlinlambdas command line option to enable and disable la…
MaartenS11 Aug 10, 2023
f4f84e4
Moved lambda inlining optimization from inline to lambdainline packag…
MaartenS11 Aug 10, 2023
9d080b2
Add CannotInlineException message to debug log
MaartenS11 Aug 10, 2023
b8fbcd0
Add ProGuard header to every class
MaartenS11 Aug 11, 2023
3a866fc
Added detection for the second null check method for parameters in Ko…
MaartenS11 Aug 11, 2023
a910309
Added WIP README file to the lambdainline package
MaartenS11 Aug 11, 2023
0cf13a7
Fix example and explain more about the cast removal/boxing/unboxing
MaartenS11 Aug 11, 2023
1ba33ba
Mention a bit more which classes are used to do which operations
MaartenS11 Aug 11, 2023
ceea1b6
Be more specific when using the ClassReferenceInitializer, not all re…
MaartenS11 Aug 12, 2023
93d3bb6
Only re-run the ClassReferenceInitializer on the consuming class
MaartenS11 Aug 12, 2023
ac7f42d
Only run AccessFixer on consumingClass instead of the entire programC…
MaartenS11 Aug 12, 2023
370704b
Adjust conditional inlining attempt condition to use sum of consuming…
MaartenS11 Aug 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Cleaned up the while loop in the LambdaUsageFinder by using the new I…
…terativeInstructionVisitor class
  • Loading branch information
MaartenS11 committed Aug 7, 2023
commit cf74760194259f6fc62084772d6ad6d3fe057ced
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package proguard.optimize.inline;

import proguard.classfile.Clazz;
import proguard.classfile.Method;
import proguard.classfile.attribute.CodeAttribute;
import proguard.classfile.instruction.Instruction;
import proguard.classfile.instruction.InstructionFactory;
import proguard.classfile.instruction.visitor.InstructionVisitor;

public class IterativeInstructionVisitor {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you describe/show a case where this class is necessary?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lambda usage finder looks at the program and tries to identify lambda usages, when it finds a lambda usage it will call the usage handler and it will inline the lambda. Because inlining the lambda results in a method call being replaced by a call to another method which has the lambda inlined and we could still inline lambdas into that new method because the method has multiple lambda arguments we need to stop the current visit and visit the new changed code attribute instead. This class provides a simple API to do exactly that, iterate over the instructions and restart the iteration once the code has changed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see the need for this when you have nested lambdas. But I don't understand why it's necessary when you have multiple separate lambdas in one method. Why can't you register all the changes that need to be done and apply them with the CodeAttributeEditor in one go?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lambda inliner inlines one lambda at a time so you might have a call instruction to a method, the method has for example 2 lambdas as arguments. If we inline the first lambda we create a new method that now only has one argument. If we want to inline the second lambda we have to inline that lambda into the newly created method to get a new method that now has 0 arguments. We revisit the code attribute because when finding the usages we want to find the new method so we inline into the new method. If we don't do that you will get one method that takes in lambda 1 but not lambda 2 and one method that takes in lambda 2 but not lambda 1.

private boolean changed;

/**
* This helper method does something similar to codeAttribute.instructionsAccept but has the ability to stop and
* restart visiting the instructions, this is useful for when you change the code during iteration.
*/
public void instructionsAccept(Clazz clazz, Method method, CodeAttribute codeAttribute, InstructionVisitor instructionVisitor) {
int offset = 0;
while (offset < codeAttribute.u4codeLength) {
Instruction instruction = InstructionFactory.create(codeAttribute.code, offset);
int instructionLength = instruction.length(offset);
changed = false;
instruction.accept(clazz, method, codeAttribute, offset, instructionVisitor);
if (changed) {
offset = 0;
continue;
}
offset += instructionLength;
}
}

public boolean isChanged() {
return changed;
}

public void setChanged(boolean changed) {
this.changed = changed;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ public void execute(AppView appView) {
inlinedAllUsages = true;
InitializationUtil.initialize(appView.programClassPool, appView.libraryClassPool);
lambda.codeAttribute().accept(lambda.clazz(), lambda.method(),
new LambdaUsageFinder(lambda, lambdaLocator.getStaticLambdaMap(), appView,
new LambdaUsageFinder(lambda, lambdaLocator.getStaticLambdaMap(),
new LambdaUsageHandler(appView, remainder)
)
);
Expand Down
107 changes: 32 additions & 75 deletions base/src/main/java/proguard/optimize/inline/LambdaUsageFinder.java
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package proguard.optimize.inline;

import proguard.AppView;
import proguard.classfile.Clazz;
import proguard.classfile.Method;
import proguard.classfile.attribute.CodeAttribute;
Expand All @@ -10,8 +9,8 @@
import proguard.classfile.constant.visitor.ConstantVisitor;
import proguard.classfile.instruction.ConstantInstruction;
import proguard.classfile.instruction.Instruction;
import proguard.classfile.instruction.InstructionFactory;
import proguard.classfile.instruction.SimpleInstruction;
import proguard.classfile.instruction.visitor.InstructionOpCodeFilter;
import proguard.classfile.instruction.visitor.InstructionVisitor;
import proguard.classfile.util.ClassUtil;
import proguard.classfile.visitor.ClassPrinter;
Expand All @@ -20,8 +19,6 @@
import proguard.optimize.inline.lambda_locator.Lambda;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
Expand All @@ -31,76 +28,34 @@ public class LambdaUsageFinder implements InstructionVisitor, AttributeVisitor,
private final Lambda targetLambda;
private PartialEvaluator partialEvaluator;
private final Map<Integer, Lambda> lambdaMap;
private final AppView appView;
private final LambdaInliner.LambdaUsageHandler lambdaUsageHandler;
public MethodrefConstant methodrefConstant;
public FieldrefConstant referencedFieldConstant;
private final boolean inlineFromFields;
private final boolean inlineFromMethods;
private boolean changed = false;
private final int[] typedReturnInstructions = new int[] {
Instruction.OP_IRETURN,
Instruction.OP_LRETURN,
Instruction.OP_FRETURN,
Instruction.OP_DRETURN,
Instruction.OP_ARETURN
};

public LambdaUsageFinder(Lambda targetLambda, Map<Integer, Lambda> lambdaMap, AppView appView, boolean inlineFromFields, boolean inlineFromMethods, LambdaInliner.LambdaUsageHandler lambdaUsageHandler) {
private final IterativeInstructionVisitor iterativeInstructionVisitor;

public LambdaUsageFinder(Lambda targetLambda, Map<Integer, Lambda> lambdaMap, LambdaInliner.LambdaUsageHandler lambdaUsageHandler) {
this.targetLambda = targetLambda;
this.partialEvaluator = new PartialEvaluator();
this.lambdaMap = lambdaMap;
this.appView = appView;
this.lambdaUsageHandler = lambdaUsageHandler;
this.inlineFromFields = inlineFromFields;
this.inlineFromMethods = inlineFromMethods;
}

public LambdaUsageFinder(Lambda targetLambda, Map<Integer, Lambda> lambdaMap, AppView appView, LambdaInliner.LambdaUsageHandler lambdaUsageHandler) {
this(targetLambda, lambdaMap, appView, false, false, lambdaUsageHandler);
this.iterativeInstructionVisitor = new IterativeInstructionVisitor();
}

@Override
public void visitCodeAttribute(Clazz clazz, Method method, CodeAttribute codeAttribute) {
partialEvaluator.visitCodeAttribute(clazz, method, codeAttribute);
/*codeAttribute.instructionsAccept(clazz, method,
new InstructionOpCodeFilter(
new int[] {
Instruction.OP_INVOKESTATIC,
Instruction.OP_INVOKEVIRTUAL,
Instruction.OP_IRETURN,
Instruction.OP_LRETURN,
Instruction.OP_FRETURN,
Instruction.OP_DRETURN,
Instruction.OP_ARETURN
},
this
)
);*/
long lastTime = System.currentTimeMillis();
int offset = 0;
while (offset < codeAttribute.u4codeLength)
{
// Note that the instruction is only volatile.
Instruction instruction = InstructionFactory.create(codeAttribute.code, offset);
int instructionLength = instruction.length(offset);
HashSet<Byte> opcodes = new HashSet<>(Arrays.asList(Instruction.OP_INVOKESTATIC, Instruction.OP_INVOKEVIRTUAL));
if (opcodes.contains(instruction.opcode)) {
changed = false;
instruction.accept(clazz, method, codeAttribute, offset, this);
if (changed) {
System.out.println("Start another iteration");
//visitCodeAttribute(clazz, method, codeAttribute);
codeAttribute.accept(clazz, method, this);
break;
}
}
if (System.currentTimeMillis() > lastTime + 2000) {
System.out.printf("Progress %s/%s\n", offset, codeAttribute.u4codeLength);
lastTime = System.currentTimeMillis();
}
offset += instructionLength;
}
iterativeInstructionVisitor.instructionsAccept(
clazz,
method,
codeAttribute,
new InstructionOpCodeFilter(
new int[] {
Instruction.OP_INVOKESTATIC,
Instruction.OP_INVOKEVIRTUAL
},
this
)
);
}

@Override
Expand Down Expand Up @@ -160,19 +115,21 @@ private void findUsage(Clazz clazz, Method method, CodeAttribute codeAttribute,
}

if (match) {
changed = lambdaUsageHandler.handle(
targetLambda,
methodrefConstant.referencedClass,
methodrefConstant.referencedMethod,
offset,
clazz,
method,
codeAttribute,
trace,
leafNodes.stream().filter(it -> it.instruction().opcode == Instruction.OP_GETSTATIC).map(it -> {
ConstantInstruction getStaticInstruction = (ConstantInstruction) it.instruction();
return lambdaMap.get(getStaticInstruction.constantIndex);
}).collect(Collectors.toList())
iterativeInstructionVisitor.setChanged(
lambdaUsageHandler.handle(
targetLambda,
methodrefConstant.referencedClass,
methodrefConstant.referencedMethod,
offset,
clazz,
method,
codeAttribute,
trace,
leafNodes.stream().filter(it -> it.instruction().opcode == Instruction.OP_GETSTATIC).map(it -> {
ConstantInstruction getStaticInstruction = (ConstantInstruction) it.instruction();
return lambdaMap.get(getStaticInstruction.constantIndex);
}).collect(Collectors.toList())
)
);
}
}
Expand Down