Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up Refaster bug checker #261

Draft
wants to merge 58 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
69386f0
Emit website link along with Refaster refactor suggestions
Stephan202 Sep 21, 2022
84061fc
More extensible approach
Stephan202 Sep 22, 2022
827880b
WIP: Some plumbing for annotation support
Stephan202 Sep 22, 2022
0f44844
Another round
Stephan202 Sep 22, 2022
0e46f9c
Tweaks
Stephan202 Sep 23, 2022
914d30a
Better annotation support, simplify setup
Stephan202 Sep 23, 2022
458fb99
Flag why build currently doesn't fail, while it should.
Stephan202 Sep 23, 2022
db8cf3c
Improve logic and test coverage
Stephan202 Sep 24, 2022
abb6cea
Properly document URL placeholder usage
Stephan202 Sep 24, 2022
cf772c4
Expand test coverage
Stephan202 Sep 24, 2022
d26bc18
Flag classpath issue
Stephan202 Sep 25, 2022
01b7e5b
Improve text and minor improvements
rickie Sep 27, 2022
3482641
Use `@AutoValue` for the `AnnotatedCompositeCodeTransformer`
rickie Sep 29, 2022
12d09ad
Support `AllSuggestionsAsWarnings` and add a suggestion
rickie Sep 29, 2022
74d6c9a
Tweak
rickie Sep 29, 2022
c4b6a5f
Suggestions, introduce `ErrorProneFork`
Stephan202 Sep 29, 2022
d899a8a
Also this
Stephan202 Sep 29, 2022
20d194b
Clarify how the default Refaster hit severity comes to be
Stephan202 Sep 30, 2022
ad1c98d
Align documentation of reported description names by the Refaster check
Badbond Sep 30, 2022
50443d1
Suggestion
Badbond Sep 30, 2022
7e49b08
Pass null for urlLink to Description.Builder
Badbond Sep 30, 2022
594f51c
Tweaks
Stephan202 Sep 30, 2022
e6caceb
Move `AnnotatedCompositeCodeTransformer` and `ErrorProneFork` to `ref…
rickie Oct 4, 2022
b2ef631
`s/information/content/`
rickie Oct 4, 2022
8bd88bb
Improve Javadoc `AnnotatedCompositeCodeTransformer`
rickie Oct 4, 2022
302e20b
Revert changes in `OptionalTemplates`
rickie Oct 4, 2022
32300ff
Tweak `AnnotatedCompositeCodeTransformer` Javadoc
Badbond Oct 4, 2022
d21ac59
Apply `StringJoin` suggestion
Stephan202 Oct 7, 2022
0484762
Suggestions
Stephan202 Oct 8, 2022
1624ebf
WIP: Speed up `Refaster` check
Stephan202 May 1, 2022
ee74279
Compatibility with stock Error Prone
Stephan202 Sep 12, 2022
344f4e4
Initial copy over of the improved algorithm
rickie Sep 23, 2022
6739cb4
Improve code and algorithm for Refaster
rickie Sep 23, 2022
72ff8ae
Drop unnecessary `@AutoService` annotation
rickie Sep 23, 2022
39dc9aa
Add XXXs
rickie Sep 23, 2022
2a93011
Merge `RefasterRuleSelector` type hierarchy
Stephan202 Sep 25, 2022
1c5077f
Create selector only once per `Refaster` instantiation
Stephan202 Sep 25, 2022
2c137c0
Move all `RefasterRuleSelector` construction logic into the relevant …
Stephan202 Sep 25, 2022
c3a5106
Push sorting requirements into `Node`, minimize tree, add tests
Stephan202 Sep 25, 2022
9151287
Suggestion
rickie Sep 27, 2022
a27d635
Extract the `TreeScanner`s to their own classes
rickie Sep 27, 2022
2003bd2
Introduce `getClass` method to deduplicate
rickie Sep 27, 2022
7889148
Reorder methods in `RefasterIntrospection`
rickie Sep 27, 2022
afdcf52
Apply some suggestions
Stephan202 Sep 30, 2022
ab03739
Optimize code, introduce benchmark, simplify test
Stephan202 Oct 1, 2022
457a8e2
Comment style, explain both performance-only pieces of code
Stephan202 Oct 2, 2022
63ad14e
Fix typo
rickie Oct 4, 2022
0cf891c
Suggestions
ibabiankou Oct 7, 2022
2cb4bd0
Suggestions
Stephan202 Oct 8, 2022
c34fa4d
Fix typo
rickie Oct 9, 2022
d59b626
Merge remote-tracking branch 'origin/sschroevers/refaster-custom-urls…
Stephan202 Oct 9, 2022
5ba7075
Introduce `AnnotatedCompositeCodeTransformerTest`
Stephan202 Oct 9, 2022
0157692
Merge remote-tracking branch 'origin/sschroevers/refaster-custom-urls…
Stephan202 Oct 9, 2022
72f8881
Suggestions
Stephan202 Oct 9, 2022
77bc107
Merge branch 'master' into sschroevers/refaster-speed-up
Stephan202 Dec 26, 2023
2687322
Merge branch master into sschroevers/refaster-speed-up
rickie Dec 24, 2024
424b520
Merge branch 'master' into sschroevers/refaster-speed-up
rickie Dec 24, 2024
8d99611
Post-merge-fix
rickie Dec 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Suggestions
  • Loading branch information
ibabiankou authored and Stephan202 committed Oct 8, 2022
commit 0cf891c87b3249726b0f7592fae5f9ce48b226c5
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import static java.util.Comparator.comparingInt;

import com.google.auto.value.AutoValue;
import com.google.common.collect.ImmutableCollection;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSortedSet;
Expand All @@ -24,7 +25,8 @@
@AutoValue
abstract class Node<T> {
static <T> Node<T> create(
List<T> values, Function<? super T, ? extends Set<? extends Set<String>>> pathExtractor) {
ImmutableCollection<T> values,
Function<? super T, ? extends Set<? extends Set<String>>> pathExtractor) {
BuildNode<T> tree = BuildNode.create();
tree.register(values, pathExtractor);
return tree.immutable();
Expand Down Expand Up @@ -89,7 +91,8 @@ private static <T> BuildNode<T> create() {
* leads to the same value.
*/
private void register(
List<T> values, Function<? super T, ? extends Set<? extends Set<String>>> pathsExtractor) {
ImmutableCollection<T> values,
Function<? super T, ? extends Set<? extends Set<String>>> pathsExtractor) {
for (T value : values) {
List<? extends Set<String>> paths = new ArrayList<>(pathsExtractor.apply(value));
/*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import com.google.common.collect.TreeRangeSet;
import com.google.errorprone.BugPattern;
import com.google.errorprone.CodeTransformer;
import com.google.errorprone.CompositeCodeTransformer;
import com.google.errorprone.ErrorProneFlags;
import com.google.errorprone.SubContext;
import com.google.errorprone.VisitorState;
Expand All @@ -34,6 +35,7 @@
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.function.Consumer;
import java.util.regex.Pattern;
import java.util.stream.Stream;

Expand Down Expand Up @@ -162,13 +164,35 @@ private static Stream<Replacement> getReplacements(
private static RefasterRuleSelector createRefasterRuleSelector(ErrorProneFlags flags) {
ImmutableListMultimap<String, CodeTransformer> allTransformers =
CodeTransformers.getAllCodeTransformers();
return RefasterRuleSelector.create(
List<RefasterRule<?, ?>> refasterRules = new ArrayList<>();
collectRefasterRules(
flags
.get(INCLUDED_TEMPLATES_PATTERN_FLAG)
.map(Pattern::compile)
.<ImmutableCollection<CodeTransformer>>map(
nameFilter -> filterCodeTransformers(allTransformers, nameFilter))
.orElseGet(allTransformers::values));
.orElseGet(allTransformers::values),
refasterRules::add);
return RefasterRuleSelector.create(ImmutableList.copyOf(refasterRules));
}

private static void collectRefasterRules(
ImmutableCollection<CodeTransformer> transformers, Consumer<RefasterRule<?, ?>> sink) {
for (CodeTransformer t : transformers) {
collectRefasterRules(t, sink);
}
}

private static void collectRefasterRules(
CodeTransformer transformer, Consumer<RefasterRule<?, ?>> sink) {
if (transformer instanceof RefasterRule) {
sink.accept((RefasterRule<?, ?>) transformer);
} else if (transformer instanceof CompositeCodeTransformer) {
collectRefasterRules(((CompositeCodeTransformer) transformer).transformers(), sink);
} else {
throw new IllegalStateException(
String.format("Can't handle `CodeTransformer` of type '%s'", transformer.getClass()));
}
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ibabiankou I agree that with the PR as-is it was an improvement to move this RefasterRule extraction logic here, but the changes in #255 make the CodeTransformer -> ImmutableSet<ImmutableSet<String>> transformation less straightforward, so I'll move it back. Of course, idea on how to better do this are welcome!


private static ImmutableList<CodeTransformer> filterCodeTransformers(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@
import com.google.common.collect.ImmutableCollection;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.google.errorprone.CodeTransformer;
import com.google.errorprone.CompositeCodeTransformer;
import com.google.errorprone.refaster.BlockTemplate;
import com.google.errorprone.refaster.ExpressionTemplate;
import com.google.errorprone.refaster.RefasterRule;
Expand Down Expand Up @@ -39,7 +37,6 @@
import java.util.IdentityHashMap;
import java.util.List;
import java.util.Set;
import java.util.function.Consumer;
import javax.annotation.Nullable;

// XXX: Add some examples of which source files would match what templates in the tree.
Expand All @@ -60,8 +57,8 @@
* <li>Extract all identifiers from the {@link CompilationUnitTree} and sort them
* lexicographically.
* <li>Traverse the tree based on the identifiers from the {@link CompilationUnitTree}. Every node
* can contain Refaster templates. Once a node is we found a candidate Refaster template and
* will therefore be added to the list of candidates.
* can contain Refaster templates. Once a node is we found a candidate Refaster template that
* might match some code and will therefore be added to the list of candidates.
* </ol>
*
* <p>This is an example to explain the algorithm. Consider the templates with identifiers; {@code
Expand All @@ -77,14 +74,14 @@
* └── D -- T3
* }</pre>
*
* <p>The tree is traversed based on the identifiers in the {@link CompilationUnitTree}. When a leaf
* contains a template and is reached, we can be certain that the identifiers from the {@link
* <p>The tree is traversed based on the identifiers in the {@link CompilationUnitTree}. When a node
* containing a template is reached, we can be certain that the identifiers from the {@link
* BeforeTemplate} are at least present in the {@link CompilationUnitTree}.
*
* <p>Since the identifiers are sorted, we can prune parts of the {@link Node tree} while we are
* <p>Since the identifiers are sorted, we can skip parts of the {@link Node tree} while we are
* traversing it. Instead of trying to match all Refaster templates against every expression in a
* {@link CompilationUnitTree} we now only return a subset of the templates that at least have a
* chance of matching. As a result, the performance of Refaster significantly increases.
* {@link CompilationUnitTree} we now only matching a subset of the templates that at least have a
* chance of matching. As a result, the performance of Refaster increases significantly.
*/
final class RefasterRuleSelector {
private final Node<RefasterRule<?, ?>> treeRules;
Expand All @@ -93,36 +90,12 @@ private RefasterRuleSelector(Node<RefasterRule<?, ?>> treeRules) {
this.treeRules = treeRules;
}

/**
* Instantiates a new {@link RefasterRuleSelector} backed by the {@link RefasterRule}s referenced
* by the given collection of {@link CodeTransformer}s.
*/
static RefasterRuleSelector create(ImmutableCollection<CodeTransformer> transformers) {
List<RefasterRule<?, ?>> refasterRules = new ArrayList<>();
collectRefasterRules(transformers, refasterRules::add);
/** Instantiates a new {@link RefasterRuleSelector} backed by the given {@link RefasterRule}s. */
static RefasterRuleSelector create(ImmutableCollection<RefasterRule<?, ?>> refasterRules) {
return new RefasterRuleSelector(
Node.create(refasterRules, RefasterRuleSelector::extractTemplateIdentifiers));
}

private static void collectRefasterRules(
ImmutableCollection<CodeTransformer> transformers, Consumer<RefasterRule<?, ?>> sink) {
for (CodeTransformer t : transformers) {
collectRefasterRules(t, sink);
}
}

private static void collectRefasterRules(
CodeTransformer transformer, Consumer<RefasterRule<?, ?>> sink) {
if (transformer instanceof RefasterRule) {
sink.accept((RefasterRule<?, ?>) transformer);
} else if (transformer instanceof CompositeCodeTransformer) {
collectRefasterRules(((CompositeCodeTransformer) transformer).transformers(), sink);
} else {
throw new IllegalStateException(
String.format("Can't handle `CodeTransformer` of type '%s'", transformer.getClass()));
}
}

/**
* Retrieve a set of Refaster templates that can possibly match based on a {@link
* CompilationUnitTree}.
Expand All @@ -135,7 +108,6 @@ private static void collectRefasterRules(
Set<RefasterRule<?, ?>> selectCandidateRules(CompilationUnitTree tree) {
Set<RefasterRule<?, ?>> candidateRules = newSetFromMap(new IdentityHashMap<>());
treeRules.collectReachableValues(extractSourceIdentifiers(tree), candidateRules::add);

return candidateRules;
}

Expand Down Expand Up @@ -166,17 +138,13 @@ private static ImmutableSet<ImmutableSet<String>> extractTemplateIdentifiers(
ImmutableList<? extends Tree> trees) {
List<Set<String>> identifierCombinations = new ArrayList<>();
identifierCombinations.add(new HashSet<>());

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to add an empty set here manually? I think it is worth adding a comment.


new TemplateIdentifierExtractor().scan(trees, identifierCombinations);

TemplateIdentifierExtractor.INSTANCE.scan(trees, identifierCombinations);
return identifierCombinations.stream().map(ImmutableSet::copyOf).collect(toImmutableSet());
}

private static Set<String> extractSourceIdentifiers(Tree tree) {
Set<String> identifiers = new HashSet<>();

new SourceIdentifierExtractor().scan(tree, identifiers);

SourceIdentifierExtractor.INSTANCE.scan(tree, identifiers);
return identifiers;
}

Expand Down Expand Up @@ -354,6 +322,8 @@ private static <T> T invokeMethod(Method method, Object instance) {
}

private static class TemplateIdentifierExtractor extends TreeScanner<Void, List<Set<String>>> {
private static final TemplateIdentifierExtractor INSTANCE = new TemplateIdentifierExtractor();

@Nullable
@Override
public Void visitIdentifier(IdentifierTree node, List<Set<String>> identifierCombinations) {
Expand Down Expand Up @@ -434,7 +404,8 @@ public Void visitBinary(BinaryTree node, List<Set<String>> identifierCombination

private static void registerOperator(
ExpressionTree node, List<Set<String>> identifierCombinations) {
identifierCombinations.forEach(ids -> ids.add(treeKindToString(node.getKind())));
String id = treeKindToString(node.getKind());
identifierCombinations.forEach(ids -> ids.add(id));
}

@Nullable
Expand Down Expand Up @@ -462,6 +433,8 @@ private static List<Set<String>> copy(List<Set<String>> identifierCombinations)
}

private static class SourceIdentifierExtractor extends TreeScanner<Void, Set<String>> {
private static final SourceIdentifierExtractor INSTANCE = new SourceIdentifierExtractor();

@Nullable
@Override
public Void visitPackage(PackageTree node, Set<String> identifiers) {
Expand Down