Linter javac painfully slow

[noseglid]

Thank you for maintaining this package, it's a good step in improving Java development on Atom. (I'm maintaining other efforts).

To the issue. A few (legacy) projects I work on have quite a few java files in the project (read 100s). The way linter-javac works now it recompiles them all on every change, which is very resource intensive and takes a very long time.

I think to make this linter useful for larger projects, it must be more intelligent. I think it should be possible to use the build in tokenizer in java (as I'm doing in autocomplete-java-minus) to intelligently figure out which classes needs to be included in the compile command.

This would also help with the E2BIG issue which have been "resolved" recently. It no longer errors, but leaves out necessary classes giving incorrect lint messages.

I understand this is not a quick-fix, but I wanted to raise this issue where we can discuss how to improve performance. Possibly there are other things we can do to make it faster?

[florianb]

Thank you for giving feedback and sharing your experience - i like your packages! ⭐

I also appreciate all contributions which were made so far. Since i initially created this package a lot of necessary functionality was added. At those days i created linter-javac to lint on the fly. And i had the strong feeling to have not enough experience or knowledge around the javac- and Atom-internals to make the step from a "light"-linter (missing class-references) to a full-stuffed linter and keep the lint-on-the-fly feature.

However, some time has gone, i tried to iron out as much bugs as possible and after it seems that we reached kind of a mature state (in terms of less bugs are popping up - hopefully) i would really like to refactor the whole thing (see #76). Now to your points:

ℹ️ The E2BIG-issue is in the state "quick fixed" because i hadn't had any better idea how to finally fix it without rewriting the whole call-stack. Issuing the whole linter via the command line is a bad idea in general and i would really like to avoid that in the future in general.

The main problem to the full-featured linter is the performant resolvement of class-references. I think this contains the following sub-problems:

1. consider all possible sources for class-files
2. provide javac access to a possibly infinite amount of class-files
3. do all of this as fail-save and performant as possible

If i understood you correctly your point would belong to 3, since it would possibly help us to reduce the amount of needed class-references. This would be helpful in the current stadium of the package. The caveat i see is that the package would need to understand which class is requested and in which file it could be found.

This sounds like hard work for me. The current idea we had is to give javac access to all necessary source paths (using the sourcepath-option and let javac decide which one is taken.
This would also reduce the current effort to collect all .java-files which should be very slow (i didn't benchmark that).

I would also prefer if linter-javac could use already existing classpath-config-information (f.e. from Gradle or Maven).

I would really appreciate your feedback, i am interested in the best possible solution for the rewrite.

[noseglid]

Thanks for the fast and elaborate answer. I too think this has reached a quite stable points, which means it is time to take it to the next level! 😄

The sourcepath option seems very attractive. I tested it on a few of my (larger) projects and it brings down the compilation time to about ~18s from previous ~55s:

With -sourcepath

7.04 real        12.99 user         0.75 sys
5.18 real        12.30 user         0.63 sys
5.32 real        12.15 user         0.62 sys
5.08 real        12.37 user         0.61 sys
5.55 real        12.51 user         0.62 sys
5.19 real        12.20 user         0.61 sys

Without -sourcepath (recompiling everything)

19.13 real        45.53 user         1.93 sys
15.33 real        43.34 user         1.50 sys
15.39 real        41.63 user         1.57 sys
15.94 real        41.74 user         1.55 sys
14.60 real        40.78 user         1.51 sys

So that's a great thing to do!

I am also very much in favor of doing #76 . Especially the ES6 part. In fact, I did exactly that a while back on build. I even wrote a blog post about it. I have never looked back since. CoffeeScript is the spawn of the devil 😈 .

With autocomplete-java-minus (or really before) I built a javascript parser of class files. I call it jdjs (Java Decompiler JavaScript). There's not much in the way of documentation yet, but essentially reads JAR and class files and returns easy-to-program around JavaScript objects classes, fields, methods, inheritance etc. I use it in autocomplete-java-minus to provide autocomplete suggestions. Perhaps it could be used by the linter to figure out which classpaths it must include in a compile command.

I don't see having to create .classpath as a huge problem. In fact, I kinda like it. It gives me good control. I'm facilitating it by having /.classpath in my ~/.gitignore, and the following gradle task in my ~/.gradle/init.gradle:

    task classpath << {
        Set deps = configurations.runtime.getFiles()
        for (set in sourceSets) {
          deps += set.runtimeClasspath;
        }

        File classpath = new File(".classpath")
        classpath.text = deps.join(':')
    }

Which automatically gets the classpath from gradle and writes it in .classpath.