Problems with backported default and static methods

[pietrodev]

I'm trying to backport some default and static methods with new version 2.0.
PRE: The project I'm backporting is successfully backported with old version of retrolambda if I remove all default and static method. The project is also obfuscated with proguard 5.1 successfully.

When trying the new version I have two problems:

1. Proguard enabled: I got a long series of warnings when obfuscating:
   [java] Warning: org.plibrary.test.TestLinq$$Lambda$55: can't find referenced method 'boolean lambda$negate$38(org.plibrary.compat.Predicate,java.lang.Object)' in program class org.plibrary.compat.Predicate$

Note that "negate" is the name of the default method but no class has been generated with that name

2. Proguard disabled. I can compile but when I run tests I got:
   initializationError(org.plibrary.test.TestLinq)
   java.lang.ClassFormatError: Method lambda$and$27 in class org/plibrary/compat/Consumer has illegal modifiers: 0x1402

"and" is another default method and Consumer is an interface.

As an additional info these are the options I'm using as printed by retrolambda
[java] Retrolambda 2.0.0
[java] Bytecode version: 50 (Java 6)
[java] Default methods: true
[java] Input directory: build\classes
[java] Output directory: build\classes

I tried also with bytecode 51, nothing changes.

Any idea of what I can do?
Thanks.

[luontola]

Does it work when you do a clean build?

Please post the resulting .class files regarding those error messages.

[pietrodev]

All test are done by cleaning the whole project every time otherwise I obtain:
java.lang.ClassFormatError: Class file version does not support constant tag 15 in class file XXX.

I replicated the errors in this very simple project with only two Java files.

The obfuscation error is exaclty the same, the second error is instead: java.lang.ClassFormatError: Class file version does not support constant tag 15 in class file test/Interface$

I'm trying to post them here but it doesn't allow me, you can find them there http://pastebin.com/wdeq3HNY

[luontola]

Thanks. I had missed the possibility of having lambda expressions inside default methods and static methods on interfaces. Please try again with Retrolambda 2.0.1 and tell me if it works now. (It may take a couple of hours for 2.0.1 to show up in Maven Central, but in the meanwhile you can get it from the https://oss.sonatype.org/content/groups/public/ repository.)

[pietrodev]

Hi, thanks for the reply.
With retrolambda 2.0.1 the test classes compile again and also compilation of my project works, but obfuscation still fails. Here you are the minimal test set where proguard fails:
http://pastebin.com/Da77y1DB

The error is:
[java] Warning: test.BooleanSupplier$: can't find referenced method 'test.BooleanSupplier lambdaFactory$(test.BooleanSupplier)' in program class test.BooleanSupplier$$Lambda$4

Note that if I remove any method (e.g., getFalse() or toSupplier()) from the interface the obfuscation is successful. Only with all the methods (2 static and 2 default methods) it fails.

[luontola]

Hmm. I can't get it to fail on just Java, so it's something obfuscation related.

I found one possible source of confusion for proguard: some generic method signatures were not updated to match their erased method descriptor. Please try if this build fixes the problem: https://oss.sonatype.org/content/groups/public/net/orfjackal/retrolambda/retrolambda/2.0.2-SNAPSHOT/retrolambda-2.0.2-20150408.191342-1.jar

[luontola]

What are your ProGuard settings? I tried processing your example classes with ProGuard, using the default settings in proguardgui.jar, but did not get any warnings about that.

There's a possibly non-deterministic hack in Retrolambda which causes lambdas in interfaces to be backported twice, but I would like to reproduce this issue before attempting to fix that hack (especially if it's not the root cause).

[pietrodev]

No, unfortunately the error is the same.

[java] Warning: test.BooleanSupplier$: can't find referenced method 'test.BooleanSupplier lambdaFactory$(test.BooleanSupplier)' in program class test.BooleanSupplier$$Lambda$4

If it can help, I decompiled the generated code and found that in the specified class (test.BooleanSupplier$$Lambda$4) there isn't such a method, but there is a similar one:
public static BooleanSupplier lambdaFactory$() {}
with the BooleanSupplier parameter missing, which instead can be found in class test.BooleanSupplier$$Lambda$2:
public static BooleanSupplier lambdaFactory$(BooleanSupplier arg0) {}

Note that, as I wrote in the other post, if I remove any method it works, if I decompile class test.BooleanSupplier$$Lambda$4 I find:
public static BooleanSupplier lambdaFactory$(BooleanSupplier arg0) {}

So the method exists when changing the source file, that's the reason it works. With the exact source as I posted the method is instead missing.

Thanks for your time and for retrolambda.

[luontola]

Thanks. The mixing up of Lambda$4 and Lambda$2 seems to suggest that the cause is the hack I mentioned earlier. I'll have a look at solving it.

[pietrodev]

This is the proguard configuration: http://pastebin.com/KTi8Fgup
it's the default configuration for libraries, honestly I didn't try with other configurations because it always worked also with retrolambda until I started using the default methods :-)

[luontola]

I'm unable to reproduce this on my machine, so please try if snapshot fixes it for you: https://oss.sonatype.org/content/groups/public/net/orfjackal/retrolambda/retrolambda/2.0.2-SNAPSHOT/retrolambda-2.0.2-20150408.201435-2.jar

[pietrodev]

The test project now obfuscate fine, but coming back to the main library it still gives a lot of errors:

[java] Warning: org.plibrary.dao.DAOObject$$Lambda$1: can't find referenced method 'boolean lambda$xor$39(org.plibrary.compat.Predicate,org.plibrary.compat.Predicate,java.lang.Object)' in program class org.plibrary.compat.Predicate$

If I decompile the class it has only the method:
public abstract Predicate xor(Predicate<? super T> paramPredicate);
without the number 39.

All the errors refers to methods with number, but no number in generated classes. I do not know it it is related to the other bug

Tomorrow I'll prepare a runnable package with all the requirements to reproduce on other machines :-)
Thanks.

[luontola]

Are you sure that you decompiled Predicate$ and not Predicate? Have a look at it with the javap command: javap -v -p Predicate\$.class

[pietrodev]

Strange, with javap I have

static boolean lambda$xor$39(org.plibrary.compat.Predicate, org.plibrary.compat.Predicate, java.lang.Object);

but decompiling the sources with jd-gui gives me no numbers.

I need to make something you can replicate.
Thanks.

[luontola]

I wonder whether the error is because that the method is package-private, but it appears to be used from another package. Please send me the .class files org.plibrary.dao.DAOObject$$Lambda$1 and org.plibrary.compat.Predicate$ and also the source code for DAOObject and Predicate.

[luontola]

Here is a snapshot build that will make that method public, but the real fix would be to not generate the method that calls it in the first place. Try whether this one fixes it and whether there are still more problems. I'll work on a proper fix tomorrow or later this week.
https://oss.sonatype.org/content/groups/public/net/orfjackal/retrolambda/retrolambda/2.0.2-SNAPSHOT/retrolambda-2.0.2-20150408.205119-3.jar

[pietrodev]

You are definitely right. The problem is that proguard can't find the package-private methods. I replicated the error by simply using a lambda from a class in another package. Here you find the whole test project with sources, classes, proguard jar and an executable batch to launch obfuscation which will cause the error (change the rt.jar path).
With your latest snapshot the obfuscation goes right.
Waiting for the final fix. Thanks.

[luontola]

This is now fixed in Retrolambda 2.0.2. This one issue covered 4 different bugs. 😅 If you still find more, please create a new issue.

[pietrodev]

OK it works. Sorry for the multiple requests but at the beginning I thought all bugs were related.
Thanks!

[luontola]

No problem. They all were related to the new default methods feature. I expected some bugs to remain in v2.0.0, due to the huge amount of language features which may possibly interact with default methods, but even then was surprised at how many bugs there were. 😆

[denis-itskovich]

I think I still see this issue:

Proguard seems to shrink methods, referenced as method references. See an example below:

public class AsyncTaskBuilder<Progress, Result> {

    public interface BackgroundWorkWithProgress<_Progress, _Result> {
        _Result execute(ProgressObserver<_Progress> progressObserver);
    }

    private BackgroundWorkWithProgress<Progress, Result> work;

    public AsyncTask<Void, Progress, Result> build() {

        return new AsyncTask<Void, Progress, Result>() {
            @Override
            protected Result doInBackground(Void... voids) {
                // this reference will not be taken into account by Proguard
                return work.execute(this::publishProgress);
            }

        };
    }
}

After building, the following error is shown:

:app:proguardRelease
Warning: com.github.retrolambda_methodref_repro.AsyncTaskBuilder$1$$Lambda$1: can't find referenced method 'void publishProgress(java.lang.Object[])' in library class android.os.AsyncTask
Warning: there were 1 unresolved references to library class members.
         You probably need to update the library versions.
         (http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedlibraryclassmember)
Exception while processing task 
java.io.IOException: Please correct the above warnings first.
    at proguard.Initializer.execute(Initializer.java:473)
    at proguard.ProGuard.initialize(ProGuard.java:233)
    at proguard.ProGuard.execute(ProGuard.java:98)
    at proguard.gradle.ProGuardTask.proguard(ProGuardTask.java:1074)
    at com.android.build.gradle.tasks.AndroidProGuardTask.doMinification(AndroidProGuardTask.java:139)
    at com.android.build.gradle.tasks.AndroidProGuardTask$1.run(AndroidProGuardTask.java:115)
    at com.android.builder.tasks.Job.runTask(Job.java:48)
    at com.android.build.gradle.tasks.SimpleWorkQueue$EmptyThreadContext.runTask(SimpleWorkQueue.java:41)
    at com.android.builder.tasks.WorkQueue.run(WorkQueue.java:227)
    at java.lang.Thread.run(Thread.java:745)

Full log is available at travis-ci

Simple android project (with 1 class), reproducing the issue can be found here:
https://github.com/denis-itskovich/retrolambda-method-ref-repro