How to pass image data from JS to WASM and vice versa

[TrOllOchamO]

Hello !
I'm currently trying to process image data on the browser using java code compiled to WASM.
To do so, I'm looking for a way to pass an Uint8Array from the JS side to WASM and get one in return, like below :

@Export("process")
public static byte[] process(byte[] data) {
  // process the data and return a new byte array
}

const imageData = new Uint8Array([1, 2, 3, 4]);
const processedImageData = bytecoder.instance.exports.process(0, imageData); // why should i pass 0 ?

The issue is that in the documentation it is said that supported types are :

• Primitives (int, float, double, long, char, byte, short)
• java.lang.String
• de.mirkosertic.bytecoder.api.OpaqueReferenceType and sub classes of it
• de.mirkosertic.bytecoder.api.Callback and sub classes of it

From what I understand from #433 and #542 it would be possible in my case to use an OpaqueReferenceType, but I wasn't able to find how I should use them.

Could you provide me (if it's possible) an example on how I could pass my data around ?

Also on a side note, I wasn't able to find an explanation on why we have to provide an unused parameter for all functions exported from WASM. Am I missing something ?

Thanks in advance,
--Barnabé

[TrOllOchamO]

Ok,
After some time looking at the sources, I thought I had something :

import de.mirkosertic.bytecoder.api.Export;
import de.mirkosertic.bytecoder.api.web.Int8Array;
import de.mirkosertic.bytecoder.api.web.OpaqueArrays;

public class ImageModifier {
  public static void main(String[] args) {
  }

  @Export("process")
  public static Int8Array process(Int8Array imageData) {
    byte[] bytes = int8ToByteArray(imageData);
    // do some image data processing
    int sum = 0;
    for(byte b: bytes) {
      sum += b;
    }
    System.out.println(sum);
    return byteArrayToInt8(bytes); // return the processed bytes (in this case the unchanged data)
  }

  private static byte[] int8ToByteArray(Int8Array int8Array) {
    final int arrayLen = int8Array.byteArrayLength();
    byte[] res = new byte[arrayLen];
    for (int i = 0; i < arrayLen; ++i) {
      res[i] = int8Array.getByte(i);
    }
    return res;
  }

  private static Int8Array byteArrayToInt8(byte[] byteArray) {
    final int arrayLen = byteArray.length;
    final Int8Array res = OpaqueArrays.createInt8Array(arrayLen);
    for (int i = 0; i < arrayLen; ++i) {
      res.setByte(i, byteArray[i]);
    }
    return res;
  }
}

// properly initialize everything with no errors
const array = new Int8Array([1, 2, 3, 4]);
const res = bytecoder.exports.process(null, array); // TypeError: bad type
console.log(res);

But it raises a TypeError when calling the exported process function, and I'm a little bit stuck here, because I don't see where the types are wrong. I tried to take a look at the .wat but was unable to understand what's the issue.

Thanks in advance for your help,
--Barnabé

[tzirit]

Yeah, I tried to do it in a similar way and got stuck like you.
I also tried turning the image into Base64 and accesing it that way in Java (directly as method parameter and by saving it to an innerHTML in JS and accessing it with .getElementById("e").innerHTML()) but ended up with the same error.

edit: I also worked with the bytecoder.toBytecoderString(), but I might have made a mistake while using it

[TrOllOchamO]

@tzirit
I've been doing some researches on Java with WebAssembly for my company lately, and I ended up using TeaVM.
I managed to pass arrays of data around and the solution is detailed in various issues I posted on their GitHub, all starting with [WASM]. You can take a look at those if you want, I even added repos to reproduces on some of them.

Quick disclaimer :
I failed to manipulate images in WebAssembly because none of the library I tried have compiled.
We gave up on the idea for now because rewriting the code to support all images format available in the world would be a HUGE pain. If your images are not too large, you might consider using the JS canvas API to convert the raw image data into a bitmap you could process easily in the WASM. Since our initial goal was to be able to manipulate large images without using the canvas API, we didn't investigate further.

If you have any questions on which I might have an answer don't hesitate to ping me, also the TeaVM maintainer is very nice and helpful.
Good luck,
--Barnabé