Environment Lights by ThomasGorisse · Pull Request #156 · SceneView/sceneform-android

ThomasGorisse · 2021-09-02T11:21:00Z

I have to say that I'm very enthusiastic about this enhancement and here is why:

Everything we can see, every beauty comes from lights. They are the main factor of every visual things and so our physical rendering SDK. From spectacular to realistic results, everything is about lighting.
Sceneform had a big lack on this part and was archived just after the ARCore HDR lighting release.
Sorry for original developers on this part but just as a quick example setUseHdrLightEstimate(boolean) was defined in 5 different classes calling each others every frames.
As an other example, the directional light retrieved from ARCore was implemented as a SunNode which could have been great considering it should correspond to an anchor in the real world. But the fact is that it was enormously frame consuming for just handling a direction and intensity (you will be convinced when testing this release).
This PR is the entry point to Kotlin migration and every new features will also be written in Kotlin.
This will bring a lot to Sceneform Maintained which can actually be pretty hard to understand since everything is fragmented in not very clear classes.
As a quick example, this PR removes 5 unnecessary/understandable classes in favor of a unique EnvironmentLights.kt with only 350 lines of code and only 4 internal calls: EnvironmentLights(scene: Scene, assetManager: AssetManager), setSessionConfig(config: Config), doFrame(frame: Frame) and destroy()
The old LightProbe was the last SFB user so this PR is opening the gate to a big cleanup of unused classes and most important, we will be able to remove the SFB, JNI and lull parts.

Let's stop speaking and see what a good light management is bringing

Help wanted!!!!

Lightning could just be about applying the right light physical functions like Filament is doing very nicely (if you want to go further: Filament Materials Guide) but Sceneform Maintained is facing the challenge of applying ARCore guessed values to Filament concrete values.

Over than letting users chose from Realistic or Spectacular depending on their usage, we inevitably have to define a static ajustement factor value between the estimation and the rendering.
For that, here comes, included in this PR, a good friend app that let's you adjust indirect/directional lights, enabled ARCore light estimation modes and share annotated screenshots.

@RGregat @imbrig @grassydragon @issacclee @Venthorus @miftahulbagusp @suryanadiminti99 @fvito @KonsanAlide @Methew5 @Oleur @senthil5053 @D-Lav @VojtaMaiwald @SimonMarquis or anyone seeing this PR, we need as much as possible annotated screenshots of various lighting conditions to refine the magic conversion numbers.

Could you install the sample app available or soon (pending publication validation when writing) here Google Play - AR Environment lights and send your screenshots here?
The sample source code is in samples/environment-lights

@devbridie Could you have a quick look here to confirm that every lighting estimation is called correctly:
https://github.com/ThomasGorisse/sceneform-android-sdk/blob/56b281da1d73002a33148709580da27249ca2add/core/src/main/java/com/google/ar/sceneform/lights/EnvironmentLights.kt#L1-L396

@prideout @pixelflinger @romainguy @bejado I have annotated with todos some parts of this PR concerning the camera exposure, defaults indirect/directional light intensities and ARCore cubemap to reflections texture parts, would you mind if I mention you on the review comments?

Sorry for the spam but this PR is a big entry point to a new world for Sceneform Maintained and we want to be sure to have a good/clean restart.

Thanks

devbridie · 2021-09-02T11:26:50Z

Just taking a precursory look for now, but the results look fantastic! Great work here.
I'm going to be out of office for the next two weeks but will try to slot in some time to look over the implementation.

issacclee · 2021-09-02T14:18:46Z

Absolutely bonkers!

ThomasGorisse · 2021-09-02T14:24:04Z

@devbridie You (ARCore team) are half part (Filament for the other half) responsible of those great results and it confirms one more time that your both versions are better and better.
BTW, thanks for your quick configuration push when we have a device issue.
And I also confirm that a lot of leaks are gone with the 1.26 specially with the Image reader.

romainguy · 2021-09-02T16:47:38Z

Some info about this:

//    private static final float ARCORE_HDR_LIGHTING_CAMERA_APERATURE = 1.0f;
//    private static final float ARCORE_HDR_LIGHTING_CAMERA_SHUTTER_SPEED = 1.2f;
//    private static final float ARCORE_HDR_LIGHTING_CAMERA_ISO = 100.0f;

Filament uses a physically based camera model, with aperture, shutter speed, and sensitivity (ISO). This plays nicely with physical light units (for instance if you measure the sun outside on a sunny day in California you'll get ~110,000 lux and if you set your camera settings to ƒ/16 1/125s ISO 100, exposure will be correct). However many engines use "relative exposure" instead, where exposure is just a float and set to 1.0 by default. ARCore's light estimation was designed to work with those engines and that's why the constants above are what Sceneform uses. When used together, these constants yield an exposure factor of 1.0. There is now an API in Filament (Camera.setExposure(float)) which does this for you, but all it does internally is:

    void setExposure(float exposure) noexcept {
        setExposure(1.0f, 1.2f, 100.0f * (1.0f / exposure));
    }

So you have two choices: keep a relative exposure model or scale light intensities. To do so you should be able to multiply the values provided by ARCore by 1.0 / log2((aperture * aperture) / shutterSpeed * 100.0f / sensitivity) (source). This should raise ARCore's values to something that's physically plausible for the chosen exposure.

Filament also provides a (native) library called libiblprefilter that lets you do what cmgen does at runtime, but on the GPU. You may find it useful if you want developers to be able to load an arbitrary env map at runtime. On the Java side, the API is part of filament-utils-android, see IBLPrefilterContext.

RGregat · 2021-09-02T18:44:00Z

Awesome work! I do my best to collect as many annotated screenshot as possible!

core/src/main/java/com/google/ar/sceneform/lights/EnvironmentLights.kt

romainguy · 2021-09-03T15:08:35Z

core/src/main/java/com/google/ar/sceneform/lights/EnvironmentLights.kt

+            val rgbaBuffer = this[index].planes[0].buffer
+            while (rgbaBuffer.hasRemaining()) {
+                for (byteIndex in 0 until rgbaBytesPerPixel) {
+                    val byte = rgbaBuffer.get()


Don't do this copy byte by byte, get an array of byte at a time (32 KiB for instance). Even better, just use ByteBuffer.put(ByteBuffer) :)

I can't find a better way to skip the alpha channel than that:

ByteBuffer.allocateDirect( arImages[0].width * arImages[0].height * arImages.size * // RGB Bytes per pixel 6 * 2 ).apply { // Use the device hardware's native byte order order(ByteOrder.nativeOrder()) val rgbaBytes = ByteArray(8) // ARGB Bytes per pixel arImages.forEach { image -> image.planes[0].buffer.let { imageBuffer -> while (imageBuffer.hasRemaining()) { // Only take the RGB channels put(rgbaBytes.apply { imageBuffer.get(this) } // Skip the Alpha channel .sliceArray(0..5)) } } } rewind() }

I have to do this because generatePrefilterMipmap() takes a Texture.Format.RGB and the ARImage is in RGBA16F
Do you see something I could do to make it cleaner?

core/src/main/java/com/google/ar/sceneform/lights/EnvironmentLights.kt

ThomasGorisse · 2021-09-05T00:59:54Z

Thanks for the clarification and advice.
I will respond to the line code comments after pushing.

I'm quite near from the final result but I need one or two more nights with my bedside book/bible
I'm not physically based rendering engine fluent but I think I can now at least understand a Filament team discussion.

Forced directional light color for test purposes to (RGB) 1.0, 0.0 ,0.0

Still some work to do on the spherical harmonics and reflections.

@romainguy

So you have two choices: keep a relative exposure model or scale light intensities. To do so you should be able to multiply the values provided by ARCore by 1.0 / log2((aperture * aperture) / shutterSpeed * 100.0f / sensitivity) (source). This should raise ARCore's values to something that's physically plausible for the chosen exposure.

From what I saw on different engines (mostly Unreal - Physical Lighting Units), the unit-less intensity has been deprecated in favor to cd and lux.
More over, that makes the sdk lighting usage more constant/consistent for different cases including non AR usage.
So I went to the intensity scaling solution.

val Camera.ev100: Float
    get() = log2((aperture * aperture) / shutterSpeed * 100.0f / sensitivity)

val Camera.exposureFactor get() = 1.0f / ev100

Filament also provides a (native) library called libiblprefilter that lets you do what cmgen does at runtime, but on the GPU. You may find it useful if you want developers to be able to load an arbitrary env map at runtime. On the Java side, the API is part of filament-utils-android, see IBLPrefilterContext.

/**
 * ### Utility for decoding an HDR file and producing a Filament environment resources.
 *
 * Consumes the content of an HDR file and optionaly produces the results
 *
 * @param reflections the reflections texture with a specular filter. Keep it null if you don't need
 * it.
 * @param ibl the indirect light. Keep it null if you don't need it.
 * @param skybox the environment map. Keep it null if you don't need it.
 */
fun HDRLoader.loadHdr(
    assets: AssetManager,
    hdrFileName: String,
    reflections: ((Texture) -> Unit)? = null,
    ibl: ((IndirectLight) -> Unit)? = null,
    skybox: ((Skybox) -> Unit)? = null,
    error: (Exception) -> Unit = {}
)

...and I confirm that I'm using only one IBLPrefilterContext, EquirectangularToCubemap, SpecularFilter instance per Engine context.

Do you think it's reasonable to apply the SpecularFilter (Launch the heaver computation. Expect 100-100ms on the GPU.) to the AR ML Cubemap or should I just call the setImage() on the unique Texture instance ?
I will try to include a boolean in order to activate/deactivate it from the sample.

romainguy · 2021-09-05T01:34:36Z

The ARCore ML cubemap is small (16x16 per face), it seems perfectly reasonable to run it on the GPU, it shouldn't take 100ms. It will however take GPU time away from scene rendering, so it's a tradeoff. Texture.generatePrefilterMipmap() will do the same as libiblprefilter, except on the CPU. It was meant for ARCore's ML cubemaps and should take ~1ms on a Pixel 4 for a 16x16 cubemap.

KonsanAlide · 2021-09-07T09:20:30Z

@ThomasGorisse Hi Thomas, i'm sorry for my lateness, i've been busy with some company affairs these days. I've seen your notice. I can't submit the test results to you because I don't bring a test cell phone. Now I've taken some pictures. I don't know if it's still time. Maybe you need some videos yet?

I'm surprised at what you did，it's a very meaningful thing，i support you very much, if you need other help, please tell me.
I am also changing some content recently for this SceneForm project. My AR community application is very unstable. For example, the environment detection fails due to the rotation of the mobile phone, and the models ware lost. For example, some models suddenly disappear after moving a certain distance, such as a crash of the program, many, many. Maybe I need to study filament next, which is a very headache and difficult thing.

And total screenshot files and video had uploaded in: EnvironmentLightsTest

RGregat · 2021-09-13T15:28:33Z

I'm a little bit late to this game :)
Here are some annotated screenshots https://github.com/RGregat/sceneform_environment_test

ThomasGorisse · 2021-09-13T23:41:21Z

Better/Faster/Stronger

...and more dynamic with the server side models and environments (@romainguy including HDR and KTX).
...and funnier with the reflections skybox = Move inside a virtual environment with your phone or see what ARCore is actually seeing.

There is so much app ideas coming from this PR.

Test app on Google Play: https://play.google.com/store/apps/details?id=com.gorisse.thomas.ar.environmentlights

ThomasGorisse

@romainguy @devbridie I still have some questions to merge a cleaned version

ThomasGorisse · 2021-09-14T17:25:48Z