Platforms: does not work on OpenGL ES (Android, WebGL, Quest 2 etc.)

[smtabatabaie]

Hi, Thanks for the awesome project. I could run it via OpenXR and Quest Link (tethered to my desktop computer with a 3090), but when I try to run it on Quest standalone and build an apk, I get 50 errors regarding the shader files like this:

error in 'Gaussian Splatting/Debug/Render Boxes': 'asfloat': no matching 1 parameter intrinsic function; Possible intrinsic functions are: asfloat(float|half|int|uint) at Assets/GaussianSplatting/Shaders/GaussianSplatting.hlsl(469) (on gles3)

Has anyone succeeded in running Gaussian Splatting on Quest standalone?
Thanks

[aras-p]

Oh, OpenGL or OpenGL ES is definitely not going to be supported. If Quest supports Vulkan, try using that instead?

[cliv3dev]

nope, black screen on quest last time I tested with vulkan in URP...
I was able to run on PC in VR with Quest in QuestLink and OpenXR.
Will retry on Quest with Vulkan and OpenXR see if it is better...

[electrum-bowie]

i was aiming for:
rendering gaussian splats on Native Quest
optimizing and squeezing in as much fps as possible
having atleast 15 frames per second

sadly with Vulkan and OpenXR it's working at around 20 fps which is great but
the only thing that is rendering is the color of the skybox.. for now there's only one option which is the gaussian splatting viewer implementation in A-Frame, but that's working on WebXR at 2 fps.. and there's no room for optimization like unity has.
Thank you for your existing AMAZING work, but the best use-case for this tech is VR (standalone of course) even if it only reaches 20 fps on Quest 2 Devices (they're outdated now so it's no surprise).
for now, The Unity Gaussian Splatting plugin is useless for just that, unfortunately, hopefully you'll see this message @aras-p and realize that Native Quest support in this plugin should be priority number 1! Good Luck ! God Bless you <3

[aras-p]

sadly with Vulkan and OpenXR it's working at around 20 fps which is great but the only thing that is rendering is the color of the skybox..

Is that some known issue with Unity or OpenXR Vulkan support? Maybe it's fixed in a more recent Unity version by chance (e.g. latest 2023 alpha/beta)?

see this message @aras-p and realize that Native Quest support in this plugin should be priority number 1!

I do see this message, yes. I have exactly zero VR devices around, so adding any sort of VR support is quite a bit outside of my reach.

[electrum-bowie]

1. i'm gonna try right now, thanks!
2. if you agree that this feature should happen at one point, i can certainly help you test it because i'm free all the time, i go to a special school so i'm almost active all day!

[electrum-bowie]

Shader error in 'Gaussian Splatting/Debug/Render Boxes': no matching function for call to 'asfloat' at Assets/GSPLAT/GaussianSplatting/Shaders/GaussianSplatting.hlsl(527) (on vulkan)

[electrum-bowie]

@aras-p

[electrum-bowie]

i don't know if this is why it's not rendering, i updated to 2023.1.16f1

[smtabatabaie]

Shader error in 'Gaussian Splatting/Debug/Render Boxes': no matching function for call to 'asfloat' at Assets/GSPLAT/GaussianSplatting/Shaders/GaussianSplatting.hlsl(527) (on vulkan)

Exactly what I'm getting when I try to run on Quest 2

[aras-p]

@electrum-bowie I edited your post to make it shorter. And yes, that's exactly the error mentioned at the beginning of this issue. But until either someone contributes a fix, or I get a VR device (which is unlikely to happen), it will stay like that.

[francescofugazzi]

Shader error in 'Gaussian Splatting/Debug/Render Boxes': no matching function for call to 'asfloat' at Assets/GSPLAT/GaussianSplatting/Shaders/GaussianSplatting.hlsl(527) (on vulkan)

Exactly what I'm getting when I try to run on Quest 2

The same issue appears when targeting any Android device. What makes me curious is that by insisting it compiles and export apk but then, of course, just a black screen. When trying to compile version 0.2.0 it works, sort of. It display only some splats with wrong sizes and positions and not all of them and it's very heavy task for android devices, even with powerful ones. By selecting debug points instead of splats it becomes "usable" but again not all points are rendered, only a few of them. Maybe it was partially working because there was also another code for sorting the splats.
I also tried exporting for iOS but only 0.6.0 and as expected I had only black screen but on conosle debugger it was reporting problems related to FfxParallelSort.

[JiamingSuen]

With the latest code v0.7.0, instead of producing the abovementioned error, sorting kernel compatibility check won't pass (here in the code, only m_KernelSum is supported) when targeting to Android.

---

Edit:
The above error only occurs on macOS. On Windows it's the same "no matching function for call" error.
I tried to remove

UnityGaussianSplatting/package/Shaders/GaussianSplatting.hlsl

Lines 525 to 542 in fc97e07

	if (shFormat > VECTOR_FMT_32F && shFormat <= VECTOR_FMT_6)
	{
	s.sh.sh1 = lerp(shMin, shMax, asfloat(s.sh.sh1 ));
	s.sh.sh2 = lerp(shMin, shMax, asfloat(s.sh.sh2 ));
	s.sh.sh3 = lerp(shMin, shMax, asfloat(s.sh.sh3 ));
	s.sh.sh4 = lerp(shMin, shMax, asfloat(s.sh.sh4 ));
	s.sh.sh5 = lerp(shMin, shMax, asfloat(s.sh.sh5 ));
	s.sh.sh6 = lerp(shMin, shMax, asfloat(s.sh.sh6 ));
	s.sh.sh7 = lerp(shMin, shMax, asfloat(s.sh.sh7 ));
	s.sh.sh8 = lerp(shMin, shMax, asfloat(s.sh.sh8 ));
	s.sh.sh9 = lerp(shMin, shMax, asfloat(s.sh.sh9 ));
	s.sh.sh10 = lerp(shMin, shMax, asfloat(s.sh.sh10));
	s.sh.sh11 = lerp(shMin, shMax, asfloat(s.sh.sh11));
	s.sh.sh12 = lerp(shMin, shMax, asfloat(s.sh.sh12));
	s.sh.sh13 = lerp(shMin, shMax, asfloat(s.sh.sh13));
	s.sh.sh14 = lerp(shMin, shMax, asfloat(s.sh.sh14));
	s.sh.sh15 = lerp(shMin, shMax, asfloat(s.sh.sh15));
	}

(where the asfloat() error occurred) and use "High" quality assets. The compilation passed but it's still a black screen when running on an Android phone.

I tried to dig into the Android logcat and here are the outputs:

(click to expand)

2023/10/21 13:55:59.228 14372 14372 Info ussianSplatting Using CollectorTypeCC GC.
2023/10/21 13:55:59.232 14372 14372 Error ussianSplatting Not starting debugger since process cannot load the jdwp agent.
2023/10/21 13:55:59.246 14372 14372 Debug CompatibilityChangeReporter Compat change id reported: 171979766; UID 10271; state: ENABLED
2023/10/21 13:55:59.251 14372 14372 Info ussianSplatting Failed to add image file: Failed to find OatDexFile for DexFile /data/app/~~ABvvwi_g9gV8-p8ONtiKsw==/com.DefaultCompany.UnityGaussianSplatting-q0yqyv6ymmbnmyRISEx4CQ==/base.apk (canonical path /data/app/~~ABvvwi_g9gV8-p8ONtiKsw==/com.DefaultCompany.UnityGaussianSplatting-q0yqyv6ymmbnmyRISEx4CQ==/base.apk) in OatFile /data/app/~~DeR5zeOZPZKFkj3wDpad5g==/com.DefaultCompany.UnityGaussianSplatting-lCiPGx0FemteYOJ_n4J-8Q==/oat/arm/base.odex
2023/10/21 13:55:59.252 14372 14372 Debug nativeloader Configuring clns-4 for other apk /data/app/~~DeR5zeOZPZKFkj3wDpad5g==/com.DefaultCompany.UnityGaussianSplatting-lCiPGx0FemteYOJ_n4J-8Q==/base.apk. target_sdk_version=32, uses_libraries=, library_path=/data/app/~~DeR5zeOZPZKFkj3wDpad5g==/com.DefaultCompany.UnityGaussianSplatting-lCiPGx0FemteYOJ_n4J-8Q==/lib/arm:/data/app/~~DeR5zeOZPZKFkj3wDpad5g==/com.DefaultCompany.UnityGaussianSplatting-lCiPGx0FemteYOJ_n4J-8Q==/base.apk!/lib/armeabi-v7a, permitted_path=/data:/mnt/expand:/data/user/0/com.DefaultCompany.UnityGaussianSplatting
2023/10/21 13:55:59.262 14372 14372 Verbose GraphicsEnvironment ANGLE Developer option for 'com.DefaultCompany.UnityGaussianSplatting' set to: 'default'
2023/10/21 13:55:59.262 14372 14372 Verbose GraphicsEnvironment ANGLE GameManagerService for com.DefaultCompany.UnityGaussianSplatting: false
2023/10/21 13:55:59.262 14372 14372 Verbose GraphicsEnvironment Updatable production driver is not supported on the device.
2023/10/21 13:55:59.266 14372 14372 Debug NetworkSecurityConfig No Network Security Config specified, using platform default
2023/10/21 13:55:59.266 14372 14372 Debug NetworkSecurityConfig No Network Security Config specified, using platform default
2023/10/21 13:55:59.304 14372 14372 Info Mono JNI_OnLoad called
2023/10/21 13:55:59.315 14372 14372 Debug CompatibilityChangeReporter Compat change id reported: 237531167; UID 10271; state: DISABLED
2023/10/21 13:55:59.345 14372 14372 Error SurfaceSyncer Failed to find sync for id=0
2023/10/21 13:55:59.351 14372 14372 Warn AdrenoUtils <ReadGpuID_from_sysfs:197>: Failed to open /sys/class/kgsl/kgsl-3d0/gpu_model
2023/10/21 13:55:59.351 14372 14372 Warn AdrenoUtils <ReadGpuID:221>: Failed to read chip ID from gpu_model. Fallback to use the GSL path
2023/10/21 13:55:59.377 14372 14392 Info Unity MemoryManager: Using 'Dynamic Heap' Allocator.
2023/10/21 13:55:59.379 14372 14372 Error SurfaceSyncer Failed to find sync for id=0
2023/10/21 13:55:59.404 14372 14392 Info Unity SystemInfo CPU = ARM64 FP ASIMD AES, Cores = 8, Memory = 7461mb
2023/10/21 13:55:59.404 14372 14392 Info Unity SystemInfo ARM big.LITTLE configuration: 2 big (mask: 0xc0), 6 little (mask: 0x3f)
2023/10/21 13:55:59.404 14372 14392 Info Unity ApplicationInfo com.DefaultCompany.UnityGaussianSplatting version 0.1
2023/10/21 13:55:59.404 14372 14392 Info Unity Built from '2022.3/staging' branch, Version '2022.3.9f1 (ea401c316338)', Build type 'Release', Scripting Backend 'mono', CPU 'armeabi-v7a', Stripping 'Disabled'
2023/10/21 13:55:59.530 14372 14392 Info Unity Company Name: DefaultCompany
2023/10/21 13:55:59.530 14372 14392 Info Unity Product Name: UnityGaussianSplatting
2023/10/21 13:55:59.536 14372 14392 Debug vulkan searching for layers in '/data/app/~~DeR5zeOZPZKFkj3wDpad5g==/com.DefaultCompany.UnityGaussianSplatting-lCiPGx0FemteYOJ_n4J-8Q==/lib/arm'
2023/10/21 13:55:59.536 14372 14392 Debug vulkan searching for layers in '/data/app/~~DeR5zeOZPZKFkj3wDpad5g==/com.DefaultCompany.UnityGaussianSplatting-lCiPGx0FemteYOJ_n4J-8Q==/base.apk!/lib/armeabi-v7a'
2023/10/21 13:55:59.539 14372 14392 Info AdrenoVK-0 ===== BEGIN DUMP OF OVERRIDDEN SETTINGS =====
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 ===== END DUMP OF OVERRIDDEN SETTINGS =====
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 QUALCOMM build          : 4783c89, I46ff5fc46f
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 Build Date              : 11/30/20
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 Shader Compiler Version : EV031.31.04.01
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 Local Branch            : QPR2
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 Remote Branch           : 
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 Remote Branch           : 
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 Reconstruct Branch      : 
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 Build Config            : S P 10.0.4 AArch32
2023/10/21 13:55:59.540 14372 14392 Info AdrenoVK-0 Driver Path             : /vendor/lib/hw/vulkan.adreno.so
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 QUALCOMM build          : 4783c89, I46ff5fc46f
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 Build Date              : 11/30/20
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 Shader Compiler Version : EV031.31.04.01
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 Local Branch            : QPR2
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 Remote Branch           : 
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 Remote Branch           : 
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 Reconstruct Branch      : 
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 Build Config            : S P 10.0.4 AArch32
2023/10/21 13:55:59.544 14372 14392 Info AdrenoVK-0 Driver Path             : /vendor/lib/hw/vulkan.adreno.so
2023/10/21 13:55:59.559 14372 14392 Error qdgralloc GetGpuPixelFormat: No map for format: 0x38
2023/10/21 13:55:59.559 14372 14392 Error AdrenoUtils <validate_memory_layout_input_parmas:1923>: Unknown Format 0
2023/10/21 13:55:59.559 14372 14392 Error AdrenoUtils <adreno_init_memory_layout:4634>: Memory Layout input parameter validation failed!
2023/10/21 13:55:59.559 14372 14392 Error qdgralloc GetGpuResourceSizeAndDimensions Graphics metadata init failed
2023/10/21 13:55:59.559 14372 14392 Error Gralloc4 isSupported(1, 1, 56, 1, ...) failed with 1
2023/10/21 13:55:59.559 14372 14392 Error GraphicBufferAllocator Failed to allocate (4 x 4) layerCount 1 format 56 usage b00: 1
2023/10/21 13:55:59.559 14372 14392 Error AHardwareBuffer GraphicBuffer(w=4, h=4, lc=1) failed (Unknown error -1), handle=0x0
2023/10/21 13:55:59.559 14372 14392 Error qdgralloc GetGpuPixelFormat: No map for format: 0x38
2023/10/21 13:55:59.559 14372 14392 Error AdrenoUtils <validate_memory_layout_input_parmas:1923>: Unknown Format 0
2023/10/21 13:55:59.559 14372 14392 Error AdrenoUtils <adreno_init_memory_layout:4634>: Memory Layout input parameter validation failed!
2023/10/21 13:55:59.559 14372 14392 Error qdgralloc GetGpuResourceSizeAndDimensions Graphics metadata init failed
2023/10/21 13:55:59.559 14372 14392 Error Gralloc4 isSupported(1, 1, 56, 1, ...) failed with 1
2023/10/21 13:55:59.560 14372 14392 Error GraphicBufferAllocator Failed to allocate (4 x 4) layerCount 1 format 56 usage b00: 1
2023/10/21 13:55:59.560 14372 14392 Error AHardwareBuffer GraphicBuffer(w=4, h=4, lc=1) failed (Unknown error -1), handle=0x0
2023/10/21 13:55:59.566 14372 14424 Debug Choreographer Attaching thread to JVM for AChoreographer
2023/10/21 13:55:59.570 14372 14392 Verbose SwappyVk SwappyVk initialized for VkDevice 0xb5fa0470 using VK_GOOGLE_display_timing on Android
2023/10/21 13:55:59.570 14372 14392 Info SwappyVk Returning refresh duration of 11111111 nsec (approx 90.000001 Hz)
2023/10/21 13:55:59.575 14372 14392 Verbose MediaRouter Selecting route: RouteInfo{ name=Phone, description=null, status=null, category=RouteCategory{ name=System types=ROUTE_TYPE_LIVE_AUDIO ROUTE_TYPE_LIVE_VIDEO  groupable=false }, supportedTypes=ROUTE_TYPE_LIVE_AUDIO ROUTE_TYPE_LIVE_VIDEO , presentationDisplay=null }
2023/10/21 13:55:59.575 14372 14392 Info MediaRouter Skip setBluetoothA2dpOn(): types=8388615, isPlaybackActive()=false, BT route=null
2023/10/21 13:55:59.589 14372 14392 Debug ussianSplatting PlayerBase::PlayerBase()
2023/10/21 13:55:59.590 14372 14392 Debug ussianSplatting TrackPlayerBase::TrackPlayerBase()
2023/10/21 13:55:59.590 14372 14392 Info libOpenSLES Emulating old channel mask behavior (ignoring positional mask 0x3, using default mask 0x3 based on channel count of 2)
2023/10/21 13:55:59.590 14372 14392 Warn AudioTrack set(): notificationFrames=-21 clamped to the range -1 to -8
2023/10/21 13:55:59.597 14372 14392 Info AudioTrack createTrack_l(0): AUDIO_OUTPUT_FLAG_FAST successful; frameCount 0 -> 768
2023/10/21 13:55:59.599 14372 14372 Debug MediaRouter onRestoreRoute() : route=RouteInfo{ name=Phone, description=null, status=null, category=RouteCategory{ name=System types=ROUTE_TYPE_LIVE_AUDIO ROUTE_TYPE_LIVE_VIDEO  groupable=false }, supportedTypes=ROUTE_TYPE_LIVE_AUDIO ROUTE_TYPE_LIVE_VIDEO , presentationDisplay=null }
2023/10/21 13:55:59.600 14372 14372 Verbose MediaRouter Selecting route: RouteInfo{ name=Phone, description=null, status=null, category=RouteCategory{ name=System types=ROUTE_TYPE_LIVE_AUDIO ROUTE_TYPE_LIVE_VIDEO  groupable=false }, supportedTypes=ROUTE_TYPE_LIVE_AUDIO ROUTE_TYPE_LIVE_VIDEO , presentationDisplay=null }
2023/10/21 13:55:59.831 14372 14392 Info MONO Getting locale
2023/10/21 13:55:59.831 14372 14392 Info Mono Locale en-US
2023/10/21 13:56:00.484 14372 14392 Error qdgralloc GetGpuPixelFormat: No map for format: 0x38
2023/10/21 13:56:00.484 14372 14392 Error AdrenoUtils <validate_memory_layout_input_parmas:1923>: Unknown Format 0
2023/10/21 13:56:00.484 14372 14392 Error AdrenoUtils <adreno_init_memory_layout:4634>: Memory Layout input parameter validation failed!
2023/10/21 13:56:00.484 14372 14392 Error qdgralloc GetGpuResourceSizeAndDimensions Graphics metadata init failed
2023/10/21 13:56:00.484 14372 14392 Error Gralloc4 isSupported(1, 1, 56, 1, ...) failed with 1
2023/10/21 13:56:00.485 14372 14392 Error GraphicBufferAllocator Failed to allocate (4 x 4) layerCount 1 format 56 usage b00: 1
2023/10/21 13:56:00.485 14372 14392 Error AHardwareBuffer GraphicBuffer(w=4, h=4, lc=1) failed (Unknown error -1), handle=0x0
2023/10/21 13:56:00.485 14372 14392 Error qdgralloc GetGpuPixelFormat: No map for format: 0x38
2023/10/21 13:56:00.485 14372 14392 Error AdrenoUtils <validate_memory_layout_input_parmas:1923>: Unknown Format 0
2023/10/21 13:56:00.485 14372 14392 Error AdrenoUtils <adreno_init_memory_layout:4634>: Memory Layout input parameter validation failed!
2023/10/21 13:56:00.485 14372 14392 Error qdgralloc GetGpuResourceSizeAndDimensions Graphics metadata init failed
2023/10/21 13:56:00.485 14372 14392 Error Gralloc4 isSupported(1, 1, 56, 1, ...) failed with 1
2023/10/21 13:56:00.485 14372 14392 Error GraphicBufferAllocator Failed to allocate (4 x 4) layerCount 1 format 56 usage b00: 1
2023/10/21 13:56:00.485 14372 14392 Error AHardwareBuffer GraphicBuffer(w=4, h=4, lc=1) failed (Unknown error -1), handle=0x0
2023/10/21 13:56:00.491 14372 14392 Verbose SwappyVk SwappyVk initialized for VkDevice 0xb5fa0470 using VK_GOOGLE_display_timing on Android
2023/10/21 13:56:00.492 14372 14392 Info SwappyVk Returning refresh duration of 11111111 nsec (approx 90.000001 Hz)
2023/10/21 13:56:03.810 14372 14422 Info AdrenoVK-0 Shader compilation failed for shaderType: 5
2023/10/21 13:56:03.810 14372 14422 Info AdrenoVK-0 Info log: Assertion failed: 0 && "not yet supported"
2023/10/21 13:56:04.051 14372 14441 Error SwappyVk Failed to wait for fence 2
2023/10/21 13:56:04.251 14372 14441 Error SwappyVk Failed to wait for fence 2
2023/10/21 13:56:04.739 14372 14441 Error SwappyVk Failed to wait for fence 2

It seems to be related to the unsupported texture format on mobile (related to #44? but I'm using the "High" quality). I'm using the Vulkan backend.