Video Streaming Fails on Android 12 at Low Resolutions #1803
Description
Bug Summary
Video streaming videos with resolutions lower than 320x240 on Android 12 fails and prevents further streams at valid resolutions until the app re-registers with SDL Core.
I believe this is caused by a quality floor enforcement in MediaCodec.
Beginning with Build.VERSION_CODES.S, Android's Video MediaCodecs enforce a minimum quality floor. The intent is to eliminate poor quality video encodings. This quality floor is applied when the codec is in Variable Bitrate (VBR) mode; it is not applied when the codec is in Constant Bitrate (CBR) mode. The quality floor enforcement is also restricted to a particular size range; this size range is currently for video resolutions larger than 320x240 up through 1920x1080.
This explains why streaming failed inconsistently between devices and only at certain resolutions.
This issue was discovered during streaming tests in sdl_test_suite_java.
Reproduction Steps
- Connect an Android SDL Navigation app to SDL Core with either sdl_hmi or generic_hmi
- Start streaming a video to the HMI
- Switch the video stream to a resolution lower than 320x240 (ex: 400x100)
Expected Behavior: The video streams successfully and the app continues to behave normally afterwards
Observed Behavior: The video fails to stream and the app is unable to stream again until it re-registers with Core
Which projects have you seen this bug on?
Android
Android Version(s)
12
Android Device(s)
Samsung Galaxy S21 5G
sdl_java_suite Version
5.3.1, 5.4.0_RC
Testing Environment(s)
SDL Core v8.1.0 + SDL HMI 5.7.0Relevant log output
2022-04-05 11:06:00.454 15391-15391/io.livio.sdltestsuite.app1 D/VQApply: minquality: 400x100 outside of shaping range
2022-04-05 11:06:00.454 15391-15391/io.livio.sdltestsuite.app1 D/MediaCodec: shapeMediaFormat: deltas(1): AMessage(what = 0x00000000) = {
int32_t android._encoding-quality-level = 0
}
2022-04-05 11:06:00.454 15391-15780/io.livio.sdltestsuite.app1 D/CCodec: [c2.qti.avc.encoder] buffers are bound to CCodec for this session
2022-04-05 11:06:00.455 15391-15780/io.livio.sdltestsuite.app1 I/CCodec: appPid(15391) width(400) height(100)
2022-04-05 11:06:00.455 15391-15780/io.livio.sdltestsuite.app1 I/CCodec: app-name: io.livio.sdltestsuite.app1
2022-04-05 11:06:00.455 15391-15780/io.livio.sdltestsuite.app1 D/CCodecConfig: no c2 equivalents for color-format
2022-04-05 11:06:00.455 15391-15780/io.livio.sdltestsuite.app1 D/CCodecConfig: no c2 equivalents for flags
2022-04-05 11:06:00.455 15391-15780/io.livio.sdltestsuite.app1 D/CCodecConfig: no c2 equivalents for encoder
2022-04-05 11:06:00.455 15391-15780/io.livio.sdltestsuite.app1 D/CCodecConfig: Bad parameter value
2022-04-05 11:06:00.455 15391-15780/io.livio.sdltestsuite.app1 D/CCodecConfig: c2 config diff is c2::u32 coded.bitrate.value = 512000
c2::float coded.frame-rate.value = 20
c2::i64 coding.sync-frame-interval.value = 5000000
c2::u32 raw.pixel-format.value = 34
c2::u32 raw.size.width = 400
2022-04-05 11:06:00.455 15391-15780/io.livio.sdltestsuite.app1 W/ColorUtils: expected specified color aspects (0:0:255:255)
2022-04-05 11:06:00.456 15391-15780/io.livio.sdltestsuite.app1 W/CCodec: failed to configure c2 params
2022-04-05 11:06:00.456 15391-15779/io.livio.sdltestsuite.app1 E/MediaCodec: Codec reported err 0xffffffea, actionCode 0, while in state 3/CONFIGURING
2022-04-05 11:06:00.456 15391-15391/io.livio.sdltestsuite.app1 E/MediaCodec: configure failed with err 0xffffffea, resetting...
2022-04-05 11:06:00.456 15391-15779/io.livio.sdltestsuite.app1 I/CCodec: state->set(RELEASING)
2022-04-05 11:06:00.456 15391-15782/io.livio.sdltestsuite.app1 I/CCodec: [c2.qti.avc.encoder] release(1)
2022-04-05 11:06:00.456 15391-15782/io.livio.sdltestsuite.app1 I/CCodec: state->set(RELEASED)
2022-04-05 11:06:00.456 15391-15779/io.livio.sdltestsuite.app1 I/MediaCodec: Codec shutdown complete
2022-04-05 11:06:00.457 15391-15782/io.livio.sdltestsuite.app1 I/hw-BpHwBinder: onLastStrongRef automatically unlinking death recipients
2022-04-05 11:06:00.458 15391-15779/io.livio.sdltestsuite.app1 I/CCodec: state->set(ALLOCATING)
2022-04-05 11:06:00.459 15391-15780/io.livio.sdltestsuite.app1 I/CCodec: allocate(c2.qti.avc.encoder)
2022-04-05 11:06:00.460 15391-15780/io.livio.sdltestsuite.app1 I/CCodec: setting up 'default' as default (vendor) store
2022-04-05 11:06:00.465 15391-15780/io.livio.sdltestsuite.app1 I/CCodec: Created component [c2.qti.avc.encoder]
2022-04-05 11:06:00.465 15391-15780/io.livio.sdltestsuite.app1 I/CCodec: state->set(ALLOCATED)
2022-04-05 11:06:00.465 15391-15780/io.livio.sdltestsuite.app1 D/CCodecConfig: read media type: video/avc
2022-04-05 11:06:00.466 15391-15780/io.livio.sdltestsuite.app1 D/ReflectedParamUpdater: extent() != 1 for single value type: vendor.qti-ext-enc-info-metadata-cvp.reserved
2022-04-05 11:06:00.468 15391-15780/io.livio.sdltestsuite.app1 D/ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.pool-ids.values
2022-04-05 11:06:00.469 15391-15780/io.livio.sdltestsuite.app1 D/ReflectedParamUpdater: ignored struct field coding.gop.values
2022-04-05 11:06:00.470 15391-15780/io.livio.sdltestsuite.app1 D/CCodecConfig: ignoring local param raw.color (0xc2001809) as it is already supported
2022-04-05 11:06:00.471 15391-15780/io.livio.sdltestsuite.app1 I/CCodecConfig: query failed after returning 22 values (BAD_INDEX)
2022-04-05 11:06:00.471 15391-15780/io.livio.sdltestsuite.app1 D/CCodecConfig: c2 config diff is Dict {
c2::u32 algo.bitrate-mode.value = 3
c2::i32 algo.priority.value = 0
c2::float algo.rate.value = 30
c2::u32 algo.secure-mode.value = 0
c2::u32 coded.bitrate.value = 20000000
c2::float coded.frame-rate.value = 30
c2::u32 coded.pl.level = 0
c2::u32 coded.pl.profile = 20484
c2::u32 coded.vui.color.matrix = 255
c2::u32 coded.vui.color.primaries = 0
c2::u32 coded.vui.color.range = 0
c2::u32 coded.vui.color.transfer = 255
c2::u32 coding.intra-refresh.mode = 0
c2::float coding.intra-refresh.period = 0
c2::u32 coding.request-sync-frame.value = 0
c2::i64 coding.sync-frame-interval.value = 1000000
Buffer coding.temporal-layering = {
00000000: 10 00 00 00 07 20 01 52 00 00 00 00 00 00 00 00 ..... .R........
}
c2::u32 default.color.matrix = 255
c2::u32 default.color.primaries = 0
c2::u32 default.color.range = 0
c2::u32 default.color.transfer = 255
c2::u32 input.delay.value = 0
string input.media-type.value = "video/raw"
c2::u32 output.buffe
2022-04-05 11:06:00.471 15391-15780/io.livio.sdltestsuite.app1 W/ColorUtils: expected specified color aspects (0:0:255:255)
2022-04-05 11:06:00.472 15391-15391/io.livio.sdltestsuite.app1 W/System.err: java.lang.IllegalArgumentException
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.media.MediaCodec.native_configure(Native Method)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.media.MediaCodec.configure(MediaCodec.java:2176)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.media.MediaCodec.configure(MediaCodec.java:2092)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.encoder.VirtualDisplayEncoder.prepareVideoEncoder(VirtualDisplayEncoder.java:507)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.encoder.VirtualDisplayEncoder.start(VirtualDisplayEncoder.java:202)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.managers.video.VideoStreamManager.startEncoder(VideoStreamManager.java:484)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.managers.video.VideoStreamManager.access$800(VideoStreamManager.java:97)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.managers.video.VideoStreamManager$1.onServiceStarted(VideoStreamManager.java:147)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.session.SdlSession.onServiceStarted(SdlSession.java:172)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase.handleStartServiceACK(SdlProtocolBase.java:1122)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase$MessageFrameAssembler.handleControlFrame(SdlProtocolBase.java:1467)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase$MessageFrameAssembler.handleFrame(SdlProtocolBase.java:1423)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase.handlePacketReceived(SdlProtocolBase.java:715)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase$2.onPacketReceived(SdlProtocolBase.java:1187)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.transport.TCPTransportManager$TCPHandler.handleMessage(TCPTransportManager.java:130)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.os.Handler.dispatchMessage(Handler.java:106)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.os.Looper.loopOnce(Looper.java:226)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.os.Looper.loop(Looper.java:313)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.app.ActivityThread.main(ActivityThread.java:8663)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at java.lang.reflect.Method.invoke(Native Method)
2022-04-05 11:06:00.473 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:567)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 E/VirtualDisplayEncoder: RC_5.4.0: Unable to create Virtual Display.
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: java.lang.RuntimeException: java.lang.RuntimeException: invalid surface: null
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.encoder.VirtualDisplayEncoder.start(VirtualDisplayEncoder.java:238)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.managers.video.VideoStreamManager.startEncoder(VideoStreamManager.java:484)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.managers.video.VideoStreamManager.access$800(VideoStreamManager.java:97)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.managers.video.VideoStreamManager$1.onServiceStarted(VideoStreamManager.java:147)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.session.SdlSession.onServiceStarted(SdlSession.java:172)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase.handleStartServiceACK(SdlProtocolBase.java:1122)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase$MessageFrameAssembler.handleControlFrame(SdlProtocolBase.java:1467)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase$MessageFrameAssembler.handleFrame(SdlProtocolBase.java:1423)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase.handlePacketReceived(SdlProtocolBase.java:715)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.protocol.SdlProtocolBase$2.onPacketReceived(SdlProtocolBase.java:1187)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.transport.TCPTransportManager$TCPHandler.handleMessage(TCPTransportManager.java:130)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.os.Handler.dispatchMessage(Handler.java:106)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.os.Looper.loopOnce(Looper.java:226)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.os.Looper.loop(Looper.java:313)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at android.app.ActivityThread.main(ActivityThread.java:8663)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at java.lang.reflect.Method.invoke(Native Method)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:567)
2022-04-05 11:06:00.474 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)
2022-04-05 11:06:00.475 15391-15391/io.livio.sdltestsuite.app1 W/System.err: Caused by: java.lang.RuntimeException: invalid surface: null
2022-04-05 11:06:00.475 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.android.grafika.gles.EglCore.createWindowSurface(EglCore.java:232)
2022-04-05 11:06:00.475 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.android.grafika.gles.EglSurfaceBase.createWindowSurface(EglSurfaceBase.java:60)
2022-04-05 11:06:00.475 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.android.grafika.gles.WindowSurface.<init>(WindowSurface.java:41)
2022-04-05 11:06:00.475 15391-15391/io.livio.sdltestsuite.app1 W/System.err: at com.smartdevicelink.encoder.VirtualDisplayEncoder.start(VirtualDisplayEncoder.java:202)
2022-04-05 11:06:00.475 15391-15391/io.livio.sdltestsuite.app1 W/System.err: ... 17 more
2022-04-05 11:06:00.477 15391-15391/io.livio.sdltestsuite.app1 I/MyPresentation: RC_5.4.0: mew view size, width: 400 height: 100