H264 encoding crashes in vaEndPicture, since 21.x.x

[artlav]

H264 encoding crashes in vaEndPicture, since 21.x.x or thereabout.

Tested on:

i5-6260U (Iris Graphics 540)
m5-6Y54 (HD Graphics 515)

Version 20.4.5 works fine, version 21.1.3 crashes, all else being the same.

Reproduction:

Run h264encode from libva-utils.

On 20.4.5 it would work fine.

On 21.1.3 it would segfault after printing "Loading data into surface 15.....Complete surface loading", while calling vaEndPicture.

Full log of the crash on m5-6Y54 (current git built with debug): log.txt

I was not able to reproduce it with ffmpeg's h264_vaapi codec, and curiously enough about one time in 10 running h264encode after ffmpeg makes h264encode work once.

[XinfengZhang]

I debugged it, it is not a regression.

1. application should set the reference in slice parameter to be INVALID
2. driver should add some check statement to avoid crash.

backtrace:
#0 0x00007ffff2148e96 in CodecHalSetRcsSurfaceState (hwInterface=0x55555586f190,
cmdBuffer=0x7fffffffcfa0, surfaceCodecParams=0x7fffffffcc60, kernelState=0x555555882f48)
at /home/intel/carl/open_driver/media-driver/media_driver/agnostic/common/codec/hal/codechal_ut ilities.cpp:463
#1 0x00007ffff24dcdb5 in CodechalEncodeAvcEncG9::SendAvcMbEncSurfaces (this=0x5555558714f0,
cmdBuffer=0x7fffffffcfa0, params=0x7fffffffd150)
at /home/intel/carl/open_driver/media-driver/media_driver/agnostic/gen9/codec/hal/codechal_enco de_avc_g9.cpp:1955
#2 0x00007ffff227928c in CodechalEncodeAvcEnc::MbEncKernel (this=0x5555558714f0,
mbEncIFrameDistInUse=true)
at /home/intel/carl/open_driver/media-driver/media_driver/agnostic/common/codec/hal/codechal_en code_avc.cpp:3896
#3 0x00007ffff2286887 in CodechalEncodeAvcEnc::ExecuteKernelFunctions (this=0x5555558714f0)
at /home/intel/carl/open_driver/media-driver/media_driver/agnostic/common/codec/hal/codechal_en code_avc.cpp:6372
#4 0x00007ffff223a917 in CodechalEncoderState::ExecuteEnc (this=0x5555558714f0,
encodeParams=0x55555584c048)
at /home/intel/carl/open_driver/media-driver/media_driver/agnostic/common/codec/hal/codechal_en coder_base.cpp:4767
#5 0x00007ffff222765e in CodechalEncoderState::Execute (this=0x5555558714f0,
params=0x55555584c048)
at /home/intel/carl/open_driver/media-driver/media_driver/agnostic/common/codec/hal/codechal_en coder_base.cpp:610
#6 0x00007ffff2ae81f0 in DdiEncodeAvc::EncodeInCodecHal (this=0x55555583f300, numSlices=1)
at /home/intel/carl/open_driver/media-driver/media_driver/linux/common/codec/ddi/media_ddi_enco de_avc.cpp:1146
#7 0x00007ffff2aaeb95 in DdiEncodeBase::EndPicture (this=0x55555583f300, ctx=0x5555557e1e30,
context=536870912)
at /home/intel/carl/open_driver/media-driver/media_driver/linux/common/codec/ddi/media_ddi_enco de_base.cpp:77
#8 0x00007ffff2abc187 in DdiEncode_EndPicture (ctx=0x5555557e1e30, context=536870912)
---Type to continue, or q to quit---
at /home/intel/carl/open_driver/media-driver/media_driver/linux/common/codec/ddi/media_libva_en coder.cpp:629
#9 0x00007ffff2b3702a in DdiMedia_EndPicture (ctx=0x5555557e1e30, context=536870912)
at /home/intel/carl/open_driver/media-driver/media_driver/linux/common/ddi/media_libva.cpp:3827
#10 0x00007ffff726a210 in vaEndPicture (dpy=0x5555557e1cc0, context=536870912) at va.c:1629
#11 0x0000555555556f94 in encode_frames () at h264encode.c:2273
#12 main (argc=, argv=) at h264encode.c:2460

the problem is
https://github.com/intel/media-driver/blob/master/media_driver/agnostic/gen9/codec/hal/codechal_encode_avc_g9.cpp#L1912
the refPic.FrameIdx = 128 it actually is invalid, and params->pAvcPicIdx length is 16.

[artlav]

Could you clarify what should be changed on application side?

I tried initializing RefPicList0 and RefPicList1 in render_slice

slice_param.RefPicList0[i].picture_id = VA_INVALID_SURFACE;
slice_param.RefPicList0[i].flags = VA_PICTURE_H264_INVALID;

And that did solve the crash on the first call of vaEndPicture, but then the next frame still crashes it.

After that i tried to also initialize ReferenceFrames[16], RefPicList0_P[32], RefPicList0_B[32], RefPicList1_B[32] the same way at the start, and that worked fine until intra_idr_period of frames was processed (i.e. run h264encode -n 120), then crashed again the same way.

Not sure what else is there to initialize.

[XinfengZhang]

@artlav sorry for late response, suppose we could provide a PR in libva-utils or media_driver later.