The h264encode doesn't free memory when it should

[seanvk]

migrated from Bugzilla #75287
status NEEDINFO severity major in component intel for ---
Reported in version unspecified on platform x86-64 (AMD64)
Assigned to: haihao

Original attachment names and IDs:

• 0001-Destroy-the-buffer-after-vaRenderPicture-for-encodin.patch (ID 96265)

On 2014-02-20 22:59:05 +0000, Bryan Christ wrote:

In h264encode, we observe that memory usage increases as the program proceeds. For large files, this results in unreasonably large memory usage.

Example case:

Input: 18G yuv file
Output: 3.3G h264 video
run with valgrind using the massif tool:
$ valgrind --tool=massif --stacks=yes /usr/local/bin/h264encode --srcyuv output2.yuv -framecount 0 -f 25 -o massif_test_out.mp4

There is a malloc at i965_drv_video.c:1671 that accounts for 92.58% of memory usage at the peak. This call creates a buffer which has a reference count on it. When the reference count hits 0, the memory is cleared. Many of the buffers stored in this way never have their reference counts set to zero during encoding. As a result, the memory usage keeps growing during encoding. On completion, all of this memory is freed, so a simple valgrind run does not find this leak.

Excerpts from the ms_print massif file:

Command: /usr/local/bin/h264encode --srcyuv output2.yuv -framecount 0 -f 25 -o massif_test_out
Massif arguments: --stacks=yes
ms_print arguments: massif.out.31611

GB

1.202^ #
| @@@@#
| @@@@@@@#
| @@@@@@@@@@#
| @@@@@@@@@@@@@#
| @@@@@@@@@@@@@@@@#
| :@@@ @@@@@@@@@@@@@@#
| @:@@:@ @ @@@@@@@@@@@@@@#
| @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @@@@@@@@@:@@:@ @ @@@@@@@@@@@@@@#
| :::@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @@@@:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @@:@@@@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| :@@@@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @@::@ @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @@:::@ ::@ @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @@@:@@:: @ ::@ @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
| @@:@@ :@@:: @ ::@ @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
0 +----------------------------------------------------------------------->Gi
0 75.59

---

n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B)

79 80,394,261,652 1,290,739,176 1,255,405,747 35,332,005 1,424
97.26% (1,255,405,747B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->92.58% (1,194,924,096B) 0x567936F: i965_create_buffer_internal.isra.18 (i965_drv_video.c:1671)
| ->92.58% (1,194,924,096B) 0x4C27E60: vaCreateBuffer (va.c:949)
| ->78.39% (1,011,857,700B) 0x4026E5: main (h264encode.c:1656)
| |
| ->13.44% (173,461,320B) 0x40664B: render_picture (h264encode.c:1426)
| | ->13.21% (170,570,160B) 0x402634: main (h264encode.c:1995)
| | |
| | ->00.22% (2,891,160B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.74% (9,605,076B) in 1+ places, all below ms_print's threshold (01.00%)
|
->02.51% (32,397,504B) 0x569361B: object_heap_expand (object_heap.c:62)
| ->02.51% (32,382,464B) 0x56937FE: object_heap_allocate (object_heap.c:121)
| | ->02.51% (32,379,904B) 0x5679235: i965_create_buffer_internal.isra.18 (i965_drv_video.c:1612)
| | | ->02.38% (30,683,648B) 0x567C0CB: i965_DeriveImage (i965_drv_video.c:3041)
| | | | ->02.38% (30,683,648B) 0x405464: load_surface (loadsurface.h:304)
| | | | ->02.38% (30,683,136B) 0x405F1E: storage_task (h264encode.c:1903)
| | | | | ->02.38% (30,683,136B) 0x4061A7: storage_task_thread (h264encode.c:1925)
| | | | | ->02.38% (30,683,136B) 0x3A95607C51: start_thread (in /usr/lib64/libpthread-2.17.so)
| | | | | ->02.38% (30,683,136B) 0x3A94EF5E1B: clone (in /usr/lib64/libc-2.17.so)
| | | | |
| | | | ->00.00% (512B) in 1+ places, all below ms_print's threshold (01.00%)
| | | |
| | | ->00.13% (1,696,256B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->00.00% (2,560B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.00% (15,040B) in 1+ places, all below ms_print's threshold (01.00%)
|
->01.88% (24,285,024B) 0x5679288: i965_create_buffer_internal.isra.18 (i965_drv_video.c:1629)
| ->01.88% (24,285,024B) 0x4C27E60: vaCreateBuffer (va.c:949)
| | ->01.88% (24,285,024B) in 9 places, all below massif's threshold (01.00%)
| |
| ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|
->00.29% (3,799,123B) in 1+ places, all below ms_print's threshold (01.00%)

On 2014-03-24 01:43:46 +0000, haihao wrote:

Created attachment 96265
destroy the buffer automatically

The patch is only for testing.

On 2014-03-24 17:39:45 +0000, Bryan Christ wrote:

Initial testing indicates that patch seems to fix the problem. Will continue to test.

On 2015-11-26 03:08:58 +0000, haihao wrote:

*** Bug 90429 has been marked as a duplicate of this bug. ***

[xhaihao]

Close this bug and file a new bug in libva-utils