Core dumped when attempted to uncompress a file using a dictionary

[scottchiefbaker]

In the supplied example.tar.gz I can uncompress the text file with the CLI no problem:

$ zstd -d -D dict test.txt.zst
test.txt.zst        : 10829 bytes

$ md5sum test.txt
58834ab4da8f93dc8f9864e9d9fdfb86  test.txt

When I attempt to decompress it with PHP I get a core dump

$zst  = file_get_contents("/var/tmp/test.txt.zst");
$dict = file_get_contents("/var/tmp/dict");
$txt  = zstd_uncompress_dict($zst, $dict);

Log output:

[11-Apr-2025 20:04:10] WARNING: [pool www] child 870081 exited on signal 6 (SIGABRT - core dumped) after 518.524306 seconds from start
[11-Apr-2025 20:04:10] NOTICE: [pool www] child 870526 started

I'm running v0.14.0 of the library.

[scottchiefbaker]

Found a little bit more info in a different log file:

Process 869640 (php-fpm) of user 48 dumped core.#012#012

Stack trace of thread 869640:#012#0  
0x00007f8d3d68b52c __pthread_kill_implementation (libc.so.6 + 0x8b52c)#012#1  
0x00007f8d3d63e686 raise (libc.so.6 + 0x3e686)#012#2  
0x00007f8d3d628833 abort (libc.so.6 + 0x28833)#012#3  
0x000055e7a302230e zend_mm_panic.constprop.0 (php-fpm + 0x22230e)#012#4  
0x000055e7a3022559 _erealloc.cold (php-fpm + 0x222559)#012#5  
0x000055e7a33276e3 zend_vstrpprintf (php-fpm + 0x5276e3)#012#6  
0x000055e7a3020041 php_verror (php-fpm + 0x220041)#012#7  
0x000055e7a30205cf php_error_docref (php-fpm + 0x2205cf)#012#8  
0x00007f8d3925aea1 zif_zstd_uncompress_dict (zstd.so + 0x4ea1)#012#9  
0x000055e7a32c8903 execute_ex (php-fpm + 0x4c8903)#012#10 
0x000055e7a32c1715 zend_execute (php-fpm + 0x4c1715)#012#11 
0x000055e7a3328e44 zend_execute_script (php-fpm + 0x528e44)#012#12 
0x000055e7a31c1a18 php_execute_script_ex (php-fpm + 0x3c1a18)#012#13 
0x000055e7a3046aa3 main (php-fpm + 0x246aa3)#012#14 
0x00007f8d3d6295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#15 
0x00007f8d3d629680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#16 
0x000055e7a30488f5 _start (php-fpm + 0x2488f5)#012ELF object binary architecture: AMD x86-64

[kjdev]

unsigned long long const rSize = ZSTD_getFrameContentSize(input, input_len); // rSize = -1

Frame size is not acquired, and in that case, it is not processed properly.

I've compressed it myself, and it doesn't generate any errors.
I guess it depends on the state of the compressed file.

How did you create the compressed file?
Include the version, etc.

[scottchiefbaker]

I used the zstd CLI command and ran a command similar to:

cat /var/tmp/test.txt | /usr/bin/zstd -q -D /var/tmp/dict -o /tmp/out.txt.zst

[kjdev]

In case of ZSTD_CONTENTSIZE_UNKNOWN (-1), it seems to be necessary to process in streaming mode.

[scottchiefbaker]

To be fully transparent I created the file in Perl by shelling out the zstd CLI program where I fed it the string input via STDIN and captured the STDOUT back into Perl.

sub zstd_comp_with_dict {
    my ($str, $dict_file) = @_;

    my $cmd = "/usr/bin/zstd -q -D $dict_file --stdout";
    my @cmd = split(' ', $cmd);

    # Open the command with various file handles
    my $pid = open3(my $chld_in, my $chld_out, my $chld_err = gensym, @cmd);
    binmode($chld_out, ":raw");

    # Write to the STDIN of the process
    print $chld_in $str;
    close($chld_in);

    # Read the STDOUT from the process
    local $/ = undef; # Input rec separator (slurp)
    my $ret  = readline($chld_out);

    waitpid($pid, 0);

    return $ret;
}

The zstd CLI is doing all the heavy lifting. When I recreate the file using the CLI:

cat test.txt | zstd -q -D dict --stdout > test.txt.zst

I get the same 666 byte file. Maybe this puts zstd into streaming mode because the input is coming from STDIN and the output is going to STDOUT?

[kjdev]

As you pointed out, compressed in streaming mode seemed to cause an error.

We will fix it to support decompression in streaming mode as well.

[scottchiefbaker]

Actually you bring up a good point... now that we know it's a streaming issue.

It would be better to do this in non-streaming mode to give zstd more information about how to compress. The only reason I did it the way I did was because the built in Perl libraries don't support dictionaries, and when I shelled out I didn't want to have to worry about removing temp files.

[scottchiefbaker]

FWIW I went back and re-compressed all my data in non-stream mode and now php-ext-zstd can decompress them just fine.

[kjdev]

It seemed to be possible to decompress by using the stream function

$dictionary = file_get_contents(__DIR__ . '/dict');
$ctx = stream_context_create([
    "zstd" => [
        "dict"  => $dictionary,
    ]
]);
$file = __DIR__ . '/test.txt.zst';
$decomp = file_get_contents('compress.zstd://' . $file, false, $ctx);
var_dump(md5($decomp)); // string(32) "58834ab4da8f93dc8f9864e9d9fdfb86"

[kjdev]

fix 80a5099