[Bug #18358] Fix crash in zlib when in progress

[peterzhu2118]

Redmine ticket: https://bugs.ruby-lang.org/issues/18358

zlib has a use-after-free when Zlib::Inflate#inflate or Zlib::Deflate#deflate is called recursively. The following script demonstrates the issue.

require "zlib"
require "securerandom"

GC.stress = true
zi = Zlib::Inflate.new

s = Zlib.deflate(SecureRandom.random_bytes(1024**2))

zi.inflate(s) do
  zi.inflate(s)
end

If we run Ruby master (commit b680b632e5b88e4ea550de3f15cf6ef782efeb48) with Valgrind, we see use-after-free errors:

$ valgrind --leak-check=no --undef-value-errors=no ruby test.rb
==36376== Memcheck, a memory error detector
==36376== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==36376== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==36376== Command: ruby test.rb
==36376==
==36376== Warning: client switching stacks?  SP change: 0x1ffe8020e0 --> 0x1fff0001b0
==36376==          to suppress, use: --max-stackframe=8380624 or greater
==36376== Invalid read of size 8
==36376==    at 0x484661F: memmove (vg_replace_strmem.c:1382)
==36376==    by 0x2B967D: memcpy (string_fortified.h:34)
==36376==    by 0x2B967D: ruby_nonempty_memcpy (memory.h:659)
==36376==    by 0x2B967D: ruby_nonempty_memcpy (memory.h:656)
==36376==    by 0x2B967D: str_buf_cat (string.c:3148)
==36376==    by 0xA8FE30B: zstream_append_input (zlib.c:869)
==36376==    by 0xA8FE30B: zstream_run0 (zlib.c:1135)
==36376==    by 0xA8FE30B: zstream_run_synchronized (zlib.c:1156)
==36376==    by 0x144B73: rb_ensure (eval.c:993)
==36376==    by 0xA8FA4D5: zstream_run (zlib.c:1168)
==36376==    by 0xA8FA4D5: do_inflate (zlib.c:2041)
==36376==    by 0xA8FC1EF: rb_inflate_inflate (zlib.c:2159)
==36376==    by 0x31E36A: vm_call_cfunc_with_frame (vm_insnhelper.c:3045)
==36376==    by 0x32ABF8: vm_call_method_each_type (vm_insnhelper.c:3647)
==36376==    by 0x32B4C3: vm_call_method (vm_insnhelper.c:3758)
==36376==    by 0x338B86: vm_sendish (vm_insnhelper.c:4759)
==36376==    by 0x338B86: vm_exec_core (insns.def:758)
==36376==    by 0x32A07C: rb_vm_exec (vm.c:2214)
==36376==    by 0x13EB5A: rb_ec_exec_node (eval.c:280)
==36376==  Address 0xb110050 is 16 bytes inside an unallocated block of size 1,048,912 in arena "client"

The script does not crash on Ruby master, but crashes on 3.0.2, 2.7.4, 2.6.8.

test.rb:9: [BUG] Segmentation fault at 0x00005562d60242a6
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0003 p:---- s:0013 e:000012 CFUNC  :inflate
c:0002 p:0080 s:0008 E:000b10 EVAL   test.rb:9 [FINISH]
c:0001 p:0000 s:0003 E:000460 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
test.rb:9:in `<main>'
test.rb:9:in `inflate'

-- Machine register context ------------------------------------------------
 RIP: 0x00007fccd5afc898 RBP: 0x00005562d5f24190 RSP: 0x00007fff9c5d01b8
 RAX: 0x00007fccd1349010 RBX: 0x000000000017ffff RCX: 0x0000000000100136
 RDX: 0x0000000000100136 RDI: 0x00007fccd1349010 RSI: 0x00005562d5f24190
  R8: 0x00007fccd1349010  R9: 0x0000000000000000 R10: 0x0000000000000022
 R11: 0x0000000000000246 R12: 0x00005562d5e17cc8 R13: 0x0000000000000000
 R14: 0x0000000000100136 R15: 0xffffffffffffffff EFL: 0x0000000000010202

-- C level backtrace information -------------------------------------------
/home/peter/.rubies/ruby-3.0.2/bin/ruby(rb_print_backtrace+0x11) [0x5562d5533c12] vm_dump.c:758
/home/peter/.rubies/ruby-3.0.2/bin/ruby(rb_vm_bugreport) vm_dump.c:998
/home/peter/.rubies/ruby-3.0.2/bin/ruby(rb_bug_for_fatal_signal+0xec) [0x5562d55daa3c] error.c:786
/home/peter/.rubies/ruby-3.0.2/bin/ruby(sigsegv+0x4d) [0x5562d5489ebd] signal.c:960
/lib/x86_64-linux-gnu/libpthread.so.0(__restore_rt+0x0) [0x7fccd5d123c0] ../sysdeps/pthread/funlockfile.c:28
/lib/x86_64-linux-gnu/libc.so.6(0x7fccd5afc898) [0x7fccd5afc898]
/home/peter/.rubies/ruby-3.0.2/bin/ruby(RB_FL_TEST_RAW+0x0) [0x5562d54a72ff] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
/home/peter/.rubies/ruby-3.0.2/bin/ruby(STR_EMBED_P) ./internal/string.h:97
/home/peter/.rubies/ruby-3.0.2/bin/ruby(str_buf_cat) string.c:2928
/home/peter/.rubies/ruby-3.0.2/lib/ruby/3.0.0/x86_64-linux/zlib.so(zstream_append_input+0x11) [0x7fccd14d348f] zlib.c:863
/home/peter/.rubies/ruby-3.0.2/lib/ruby/3.0.0/x86_64-linux/zlib.so(zstream_run) zlib.c:1124
/home/peter/.rubies/ruby-3.0.2/lib/ruby/3.0.0/x86_64-linux/zlib.so(do_inflate+0x54) [0x7fccd14d4d14] zlib.c:2004
/home/peter/.rubies/ruby-3.0.2/lib/ruby/3.0.0/x86_64-linux/zlib.so(rb_inflate_inflate+0x1d8) [0x7fccd14d63c8] zlib.c:2122
/home/peter/.rubies/ruby-3.0.2/bin/ruby(vm_call_cfunc_with_frame+0x11b) [0x5562d550be8b] vm_insnhelper.c:2926
/home/peter/.rubies/ruby-3.0.2/bin/ruby(vm_call_method_each_type+0x79) [0x5562d5517d19] vm_insnhelper.c:3416
/home/peter/.rubies/ruby-3.0.2/bin/ruby(vm_call_method+0xb4) [0x5562d5518384] vm_insnhelper.c:3534
/home/peter/.rubies/ruby-3.0.2/bin/ruby(vm_sendish+0x130) [0x5562d550ede0] vm_insnhelper.c:4527
/home/peter/.rubies/ruby-3.0.2/bin/ruby(vm_exec_core+0x1f8) [0x5562d5522578] insns.def:770
/home/peter/.rubies/ruby-3.0.2/bin/ruby(rb_vm_exec+0x97d) [0x5562d55171cd] vm.c:2172
/home/peter/.rubies/ruby-3.0.2/bin/ruby(rb_ec_exec_node+0xed) [0x5562d53362bd] eval.c:317
/home/peter/.rubies/ruby-3.0.2/bin/ruby(ruby_run_node+0x4f) [0x5562d533aedf] eval.c:375
/home/peter/.rubies/ruby-3.0.2/bin/ruby(main+0x5f) [0x5562d533600f] error.c:3076