Fix use-after-free when a recursive extension proc rescues, skips or resets

[Watson1978]

A recursive extension proc that rescued an error from the unpacker, called Unpacker#skip, or called Unpacker#reset could drop stack.depth below the enclosing containers. Those containers were then no longer GC-marked while the proc ran, so they could be collected before the parser wrote to them after the proc returned. This follows up on #405, which fixed the depth underflow but left this window open.

The unpacker now tracks the innermost recursive barrier as a stack floor. Error handling and reset rewind the depth to the floor instead of 0, and skip stops at the barrier like read does.

Reproduction

require "msgpack"

factory = MessagePack::Factory.new
factory.register_type(
  0x01, Object,
  packer: ->(_obj, packer) { packer.write(nil) },
  unpacker: ->(u) {
    begin
      u.read
    rescue Exception
    end
    Object.new
  },
  recursive: true,
)

payload = "\x92\xd4\x01\xc1\x2a".b  # [ recursive-ext(type1, invalid body), 42 ]

GC.stress = true
100.times do
  begin
    factory.unpack(payload)
  rescue MessagePack::UnpackError, EOFError, StandardError
  end
end
puts "no crash"

With 1.8.5 the script crashes with a SIGSEGV. With this change it prints no crash.

🤖 Generated with Claude Code