Segfault in `Context::onResponseHeaders()`

[mandarjog]

1.3.0-rc3

[2019-09-10 14:41:33.541][1820][critical][main] [external/envoy/source/exe/terminate_handler.cc:13] std::terminate called! (possible uncaught exception, see trace)
[2019-09-10 14:41:33.541][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-09-10 14:41:33.549][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: Envoy::TerminateHandler::logOnTerminate()::$_0::operator()() [0x1fdf7d8]
[2019-09-10 14:41:33.558][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: [0x1fdf6e9]
[2019-09-10 14:41:33.565][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: std::__terminate() [0x2200473]
[2019-09-10 14:41:33.573][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: std::__1::__function::__func<>::operator()() [0xba846d]
[2019-09-10 14:41:33.580][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Extensions::Common::Wasm::Context::onResponseHeaders() [0xe14774]
[2019-09-10 14:41:33.587][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Extensions::Common::Wasm::Context::encodeHeaders() [0xe1a68f]
[2019-09-10 14:41:33.595][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeHeaders() [0x1cbaf8b]
[2019-09-10 14:41:33.603][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Http::Utility::sendLocalReply() [0x1d69379]
[2019-09-10 14:41:33.612][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply() [0x1cb7161]
[2019-09-10 14:41:33.621][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders() [0x1cb7f2e]
[2019-09-10 14:41:33.630][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete() [0x1cca46b]
[2019-09-10 14:41:33.638][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase() [0x1cc93b9]
[2019-09-10 14:41:33.647][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: Envoy::Http::Http1::ConnectionImpl::$_7::__invoke() [0x1cccc7d]
[2019-09-10 14:41:33.656][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: http_parser_execute [0x1d832cb]
[2019-09-10 14:41:33.664][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0x1cc8728]
[2019-09-10 14:41:33.674][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x1cc851f]
[2019-09-10 14:41:33.682][1820][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Http::ConnectionManagerImpl::onData() [0x1cb49e9]

[mandarjog]

@jplevyak @kyessenov

[kyessenov]

Looks like sendLocalReply is the critical bit.

[jplevyak]

What can I do to repro this?

[jplevyak]

This is very odd as the only the only throw in std::function is for a null std::function and we test right about the call if it is null:

Http::FilterHeadersStatus Context::onResponseHeaders() {
if (!wasm_->onResponseHeaders_)
return Http::FilterHeadersStatus::Continue;
if (wasm_->onResponseHeaders_(this, id_) == 0) {
return Http::FilterHeadersStatus::Continue;
}
return Http::FilterHeadersStatus::StopIteration;
}

[jplevyak]

So, without a way to repro I don't have a path forward.

[mandarjog]

SendLocalReply is from the blackhole cluster.
Try hitting something that will hit the blackhole cluster. I will try a repro tomorrow.