Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ext_proc filter crash when sending trailers when request has no body #27430

Merged
merged 3 commits into from
May 19, 2023

Conversation

yanjunxiang-google
Copy link
Contributor

ext_proc filter crash when sending trailers when request has no body.

The code is assuming the request always has body available and stored in the ext_proc buffer state when processing trailers, which leads into the crash.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
@yanjunxiang-google
Copy link
Contributor Author

/assign @adisuissa @yanavlasov

@yanjunxiang-google
Copy link
Contributor Author

yanjunxiang-google commented May 16, 2023

The detail fuzz report could be found here: https://clusterfuzz.corp.google.com/testcase-detail/5547889199546368

The crash traceback decode is:

[2023-05-16 15:50:05.109][2540875][info][misc] [test/test_common/test_random_generator.cc:20] TestRandomGenerator running with seed -38107400
[2023-05-16 15:50:05.171][2540875][debug][misc] [./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:144] Decoding headers (end_stream=false):
':path', '/foo'
':method', 'GET'
':authority', 'foo.com'

[2023-05-16 15:50:05.171][2540875][trace][ext_proc] [source/extensions/filters/http/ext_proc/ext_proc.cc:183] decodeHeaders: end_stream = false
[2023-05-16 15:50:05.172][2540875][trace][ext_proc] [source/extensions/filters/http/ext_proc/ext_proc.cc:190] decodeHeaders: Skipped
[2023-05-16 15:50:05.172][2540875][debug][misc] [./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:103] Finished with FilterHeadersStatus: 0
[2023-05-16 15:50:05.172][2540875][debug][misc] [./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:209] Decoding trailers:
'.', ''

[2023-05-16 15:50:05.172][2540875][trace][ext_proc] [source/extensions/filters/http/ext_proc/ext_proc.cc:467] decodeTrailers
source/extensions/filters/http/ext_proc/ext_proc.h:198:26: runtime error: reference binding to null pointer of type 'const Buffer::Instance'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior source/extensions/filters/http/ext_proc/ext_proc.h:198:26 in
AddressSanitizer:DEADLYSIGNAL

==2540875==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x00000298d66c bp 0x7ffd605d7550 sp 0x7ffd605d71a0 T0)
==2540875==The signal is caused by a READ memory access.
==2540875==Hint: address points to the zero page.
error: failed to decompress '.debug_aranges', zlib is not available
error: failed to decompress '.debug_info', zlib is not available
error: failed to decompress '.debug_abbrev', zlib is not available
error: failed to decompress '.debug_line', zlib is not available
error: failed to decompress '.debug_str', zlib is not available
error: failed to decompress '.debug_line_str', zlib is not available
error: failed to decompress '.debug_loclists', zlib is not available
error: failed to decompress '.debug_rnglists', zlib is not available
#0 0x298d66c in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::sendBodyChunk(Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState&, Envoy::Buffer::Instance const&, Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::CallbackState, bool) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:508:3
#1 0x299d958 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::sendBufferedData(Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState&, Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::CallbackState, bool) /proc/self/cwd/./source/extensions/filters/http/ext_proc/ext_proc.h:198:5
#2 0x2990563 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onTrailers(Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState&, Envoy::Http::HeaderMap&) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:441:5
#3 0x2991101 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::decodeTrailers(Envoy::Http::RequestTrailerMap&) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:468:23
#4 0x291cc46 in void Envoy::Extensions::HttpFilters::HttpFilterFuzzer::sendTrailersEnvoy::Http::StreamDecoderFilter(Envoy::Http::StreamDecoderFilter*, test::fuzz::HttpData const&) /proc/self/cwd/./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:210:11
#5 0x282205e in void Envoy::Extensions::HttpFilters::HttpFilterFuzzer::runDataEnvoy::Http::StreamDecoderFilter(Envoy::Http::StreamDecoderFilter*, test::fuzz::HttpData const&) /proc/self/cwd/./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:124:5
#6 0x2810d43 in LLVMFuzzerTestOneInput /proc/self/cwd/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz.cc:93:10
#7 0x5156d9a in Envoy::(anonymous namespace)::FuzzerCorpusTest_RunOneCorpusFile_Test::TestBody() /proc/self/cwd/test/fuzz/main.cc:50:3
#8 0x77ea007 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2580:10
#9 0x77bfffb in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2655:5
#10 0x77c1630 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2832:11
#11 0x77c2c4a in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2986:28
#12 0x77dbe4b in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5697:44
#13 0x77ed857 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2580:10
#14 0x77db3fd in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5280:10
#15 0x51550d5 in main /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2485:46
#16 0x7fe4e55a7189 (/lib/x86_64-linux-gnu/libc.so.6+0x27189) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#17 0x7fe4e55a7244 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x27244) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#18 0x275232d in _start (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/21808fd728b94b71544c4afa1615227e/execroot/envoy/bazel-out/k8-dbg/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x275232d)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:508:3 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::sendBodyChunk(Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState&, Envoy::Buffer::Instance const&, Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::CallbackState, bool)
==2540875==ABORTING

@yanjunxiang-google
Copy link
Contributor Author

The test case to reproduce the crash is:

cat test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_corpus/clusterfuzz-testcase-minimized-ext_proc_unit_test_fuzz-5547889199546368.test
config {
grpc_service {
envoy_grpc {
cluster_name: "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
retry_policy {
}
}
timeout {
}
}
processing_mode {
request_header_mode: SKIP
request_body_mode: BUFFERED
}
request_attributes: "\022."
}
request {
trailers {
headers {
key: "."
}
}
}
response {
response_body {
response {
header_mutation {
}
}
}
mode_override {
request_header_mode: SKIP
}
}

Copy link
Contributor

@adisuissa adisuissa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!
I don't really know the inner details of ext-proc, so I just left some high-level observations.
@mpwarres can you PTAL?

@@ -195,7 +195,9 @@ class Filter : public Logger::Loggable<Logger::Id::ext_proc>,

void sendBufferedData(ProcessorState& state, ProcessorState::CallbackState new_state,
bool end_stream) {
sendBodyChunk(state, *state.bufferedData(), new_state, end_stream);
if (state.hasBufferedData()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not an expert on ext-proc, so high-level observation.
IIUC the current fix prevents sending a body of length 0 with end_stream. If this is the desired behavior, then I think it is ok.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comments!
Actually, ext_proc filter supports sending a body of length 0:

TEST_P(ExtProcIntegrationTest, BufferBodyOverridePostWithEmptyBody) {

This change is causing test failures. So, we changed it into if the no buffered data, then sends an empty body.

After that, we seeing another issue with stream_ pointer as null, so the new change also fixed it. Please check the root cause comments for details.

@@ -195,7 +195,9 @@ class Filter : public Logger::Loggable<Logger::Id::ext_proc>,

void sendBufferedData(ProcessorState& state, ProcessorState::CallbackState new_state,
bool end_stream) {
sendBodyChunk(state, *state.bufferedData(), new_state, end_stream);
if (state.hasBufferedData()) {
sendBodyChunk(state, *state.bufferedData(), new_state, end_stream);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another observation: the call in case of an empty body, will not change the state to new_state. Is this the desired outcome?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's a good catch. We should send an empty body if no buffered data as in the new diffs.

@mpwarres
Copy link
Contributor

@stevenzzzz as an additional pair of eyes

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
@yanjunxiang-google
Copy link
Contributor Author

This fuzzer test cases exposed two issues in the ext_proc filter.

If the filter processing mode is set into request header: SKIP and request body :BUFFERED, at same time, when the request only contains headers and trailers then it hit two distinct crash:

  1. The state.bufferedData() is null pointer in sendBufferedData(). Accessing this null instance pointer causing crash.
  2. There is no gRPC stream setup(stream_ is null ptr) in the function call path: onTrailers()->sendBufferedData()-> sendBodyChunk()->stream_->send() , which caused crash.

@yanjunxiang-google
Copy link
Contributor Author

/assign @htuch

Copy link
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be good to add this scenario as a test case in one of the suites (filter or integration). As I think it is hard to read what is going form just the corpus file.

/wait

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
@yanjunxiang-google
Copy link
Contributor Author

I think it would be good to add this scenario as a test case in one of the suites (filter or integration). As I think it is hard to read what is going form just the corpus file.

/wait

Done!

@yanavlasov yanavlasov merged commit 1fd0b99 into envoyproxy:main May 19, 2023
reskin89 pushed a commit to reskin89/envoy that referenced this pull request Jul 11, 2023
…nvoyproxy#27430)

* ext_proc filter crash when sending trailers without sending body.

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
Signed-off-by: Ryan Eskin <ryan.eskin89@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants