bug: grpc-web unexpected error "upstream grpc status not received"

[Tereius]

Current Behavior

Hi,

I am in the process of replacing Envoy with Apisix for grpc-web request transformation. My tests show that many things seem to work already, but I experienced a bug with the following edge case:
I have a server side grpc streaming response that immediately returns an error when called. This results in two calls of the error handler function on the frontend. The first error contains the following unexpected error message: upstream grpc status not received
The second error contains the expected error message from the server.
With Envoy only the expected server error is received on the frontend.

Expected Behavior

Only the expected server error is delivered to the frontend.

Error Logs

No response

Steps to Reproduce

I created a minimal example to reproduce this edge case here:
Here you find the immediate error response on the server side.
Here you find the frontend error handler that gets called twice.

Environment

• APISIX version (run apisix version): 3.12.0
• Operating system (run uname -a): Linux 6.14.3-zen1-1-zen
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.27.1.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): -
• APISIX Dashboard version, if relevant: -
• Plugin runner version, for issues related to plugin runners: -
• LuaRocks version, for installation issues (run luarocks --version): -

[Tereius]

I have done some research. It seems like the Nginx variables ctx.var.upstream_trailer_grpc_status and ctx.var.upstream_trailer_grpc_message contain nil for the described edge case. But in Wireshark I can actually see the grpc status and message in the grpc trailer sent from upstream

I have patched the grpc-web.lua plugin, which fixes the problem for me. But this is only a workaround and not a clean solution. For a clean solution, it is necessary to investigate why the two Nginx variables do not contain the expected values.

diff --git a/apisix/plugins/grpc-web.lua b/apisix/plugins/grpc-web.lua
index 43a075c4..0960370a 100644
--- a/apisix/plugins/grpc-web.lua
+++ b/apisix/plugins/grpc-web.lua
@@ -213,15 +213,16 @@ function _M.body_filter(conf, ctx)
 
         -- When the response body completes and still does not receive the grpc status
         local resp_ok = status ~= nil and status ~= ""
-        local trailer_buf = build_trailer(
-            resp_ok and status  or 2,
-            resp_ok and message or "upstream grpc status not received"
-        )
-        if ctx.grpc_web_encoding == CONTENT_ENCODING_BASE64 then
-            trailer_buf = encode_base64(trailer_buf)
+        if resp_ok then
+            local trailer_buf = build_trailer(
+                status,
+                message
+            )
+            if ctx.grpc_web_encoding == CONTENT_ENCODING_BASE64 then
+                trailer_buf = encode_base64(trailer_buf)
+            end
+            ngx_arg[1] = ngx_arg[1] .. trailer_buf
         end
-
-        ngx_arg[1] = ngx_arg[1] .. trailer_buf
     end
 end

[SkyeYoung]

Hi, First of all, thank you very much for providing such a good minimal reproduction environment.

I spent a lot of time using gdb to trace functions in nginx, but to no avail.

But @bzp2010 prompts me that nginx may not parse correctly without Header Trailer. (This is how it is in http/1.1)

So I tried change your minimal example:

// cmd/main.go
func (s *server) SayHello(req *example.HelloRequest, stream grpc.ServerStreamingServer[example.HelloReply]) error {
+	trailerMD := metadata.Pairs(
+		"Trailer", "grpc-message, grpc-status",
+	)
+
+	stream.SendHeader(trailerMD)
	return status.Errorf(codes.Internal, "this error is expected")
}

It works.

BEFORE	AFTER

Then I tried further and found that sending an empty header also works

// cmd/main.go
func (s *server) SayHello(req *example.HelloRequest, stream grpc.ServerStreamingServer[example.HelloReply]) error {
+	trailerMD := metadata.Pairs()
+
+	stream.SendHeader(trailerMD)
	return status.Errorf(codes.Internal, "this error is expected")
}

I made a further comparison.

BEFORE	AFTER

---

For your use case, besides the solution you provided, please see if it's possible to inject the above-mentioned method before returning as a temporary solution.