OnRequestBody not called if onRequestHeaders returns stopIteration

[bladedancer]

I'll preface this with it could be user error but my scenario is I'm making a grpc call out to some backend service. The backend server will mutate both the headers and the body. So I need to execute both onRequestHeaders and onRequestBody before getting the result.

The problem is it hangs indefinitely, but I'm told it should work. Assuming it did work once then I believe it's related to this change:

#95

Sampe code is here: https://github.com/bladedancer/envoy-wasm-stopitertest/blob/main/wasm-filter/context.cc#L32.

Is this scenario achievable? Is it a bug or the expected behaviour?

[PiotrSikora]

It's the expected behavior. It did work in the past, but such behavior is error-prone and resulted in a lot of weird edge cases.

Why exactly do you need to block headers?

[bladedancer]

I have a legacy decision point that potentially operates on the headers and the body. My problem is that I cannot mutate the headers after sending the body to the service... Because the only way i can send the body is to continue the headers.

[PiotrSikora]

The question is why do you need to modify headers after mutating the body?

[bladedancer]

I haven't mutated the body at this point - I'm forwarding the entire request to the decision point. It will look at the entire message - body and headers and apply policy to the request - it may redact headers, transcode content, enrich content, add headers, etc. I was using the wasm instead of lua (which I believe can do the header modification) here because I wanted to use grpc for the communication.

I suppose the question I'd have is - is this a fundamental design decision for the wasm filters? I know you say it's "error prone", and I'm sure I could misuse it but it also opens up scope for legacy integrations like this.

[wbpcode]

😢 I encountered the same problem. I wanted to implement a custom ext auth filter using wasm, but found that I couldn't stop processing headers and continue processing bodies, so I couldn't send my entire request to an external authentication service and decide my next action based on the results.

[PiotrSikora]

It's a bit of a fundamental issue. Buffering complete requests/responses doesn't work at scale.

If you offload buffering to the proxy, then you're going to hit proxy buffer limits pretty fast (those are usually 16/32/64 KiB).

If you start buffering inside WasmVM, then you're going to hit Wasm memory limits pretty fast... unless there is no upper limit, in which case you're risking OOM or perceived memory leaks (currently Wasm memory can grow, but it cannot shrink back when unused).

Basically, buffering works when there is little traffic, but it either breaks under load or requires extreme care and awarness of the memory limits when developing a plugin, so not the most friendly feature.

Having said that, it seems like there are legitimate use cases for it, and we might need to support them, but it's unlikely to happen in the next few weeks.

@bladedancer if I'm understanding your use case correctly, then you could stream the original request to the authorization server, and stream the response from the authorization server to the backend, without any buffering. There is no need to buffer the original request, correct?

@wbpcode on the other hand, you need to always buffer complete request, and send it to the authorization server and the backend, correct?

[wbpcode]

Yep, but it will do some limit to buffer size. For example, 4 Kb.

[bladedancer]

@PiotrSikora yes, in the policy decision point the goal will be to minimize the scenarios that require buffering but if buffering is required it will be done there, not in envoy.

As an aside - I tried with the ext_proc also, that resulted in this one envoyproxy/envoy#15798

[nullpo-head]

@PiotrSikora We've encountered the same problem. In our case, we don't modify the request header, but our backend server decides whether to deny or allow it after the whole request is processed. So, we want to stop the request headers until the request body is processed.

There seems to be a fair amount of requests for the ability to reject/modify headers after inspecting the request/response body. So do you have any idea when to proceed with this feature? Also, are there any other obstacles besides the buffer limitation issue?

• add new HTTP header based on data from body proxy-wasm-rust-sdk#114
• modify response headers values based on response body contents proxy-wasm-cpp-sdk#62
• https://envoyproxy.slack.com/archives/CM2CQ4XJ5/p1623261646039100

[pdeligia]

+1 to several valid use cases for this.

We have also encountered the same problem. Besides what @bladedancer and @nullpo-head mentioned, which is really useful in various scenarios where you need both headers+body to enforce some policy, we also have a scenario where there might be some processing of the body (or parts of the body) -- e.g. decryption or some aggregation -- and then a header has to be updated accordingly (e.g. the body might have different length, so the content-length header would be different). And all these have to be done inside the proxy before they reach the destination. All these scenarios are sadly not possible today.

This is somewhat related to the other issue we discussed before @PiotrSikora envoyproxy/envoy#17208, in general being able to set headers/bodies outside the scope of the respective callbacks would be extremely useful feature.