error during reading http1.1 response body with trailers

[strobil]

Bug Report

What is the issue?

linkerd proxy unable to read http1.1 response with trailers

How can it be reproduced?

http server must respond with chunked transfer-encoding with trailers, example (wireshark dump) in attachment below

http_response_with_trailers.zip

Logs, error output, etc

proxy DEBUG log

[   904.887268887s] DEBUG inbound: linkerd2_proxy_http::glue: http body error: error reading a body from connection: Invalid chunk end CR
[   904.887340506s] DEBUG inbound: tokio_reactor: dropping I/O source: 69
[   904.887382139s] DEBUG inbound: linkerd2_app_core::accept_error: Connection failed error=error from user's Payload stream: error reading a body from connection: Invalid chunk end CR
[   904.887421496s] DEBUG inbound:accept{peer.addr=10.1.1.1:57056}:source{target.addr=10.42.5.221:8080}:http1{name=dev.api.sweet.tv:80 port=8080 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:target:http1: hyper::proto::h1::conn: decode stream error: Invalid chunk end CR
[   904.887514171s] DEBUG tokio_reactor: dropping I/O source: 61
[   904.887587594s] DEBUG tokio_reactor: loop process - 1 events, 0.000s
[   904.887614621s] DEBUG inbound: linkerd2_proxy_http::glue: http body error: error reading a body from connection: Invalid chunk end CR
[   904.887716575s] DEBUG inbound: tokio_reactor: dropping I/O source: 77
[   904.887748318s] DEBUG inbound: linkerd2_app_core::accept_error: Connection failed error=error from user's Payload stream: error reading a body from connection: Invalid chunk end CR

linkerd check output

strobil@strobil-pc ~ $ linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-ha-checks
-----------------
√ pod injection disabled on kube-system
√ multiple replicas of control plane pods

linkerd-addons
--------------
√ 'linkerd-config-addons' config map exists

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are √

Environment

• Kubernetes Version: v1.17.6
• Cluster Environment: Bare metal
• Host OS: CentOS Linux release 7.8.2003 Kernel: 5.7.0-1.el7.elrepo.x86_64
• Linkerd version: stable-2.8.1

Possible solution

It looks like a bug in hyperium/hyper

Additional context

We are using grpc-gateway as a REST proxy for our GRPC services.

grpc-gateway is adding Grpc-Trailer-Content-Type: application/grpc trailer to the end of response body, then, linkerd2 proxy unable to read it correctly and request fails with net::ERR_INCOMPLETE_CHUNKED_ENCODING in Google Chrome browser.

According to example on this page - http response from a server is absolutely legitimate.

[strobil]

Looks like i found the reason of this. hyperium/hyper#2171
As @seanmonstar says - hyper doesn't have h1 trailers support...

[cpretzer]

@strobil thanks for making us aware of this and for your PR. Once the PR to grpc-ecosystem/grpc-gateway is released, we can pull the changes into the proxy. We'd love your help with a pull request to that repo 😄

[strobil]

@cpretzer hello again!

My PR was merged to grpc-ecosystem/grpc-gateway, but this is more likely to be a workaround rather then fix, because of the root cause of this problem stays in the hyper source code, which is used by linkerd2-proxy.

I think we could close this issue for now, but this problem can appear again with some other software that are using HTTP 1.1 Trailers in its code.

What do you think?

[cpretzer]

hi @strobil, thanks for the update

When we triaged this issue, and once the hyper issue is released, we'll pull that version into the proxy codebase.

I'll close this for now since the grpc-gateway workaround is in place.