Mixed tcp and http protocol result in protocol errors

[jakubdyszkiewicz]

What happened?

Steps to reproduce

1. Deploy kuma marketplace demo
2. Add second service

apiVersion: v1
kind: Service
metadata:
  annotations:
    3001.service.kuma.io/protocol: tcp # <- notice that the first service has http here
  name: backend-v2
  namespace: kuma-demo
spec:
  ports:
  - name: api
    port: 3001
    protocol: TCP
    targetPort: 3001
  selector:
    app: kuma-demo-backend

3. Exec to container and execute requests

❯❯❯ kubectl exec -n kuma-demo kuma-demo-app-6787b4f7f5-qxjzz -ti -- sh
/ # while curl backend:3001; do echo "" && sleep 0.1; done

see that some of them fails with protocol error

Hello World! Marketplace with sales and reviews made with <3 by the OCTO team at Kong Inc.
Hello World! Marketplace with sales and reviews made with <3 by the OCTO team at Kong Inc.
upstream connect error or disconnect/reset before headers. reset reason: protocol error
upstream connect error or disconnect/reset before headers. reset reason: protocol error
Hello World! Marketplace with sales and reviews made with <3 by the OCTO team at Kong Inc.
upstream connect error or disconnect/reset before headers. reset reason: protocol error
Hello World! Marketplace with sales and reviews made with <3 by the OCTO team at Kong Inc.
Hello World! Marketplace with sales and reviews made with <3 by the OCTO team at Kong Inc.

[michaelbeaumont]

Suspicion: the client is configured to expect an HTTP upstream and so it will upgrade its outbound listener to HTTP2 and send HTTP2 requests, but the upstream will see HTTP2 directly, its sidecar being configured for TCP, which perhaps it can't handle.

The upstream has two entries in spec.networking.inbound, one with protocol: tcp, one with protocol: http but the last spec.networking.inbound wins when configuring the inbound Envoy listener. If tcp wins, this error appears. Not upgrading the outbound connection on the downstream from HTTP to HTTP2 fixes the error.

Options:

1. Configure the highest common protocol on the inbound
2. Don't upgrade to HTTP2 but it's unclear when not to upgrade

[lahabana]

xref maybe: #2445 ?

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[lukidzi]

@michaelbeaumont mentioned that configuring the protocol isn't straightforward. When multiple services point to the same address but with different protocols, the last one takes precedence. It seems odd to find the highest common protocol on the inbound side because the outbound configuration still relies on the protocol defined by the specific cluster. Not sure what is the best solution here. So far we are not doing any infer protocol on inbounds.

[lahabana]

When multiple services point to the same address but with different protocols, the last one takes precedence.
@lukidzi could you show an example?

[lukidzi]

apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: kuma-demo
  annotations:
    3001.service.kuma.io/protocol: "http"
spec:
  selector:
    app: kuma-demo-backend
  ports:
  - name: api
    port: 3001
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    3001.service.kuma.io/protocol: tcp # <- notice that the first service has http here
  name: backend-v2
  namespace: kuma-demo
spec:
  ports:
  - name: api
    port: 3001
    protocol: TCP
    targetPort: 3001
  selector:
    app: kuma-demo-backend

2nd service is going to be used.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[lahabana]

Is this no longer an issue with MeshService?

[jakubdyszkiewicz]

It's no longer an issue with MeshService