bug: istio gateway omits headers

[EyalPazz]

What happened:

istio for both vllm-sim and cpu based model's seems to omit the headers for some reason, i get these errors:
Failed to read and parse request body, unexpected end of JSON input

{"object":"error","message":"[{'type': 'missing', 'loc': ('body',), 'msg': 'Field required', 'input': None}]","type":"Bad Request","param":null,"code":400}

How to reproduce it (as minimally and precisely as possible):
Just follow the Getting Started Guide for istio
Anything else we need to know?:
It did work using Kgateway, and also when making the request from inside the pod, so i don't think the problem is in vllm/vllm-sim or in epp

Environment:
** M4 Pro Chip **

• Kubernetes version (use kubectl version): v1.33.1
• Inference extension version: v0.3.0
• Cloud provider or hardware configuration: Kind using Cloud Provider Kind for exposing the LB

cc @nirrozenbaum @danehans

[nirrozenbaum]

cc @LiorLieberman @shaneutt

[shaneutt]

cc @robscott @aslakknutsen

[danehans]

cc: @howardjohn

[LiorLieberman]

/assign

[LiorLieberman]

@nirrozenbaum @shaneutt do you have a list of bugs? I recall issues on different repositories.
will try to fix them in one goal

[nirrozenbaum]

created a thread on GIE slack channel and tagged both of you @LiorLieberman @shaneutt

[keithmattix]

Oh I think I ran into this issue! From what I understand, the issue is that the DestinationRule to configure SIMPLE TLS is hardcoded to a specific service name that doesn't match the EPP that's deployed when using vllm-sim. When I modified the destination rule to use the correct service name, it worked flawlessly:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: epp-insecure-tls
  namespace: default
spec:
  host: vllm-llama3-8b-instruct-epp
  trafficPolicy:
    tls:
      insecureSkipVerify: true
      mode: SIMPLE

[danehans]

@keithmattix thanks for looking into this bug. Would you like to submit a PR to update the example DestRule?

[keithmattix]

#958 🙂

[keithmattix]

Fixed by #958 this can probably be closed