Inconsistent pod ID returned by `px.ip_to_pod_id` function

[isala404]

Describe the bug
The px.ip_to_pod_id function in Pixie occasionally returns an incorrect pod ID for a given remote address. The issue occurs when Pixie incorrectly matches the IP address to a random cron job running within the cluster, instead of a long-running pod that should be matched. The problem happens randomly and is not specific to a certain pod type. The pods affected by this issue have a typical lifecycle of a few hours. We suspect that this issue may be caused by a bug in the metadata server.

To Reproduce
Steps to reproduce the behavior:

1. Deploy few cronjobs that does an HTTP call to an random endpoint which runs every minute.
2. Deploy few pods which talks to each other in HTTP.
3. Get the IP address of those pods from kubectl.
4. Check the output of px.pod_id_to_pod_name(px.ip_to_pod_id(POD_IP_ADDRESS)) for each pod after a while.

Note - This is not a occasionally behaviour and hard to reproduce at once.

Expected behavior
px.ip_to_pod_id should return the correct IP address.

App information (please complete the following information):

• Pixie version - 0.12.10
• K8s cluster version - v1.23.12
• Node Kernel version - 5.4.0-1091-azure
• Browser version

Additional context
Please refer this slack thread for additional context - https://pixie-community.slack.com/archives/C0405TUGY2C/p1676378960370879

[isala404]

Exact steps to reproduce

1. Deploy following two kubernetes manifest files.
   cronjobs.yaml
   services.yaml

2. Create a curl pod that query the services and let it run for about an hour.

   kubectl run curl-pod --restart=Never --image=curlimages/curl -- /bin/sh -c 'while true; do curl -o /dev/null -s -w "status-code: %{http_code} duration: %{time_total}\n" http://start-component-3313962498.pixie-test-namespace:9090/requestdatachain | xargs -I {} sh -c "echo time: \$(date +\"%Y-%m-%dT%H:%M\") {}"; sleep 300; done'

3. Open px/http_data script,

   • set source_filter as cron-test
   • set destination_filter as pixie-test-namespace
   • set start_time as -1h

4. You will see records as following even though the pods in cron-test namespace doesn't send any http calls to pixie-test-namespace namespace.

[aimichelle]

Thanks to some additional logs/info that @MrSupiri sent over, we figured out the issue here.
Currently, each PEM keeps a map from IP -> Pod ID, which is used in the ip_to_pod_id UDF. The map represents the IP mappings of the cluster at the current moment.

What happened here was:

1. curl-pod has IP X and makes requests to pixie-test-namespace. PEM correctly collects this as a request from IP X.
2. curl-pod is deleted and reassigned IP Y.
3. cron-test pod is created and assigned IP X. The likelihood of some pod being assigned IP X is higher with cronjobs, because new pods are created so frequently.
4. IP map in PEM is updated: IP X -> cron-test pod.
5. User executes query, and maps cron-test to remote_addr X.

One possible solution here is to make the IP map also track timestamp, ip_to_pod_id would also need a timestamp arg so it can select the correct pod_id for the time the HTTP request was made [if no timestamp is provided, we assume the current time, for backwards compatibility].

[isala404]

Thank you for the detailed explanation and for identifying the root cause of the issue. It's great to see that the team was able to figure out a solution to address the problem.

Please let me know if there is any further action required on my part, or if there are any additional details that would be helpful in resolving this issue.

[vihangm]

I explored various ways in which this issue could be addressed and the ideas are documented in this doc:
https://docs.google.com/document/d/1QIPUleSJhvj7kSu0QwFI1DjUGCfFdbFk_LbuNSS4cVM/edit#

A design review will be conducted to evaluate the various options on the table.