在容器发生OOMKilling时，如何让node-problen-detector向apiServer发送event时添加pod信息，以便获取到具体的pod发生OOM

[lee-lib]

参考阿里容器服务ack文档 https://help.aliyun.com/knowledge_detail/178479.html
文档中描述在2020年07月的镜像版本registry.aliyuncs.com/acs/node-problem-detector:v0.6.3-28-160499f中就可以为oomkilling事件添加pod信息，
我这边是按照此版本的node-problem-detector镜像构建的容器，但是模拟触发oomkill事件时，还是无法获取到pod信息，只能获取到node类型的信息，yaml文件如下
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-problem-detector
namespace: kube-system
labels:
app: node-problem-detector
spec:
selector:
matchLabels:
app: node-problem-detector
template:
metadata:
labels:
app: node-problem-detector
spec:
containers:
- name: node-problem-detector
command:
- /node-problem-detector
- --logtostderr
- --system-log-monitors=/config/kernel-monitor.json,/config/docker-monitor.json
- --apiserver-override=http://192.168.1.228:8080?inClusterConfig=false
image: registry.aliyuncs.com/acs/node-problem-detector:v0.6.3-28-160499f
resources:
limits:
cpu: 10m
memory: 80Mi
requests:
cpu: 10m
memory: 80Mi
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: log
mountPath: /var/log
readOnly: true
- name: kmsg
mountPath: /dev/kmsg
readOnly: true
# Make sure node problem detector is in the same timezone
# with the host.
- name: localtime
mountPath: /etc/localtime
readOnly: true
- name: config
mountPath: /config
readOnly: true
volumes:
- name: log
# Config log to your system log directory
hostPath:
path: /var/log/
- name: kmsg
hostPath:
path: /dev/kmsg
- name: localtime
hostPath:
path: /etc/localtime
- name: config
configMap:
name: node-problem-detector-config
items:
- key: kernel-monitor.json
path: kernel-monitor.json
- key: docker-monitor.json
path: docker-monitor.json

模拟oom产生的日志如下，其中involvedObject.kind信息还是Node，无法获取到Pod信息
I0214 18:22:00.008987 1 mysql.go:73] {
"metadata": {
"name": "k8s-master.16d39fea6f67823e",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/events/k8s-master.16d39fea6f67823e",
"uid": "9c9cdb3b-240a-40e8-9db2-b4490dfc4f42",
"resourceVersion": "8981033",
"creationTimestamp": "2022-02-14T10:21:58Z",
"managedFields": [
{
"manager": "node-problem-detector",
"operation": "Update",
"apiVersion": "v1",
"time": "2022-02-14T10:21:58Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:count": {},
"f:firstTimestamp": {},
"f:involvedObject": {
"f:kind": {},
"f:name": {},
"f:uid": {}
},
"f:lastTimestamp": {},
"f:message": {},
"f:reason": {},
"f:source": {
"f:component": {},
"f:host": {}
},
"f:type": {}
}
}
]
},
"involvedObject": {
"kind": "Node",
"name": "k8s-master",
"uid": "k8s-master"
},
"reason": "OOMKilling",
"message": "Memory cgroup out of memory: Kill process 2235 (stress) score 0 or sacrifice child\nKilled process 2235 (stress) total-vm:515612kB, anon-rss:168728kB, file-rss:32kB, shmem-rss:0kB",
"source": {
"component": "kernel-monitor",
"host": "k8s-master"
},
"firstTimestamp": "2022-02-14T10:21:58Z",
"lastTimestamp": "2022-02-14T10:21:58Z",
"count": 1,
"type": "Warning",
"eventTime": null,
"reportingComponent": "",
"reportingInstance": ""
}

请问，开发者在容器发生后OOMKilling时，如何配置node-problem-detector.yaml和kube-eventer.yaml文件才能获取到Pod信息？

[ringtail]

阿里云的这个功能是在NPD中实现的，目前社区的OOM是在Node维度透出Warning事件。

[crushCoin]

https://github.com/AliyunContainerService/node-problem-detector/blob/master/pkg/systemlogmonitor/log_monitor.go#L67 这里pod uid正则的原因
Cgroup Driver 为 systemd 可以；Cgroup Driver 为 cgroupfs 不行
修改源码重新编译（git clone -b alibabacloud-v0.8.10 https://githubfast.com/AliyunContainerService/node-problem-detector.git）
1、修改为 uuidRegx = regexp.MustCompile("[0-9a-f]{8}[0-9a-f]{4}[0-9a-f]{4}[0-9a-f]{4}[0-9a-f]{12}|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}")
2、https://github.com/AliyunContainerService/node-problem-detector/blob/master/pkg/systemlogmonitor/log_monitor.go#L216
修改为 （可以将 PodOOMKilling 和 OOMKilling 日志放在一起）
message = fmt.Sprintf("pod was OOM killed. node:%s pod:%s namespace:%s uuid:%s\n",
pod.Spec.NodeName, pod.Name, pod.Namespace, uuid) + message
3、修改 kernel-monitor.json （"bufferSize": 30，这个是用于日志匹配的 环形队列缓冲buffer 长度，可以适当改大）
rules 中 增加
{
"type": "temporary",
"reason": "PodOOMKilling",
"pattern": "Task in /kubepods(.+) killed as a result of limit of .(\n.)+ Kill process \d+ (.+) score \d+ or sacrifice child\nKilled process \d+ (.+) total-vm:\d+kB, anon-rss:\d+kB, file-rss:\d+kB.*"
}

上面有转义 看图