The soft-limit controller allows you to set soft resource limits via pod annotations and will kill the pods gracefully when they exceed those limits.
Create the controller with kubectl apply -f https://raw.githubusercontent.com/sethpollack/soft-limits/master/example.yaml
Add the following annotations sethpollack.net/soft-limit-cpu and sethpollack.net/soft-limit-memory to your pods to set the soft limits.
You can use any valid resource value or a percentage of the hard limit. For example sethpollack.net/soft-limit-memory: 200Mi or sethpollack.net/soft-limit-memory: 95%.