Add Ingress resource to helm chart

[dianaarnos]

While deploying Pinot to Kubernetes we noticed the provided helm chart does not provide an Ingress and this is something we need to run it inside our current environment.

After a quick discussion on Pinot's Slack, we decided to add this option to the Helm Chart, as long as an opt-in (or opt-out) mechanism is provided.

Related PR: #7997

[hpvd]

Since the solution seems to be merged, can we close this issue?