Repository navigation
Expose k8s topology labels to Solr PODs #556
Description
We support topologySpreadConstraint in #350, so Solr PODs are spread across AZs. But for Solr to be able to place replicas across AZs with the Affinity placement plugin, Solr needs access to node label topology.kubernetes.io/zone as a java sysprop availability_zone.
This is non-trivial, as node labels are not exposed to PODs, and you'd need special permissions to fetch it from ApiServer. Instead of each user going through hoops of setting up Init containers to pull in this info, let's have the operator handle it and inject topology.kubernetes.io/zone as a system property into Solr.
Elastic has done a similar thing, referenced in https://the-asf.slack.com/archives/C01JR8WE1M5/p1669108868939589?thread_ts=1668633426.926589&cid=C01JR8WE1M5
Activity
HoustonPutman commentedon Apr 18, 2023
Hey Jan, did you see this ancient PR #90? Its basically hacking the node labels in as sysprops. But yes, you need node permissions which is very unfortunate. I really wish Kubernetes solved this via the Downwards API years ago.
Given SIP-18 what do you think about a plugin that allows loading "sysProp" like key-values from external sources, like kubernetes? Getting away from hacky initContainers is the best path for us. You can see how we add new kube-friendly features to Solr, but make those features available in earlier versions through the operator doing some hacky stuff. And eventually the minimum Solr version will let us get rid of those hacks.
So I can definitely see a hacky initContainer doing the work for us here, but eventually it would be great to just use the KubePropertiesPlugin (just an example), and not have to have a complex setup process.
Eventually maybe kubernetes/kubernetes#40610 will make it in and this gets even easier for us...
janhoy commentedon Apr 18, 2023
Thanks for the link, did not find it.
While a KubePropertiesPlugin is nice long term for several things, I think it makes sense to start with the more targeted feature of propagating zone info as sys prop.
Re the configuring of placement plugins, would make sense if /clusterprops.json could also be read from / owned by a CM, then it would be simple to provide such config without touching ZK.
janhoy commentedon Sep 12, 2023
Cycling back to this, I had a look at #90 to see if it is something I could incorporate into my running clusters with plain helm templating. I'm sure I could extract each part from the go code into scripts and initcontainer yaml to make it work until it is a feature of the operator. Do you forsesee a variant of #90 every being merged?
HoustonPutman commentedon Sep 12, 2023
It could be, though I absolutely hate that it is necessary. Let's not do it for v0.8.0, because we just need to fix TLS and get it out. But let's target it for the next version. I'll ping you, so that we can work on it together.
janhoy commentedon Sep 12, 2023
I have most of the moving parts incorporated in a private helm chart, except some access permission trouble for the sa token, almost there. Please ping me after 0.8 release and I'm happy to discuss and review/test the feature. If we at the same time could make it much simpler to configure the affinity (or other) placement plugin it would be a nice combo.
janhoy commentedon Sep 13, 2023
So, for information, here are the fragments I ended up with in my setup to do this manually: https://gist.github.com/janhoy/dfc24bb128bd5a44a114a10d266b9e56
I've just cut out the relevant parts from a
helm templatecommand, not sure if this will apply, but people could copy/paste into their chart or yaml. This is particularly done to get hold oftopology.kubernetes.io/zone, don't care about other labels.lopf commentedon Feb 11, 2026
Just a heads-up: Starting with Kubernetes 1.35, node topology labels are exposed to Pods via the Downward API (currently in beta). This might be relevant here.
See https://kubernetes.io/blog/2025/12/17/kubernetes-v1-35-release/#expose-node-topology-labels-via-downward-api
janhoy commentedon Feb 11, 2026
Sounds as good news. I did not get from that text whether it will happen automatically from 1.35 or if we need to fetch them ourselves with downward api.
@HoustonPutman, is this perhaps something we want to automatically add to our pods, conditionally on k8s version >=1.35? And then we can document that users of newer k8s versions will get the ENV / sysprop automatically and can use it in placement plugin, but users of older k8s versions may need to add their own init container and query K8s API.