HULL and subcharts

[eshepelyuk]

Hello

I am interested if there any possibility to apply HULL templating \ transformations to the values of helm chart dependencies ?

I.e. I've created a chart that uses HULL for rendering Deployments or any other K8s resources. As a dependency this helm chart includes, for example Redis or Postgresql chart.

ATM in my values.yaml I can only use static values to configure subcharts, but I'd like to be able to do smth like below
regardless how hacky or dirty it would look like

hull:
  config:
    specific:
      value2: 222
  ....
redis:
  param1: value1
  param2: _HT*hull.config.specific.value2

Sorry, if this not HULL related at all. Any suggestions are appreciated.

[eshepelyuk]

One thing I've been thinking about - is putthing subchart templates directly into my chart templates folder, i.e. not using subchart as a dependency, but copy pasting subchart templates into root chart.

[gre9ory]

Hi @eshepelyuk, hope you are well.

[eshepelyuk]

Hi @eshepelyuk, hope you are well.

Hello, thanks,
but this is all I see in your comment

[gre9ory]

Yeah sorry posted to soon. More is coming up ...

[gre9ory]

Your idea is very nice and I have thought about the possibility too at some point. My assumption was however that it unfortunately will not work easily due the fact that regular Helm does not allow to modify values.yaml content like HULL does.

Regular Helm looks like this:

values.yaml <--- read values --- Helm Template File with YAML --- write output ---> Output YAML with inserted values

Here, the values.yaml is only a read only construct which is evaluated when rendering out the YAML. The source values can only be modified when evaluating them in the templates to which there is no access in a subchart.

The HULL workflow is like this:

hull.yaml with render function --- read values ---> values.yaml --- apply transformations and write output ---> Output YAML with objects generated from modified values.yaml

The hull.yaml contains a function which reads the values.yaml tree starting at the hull subkey, applies all logic (including transformations) and writes YAML objects based on the modified data.

Add to this the scope limitations that Helm applies when using subcharts. No properties from a parent charts values.yaml are accesible within the context of your sub chart (besides the pretty limited global feature).

For example, if you include redis subchart you have the following scenario:

hull: # <--- redis chart has no way of reading these values 
  config:
    ...
  objects:
    ...
redis: # <- the redis chart configuration scope is limited to the value of this key
  ...

Workarounds

However, as discussed here, the modifications made to the values.yaml tree remain effective after hull.yaml was processed in the context of rendering all templates within the scope of a single chart.
Meaning that, assuming hull.yaml is rendered first in your /templates folder, all other templates access the modified values.yaml.

I did some experiments around that and found potential workarounds.

Note that all of these workarounds require a change to HULL to make it evaluate the complete values.yaml including all keys and not only the hull subkey. Think this is not a big problem technically and could be done if it makes sense for you to proceeed here

Workaround 1 - put all templates into parent chart /templates and render them

As per your second suggestion above, if you were to:

• use the patched HULL release to be created
• take the YAML files from the redis subchart
• copy them to your parent chart /templates folder
• add a folder named zzz and put the hull.yaml into this folder (making sure it is processed first of all templates) instead of putting it at the /templates level
• configure your redis properties directly at the root level in your parent charts values.yaml (since the templates are now part of parent chart)

it would be possible to apply transformations (such as cross references with _HT*) across the complete configuration.

For example, you could do this in your parent values.yaml:

hull:
  config:
    specific:
      value2: 222
  ....

param1: value1
param2: _HT*hull.config.specific.value2

and you will find the resolved value in the rendered redis template.

Downside here mostly that you would need to copy the templates to your parent chart. Other than that the approach should work for at least one subchart, if you would have more than one then you may get into problems with clashing keys in values.yaml

Workaround 2 - inject hull.yaml into subcharts /templates

Potential steps for this approach would be:

• use the patched HULL release to be created
• in your subchart add a folder named zzz and put the hull.yaml into this folder (making sure it is processed first of all templates) instead of putting it at the /templates level
• configure your redis properties at the redis level in your parent charts values.yaml (as per usual)

The trick here would be that injecting the hull.yaml in to the context of the subchart makes it visible in the subcharts context and it is executed. Even if there were no HULL based definitions in your subchart, the values.yaml tree will be processed and transformations resolved in your subcharts context. The parent chart also can read the subchart configuration in values.yaml, allowing you to share values between the two.

The parent values.yaml could look like this for this scenario:

global:
  value2: 222  
hull:
  config:
    specific:
      value2: _HT*global.value2
  ....
redis:
  param1: value1
  param2: _HT*global.value2
postgres:
  port: _HT*global.value2

Downside here is that it requires (small) modification of the subchart(s). If you use the global section, the value defined in the global scope is visible to all subcharts. Values defined in any other place (eg. hull.config.specific) are not accessible in the sub charts as per the Helm limitations.

Let me know if something is unclear and if you want to explore these options. As mentioned, I would need to make some adjustments to HULL to make either workaround possible.

[eshepelyuk]

WOW, the 2nd approach is a pure gold.

All I need to do - is to download and unpack subcharts into templates together with adding that zzz magic.
Easy clean and convenient.

• it makes HULL magic work
• it allows multiple subcharts

[eshepelyuk]

One question about #2

The global section, will it be possible to use HULL transformations there ?
Smth like

global:
  valueHT: _HT!* printf "%-bar" "foo"
  value2: 222  
redis:
  param1:  _HT*global.valueHT
  param2: _HT*global.value2
postgres:
  port: _HT*global.value2

[gre9ory]

Due to the limited access from the subchart context to the parent charts context, only transformations that access values that are within global are resolvable. That can contain indirection, for example, the following worked in a test just now.

I am accessing a global value global.another_number via another indirection under global global.fixed_number:

global:
  service_type: NodePort
  number_111: 111
  number_222: 222
  fixed_number: _HT*global.another_number
  another_number: 333
  boolean_true: true
  test_annotations:
    annotation_1: Global Annotation 1
    annotation_2: Global Annotation 2

kube-state-metrics:
  service:
    type: _HT*global.service_type
    port: _HT*global.fixed_number

and it renders this service.yaml;

---
# Source: hull-test/charts/kube-state-metrics/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: release-name-kube-state-metrics
  namespace: default
  labels:    
    helm.sh/chart: kube-state-metrics-7.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: metrics
    app.kubernetes.io/part-of: kube-state-metrics
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "2.18.0"
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: "NodePort"
  ipFamilies: 
    - IPv6
    - IPv4
  ipFamilyPolicy: PreferDualStack
  ports:
  - name: http
    protocol: TCP
    port: 333
    targetPort: http
  
  selector:    
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/instance: release-name

If you have sth like global.fixed_value: _HT*hull.config.specific.parent_number it will give an error because from the subchart context the parents hull values are out of reach.

[gre9ory]

Additionally, besides global you can then of course access other fields within the same subchart context with transformations:

### CONFIG
kube-state-metrics:
  test_number: 999
  service:
    port: _HT*test_number

yields:

---
# Source: hull-test/charts/kube-state-metrics/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: release-name-kube-state-metrics
  namespace: default
  labels:    
    helm.sh/chart: kube-state-metrics-7.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: metrics
    app.kubernetes.io/part-of: kube-state-metrics
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "2.18.0"
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: "ClusterIP"
  ports:
  - name: http
    protocol: TCP
    port: 999
    targetPort: http
  
  selector:    
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/instance: release-name

Put differently, from within a subchart context you can do transformations that involve the subcharts context and everything that stays within global. And in consequence, if you would try to access kube-state-metrics.test_number from the parent context, this would give you an error because from parent context view the _HT*test_number cannot be resolved, only from within kube-state-metrics context that is possilbe.

[eshepelyuk]

Thanks for explanation, that should cover my needs.

[gre9ory]

Ok, will work on that topic and document the extended usage scenarios that have recently spawned (using additional non-HULL templates and subcharts).

I think a release will come next week the latest.

[eshepelyuk]

Amazing, thank you !