friction report: developing a simple generator function

[howardjohn]

This is not a bug really; rather a documentation of my experience trying to write a simple generator function. I have always found friction reports super helpful for me, but feel free to close this immediately.

Goal: write a generator that takes in a config (declaratively) with two values: revision and id, and fetch the grafana dashboard (ie https://grafana.com/api/dashboards/9614/revisions/1/download) into a ConfigMap local yaml file.

Timeline: about 3 hours

I first started digging into https://kpt.dev/book/05-developing-functions/02-developing-in-Go and the github.com/GoogleContainerTools/kpt-functions-catalog for some real examples. Right off the bat I ran into issues; the framework.Command method no longer exists in new kyaml versions, and kpt-functions-catalog uses a newer version (v0.10.21) than the docs. I don't know the context, but no deprecation period and a breaking change in a patch release seems not very user friendly.

Once I got a hello world compiling, I quickly found it challenging to debug at all to get a feel of the inputs/outputs I was expecting. Documented more in #2434.

Next I started looking for some examples of generators; however, almost everything in kpt-functions-catalog is just doing transformations on existing objects or validation. UpsertResource was somewhat similar but a bit complex to follow, so I was somewhat on my own. Luckily the task was trivial - I just needed to create a single configmap.

I started out just trying to do that, and not worry about getting the actual data into the configmap yet. I see I need to create an RNode and insert it into Items... should be simple enough. RNode has very somewhat minimal documentation on how to actually create one, so I dug through the ~100 methods it has and found a few promising ones: ConvertJSONToYamlNode and FromMap. I also found I could just yaml.Unmarshal into one (I think?).

I first attempt to create a v1.ConfigMap object, marshal to yaml string, then unmarshal into RNode. However, yaml.Marshal is actually outputting a completely incorrect format for these (missing type data, keys are all lowercase like resourceversion, etc), so this didn't work. I then attempted to use the stdlib json.Marshal, but ran into the fact that the type metadata is not auto populated then. I started trying to find some examples of actually doing this with schemes, decoders, etc but gave up after a few minutes, figuring this must not be the recommended approach to do this or it would be easier+documented.

From the looks of things, all examples are using yaml.Parse with Sprintf, or go templating. I would prefer strongly typed v1.ConfigMap, but was willing to compromise with go template, especially since TemplateProcessor seemed to handle a bunch of things.

Because my code wasn't strictly as straightforward as plumbing some data from input to output, I had created my own Processor that then called TemplateProcessor{}.Process(resourceList). This turned out to not be right; TemplateProcessor{}.Process only works when used directly, as TemplateData will get set to the resourceList.FunctionConfig. Makes sense after running through a debugger, but got me stuck for 10 minutes or so. Replacing Process with Filter got things rolling again, and I finally got configmap written to a file.

However, I didn't like the filename. There was no obvious method on RNode to set the filename, but I stumbled upon a config.kubernetes.io/path at some point in my debugging so tried it out and it worked; it appears nowhere in the kpt.dev documentation though.

Next, I just needed to get the grafana dashboard downloaded and into the configmap. The download part was simple enough, but then I ran into issues with my previous choice to use go templating -- because the data is json (ie multiline, has quotes, etc), we have escaping issues just sticking it into the configmap like

data:
  "{{.Name}}.json": {{.Dashboard}}

Because TemplateProcessor doesn't add in sprig functions, the usual fun tricks of helm (which I am using kpt to avoid using 🙂 ) like |ident 4 won't work either. There didn't seem to be a way to add functions to TemplateProcessor either.

After trying a few variations of the template, I still couldn't get the escaping to work.

My end code:

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"os"

	"sigs.k8s.io/kustomize/kyaml/fn/framework"
	"sigs.k8s.io/kustomize/kyaml/fn/framework/command"
	"sigs.k8s.io/kustomize/kyaml/fn/framework/parser"
)

type Dashboard struct {
	Revision string `json:"revision"`
	ID       string `json:"id"`
}

func main() {
	asp := GrafanaProcessor{}
	cmd := command.Build(&asp, command.StandaloneEnabled, false)
	if err := cmd.Execute(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

type GrafanaProcessor struct{}

func (urp *GrafanaProcessor) Process(rl *framework.ResourceList) error {
	c := rl.FunctionConfig
	data := c.GetDataMap()
	db := Dashboard{
		Revision: data["revision"],
		ID:       data["id"],
	}
	js, err := fetchDashboard(db)
	if err != nil {
		return fmt.Errorf("fetch dashboard: %v", err)
	}
	added, err := framework.TemplateProcessor{
		TemplateData: map[string]string{
			"Name":      c.GetName() + "-dashboard",
			"Namespace": c.GetNamespace(),
			"Label":     "grafana_dashboard",
			"Dashboard": fmt.Sprintf("%q", js),
		},
		MergeResources: true,
		ResourceTemplates: []framework.ResourceTemplate{{
			Templates: parser.TemplateStrings(`
apiVersion: v1
kind: ConfigMap
metadata:
  name: "{{.Name}}"
  namespace: "{{.Namespace}}"
  annotations:
    config.kubernetes.io/path: "{{.Name}}.yaml"
  labels:
    "{{ .Label }}": "true"
data:
  "{{.Name}}.json": {{.Dashboard}}
`)}},
	}.Filter(rl.Items)
	for _, a := range added {
		s, _ := a.String()
		fmt.Fprintln(os.Stderr, s)
	}
	rl.Items = added
	return err
}

func fetchDashboard(db Dashboard) (string, error) {
	url := fmt.Sprintf("https://grafana.com/api/dashboards/%s/revisions/%s/download", db.ID, db.Revision)
	resp, err := http.Get(url)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()
	b, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return "", err
	}
	return string(b), nil
}

Next I tried to backtrack, and instead use the typed ConfigMap. I realized I can use sigs.k8s.io/yaml instead of sigs.k8s.io/kustomize/kyaml/yaml and get proper marshaling; that worked. Now just needed to unmarshal it into a node

Ran into an interesting quirk of the library
This panics with obscure error:

	var node *yaml.Node
	if err := yaml.Unmarshal(b, node); err != nil {
		return err
	}

This works:

	var node yaml.Node
	if err := yaml.Unmarshal(b, &node); err != nil {
		return err
	}

Once that got moving, I found I was actually wrong - sigs.k8s.io/yaml does NOT automatically work, and I got an error that my output is not a KRM format. Oops. Since my function is so simple, I guess I will bite the bullet and just manually specify the type meta...:

		TypeMeta: v1.TypeMeta{
			Kind:       "ConfigMap",
			APIVersion: "v1",
		},

Next all I needed to do was get the merging TemplateProcessor was previously giving me for free. A quick look through the code led me to the MergeFilter. This might cause issues since I really want a replace, not a merge, but should be good enough for now:

	rl.Items = append(rl.Items, yaml.NewRNode(&node))
	rl.Items, err = filters.MergeFilter{}.Filter(rl.Items)

And finally things are working!

End code:

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"os"

	corev1 "k8s.io/api/core/v1"
	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"sigs.k8s.io/kustomize/kyaml/fn/framework"
	"sigs.k8s.io/kustomize/kyaml/fn/framework/command"
	"sigs.k8s.io/kustomize/kyaml/kio/filters"
	"sigs.k8s.io/kustomize/kyaml/yaml"
	kyaml "sigs.k8s.io/yaml"
)

type Dashboard struct {
	Revision string `json:"revision"`
	ID       string `json:"id"`
}

func main() {
	asp := GrafanaProcessor{}
	cmd := command.Build(&asp, command.StandaloneEnabled, false)
	if err := cmd.Execute(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

type GrafanaProcessor struct{}

func (urp *GrafanaProcessor) Process(rl *framework.ResourceList) error {
	c := rl.FunctionConfig
	data := c.GetDataMap()
	db := Dashboard{
		Revision: data["revision"],
		ID:       data["id"],
	}
	js, err := fetchDashboard(db)
	if err != nil {
		return fmt.Errorf("fetch dashboard: %v", err)
	}
	cm := &corev1.ConfigMap{
		TypeMeta: v1.TypeMeta{
			Kind:       "ConfigMap",
			APIVersion: "v1",
		},
		ObjectMeta: v1.ObjectMeta{
			Name:      c.GetName() + "-dashboard",
			Namespace: c.GetNamespace(),
			Annotations: map[string]string{
				"config.kubernetes.io/path": c.GetName() + "-dashboard.yaml",
			},
			Labels: map[string]string{
				"grafana_dashboard": "true",
			},
		},
		Data: map[string]string{c.GetName() + ".json": js},
	}
	b, err := kyaml.Marshal(cm)
	if err != nil {
		return err
	}
	var node yaml.Node
	if err := yaml.Unmarshal(b, &node); err != nil {
		return err
	}
	rl.Items = append(rl.Items, yaml.NewRNode(&node))
	rl.Items, err = filters.MergeFilter{}.Filter(rl.Items)
	if err != nil {
		return err
	}
	return nil
}

func fetchDashboard(db Dashboard) (string, error) {
	url := fmt.Sprintf("https://grafana.com/api/dashboards/%s/revisions/%s/download", db.ID, db.Revision)
	resp, err := http.Get(url)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()
	b, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return "", err
	}
	return string(b), nil
}

Overall, there was no one thing that was broken/bad/hard - but rather at most steps I encountered small barriers that overtime led to a more frustrating experience than I expected. I hope over time as the ecosystem matures a bit this will get streamlined a bit - I think a large function catalog would be great for the project

[mengqiy]

Thanks for this detailed friction log!
We will address them.

[mengqiy]

Some of the low-hanging fruits has been addressed in Q3.
We are working on improving the golang SDK in Q4.

[bgrant0607]

See also #2528