Description
This is not a bug really; rather a documentation of my experience trying to write a simple generator function. I have always found friction reports super helpful for me, but feel free to close this immediately.
Goal: write a generator that takes in a config (declaratively) with two values: revision and id, and fetch the grafana dashboard (ie https://grafana.com/api/dashboards/9614/revisions/1/download) into a ConfigMap local yaml file.
Timeline: about 3 hours
I first started digging into https://kpt.dev/book/05-developing-functions/02-developing-in-Go and the github.com/GoogleContainerTools/kpt-functions-catalog for some real examples. Right off the bat I ran into issues; the framework.Command method no longer exists in new kyaml versions, and kpt-functions-catalog uses a newer version (v0.10.21) than the docs. I don't know the context, but no deprecation period and a breaking change in a patch release seems not very user friendly.
Once I got a hello world compiling, I quickly found it challenging to debug at all to get a feel of the inputs/outputs I was expecting. Documented more in #2434.
Next I started looking for some examples of generators; however, almost everything in kpt-functions-catalog is just doing transformations on existing objects or validation. UpsertResource was somewhat similar but a bit complex to follow, so I was somewhat on my own. Luckily the task was trivial - I just needed to create a single configmap.
I started out just trying to do that, and not worry about getting the actual data into the configmap yet. I see I need to create an RNode and insert it into Items... should be simple enough. RNode has very somewhat minimal documentation on how to actually create one, so I dug through the ~100 methods it has and found a few promising ones: ConvertJSONToYamlNode and FromMap. I also found I could just yaml.Unmarshal into one (I think?).
I first attempt to create a v1.ConfigMap object, marshal to yaml string, then unmarshal into RNode. However, yaml.Marshal is actually outputting a completely incorrect format for these (missing type data, keys are all lowercase like resourceversion, etc), so this didn't work. I then attempted to use the stdlib json.Marshal, but ran into the fact that the type metadata is not auto populated then. I started trying to find some examples of actually doing this with schemes, decoders, etc but gave up after a few minutes, figuring this must not be the recommended approach to do this or it would be easier+documented.
From the looks of things, all examples are using yaml.Parse with Sprintf, or go templating. I would prefer strongly typed v1.ConfigMap, but was willing to compromise with go template, especially since TemplateProcessor seemed to handle a bunch of things.
Because my code wasn't strictly as straightforward as plumbing some data from input to output, I had created my own Processor that then called TemplateProcessor{}.Process(resourceList). This turned out to not be right; TemplateProcessor{}.Process only works when used directly, as TemplateData will get set to the resourceList.FunctionConfig. Makes sense after running through a debugger, but got me stuck for 10 minutes or so. Replacing Process with Filter got things rolling again, and I finally got configmap written to a file.
However, I didn't like the filename. There was no obvious method on RNode to set the filename, but I stumbled upon a config.kubernetes.io/path at some point in my debugging so tried it out and it worked; it appears nowhere in the kpt.dev documentation though.
Next, I just needed to get the grafana dashboard downloaded and into the configmap. The download part was simple enough, but then I ran into issues with my previous choice to use go templating -- because the data is json (ie multiline, has quotes, etc), we have escaping issues just sticking it into the configmap like
data:
"{{.Name}}.json": {{.Dashboard}}
Because TemplateProcessor doesn't add in sprig functions, the usual fun tricks of helm (which I am using kpt to avoid using 🙂 ) like |ident 4 won't work either. There didn't seem to be a way to add functions to TemplateProcessor either.
After trying a few variations of the template, I still couldn't get the escaping to work.
My end code:
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
"sigs.k8s.io/kustomize/kyaml/fn/framework"
"sigs.k8s.io/kustomize/kyaml/fn/framework/command"
"sigs.k8s.io/kustomize/kyaml/fn/framework/parser"
)
type Dashboard struct {
Revision string `json:"revision"`
ID string `json:"id"`
}
func main() {
asp := GrafanaProcessor{}
cmd := command.Build(&asp, command.StandaloneEnabled, false)
if err := cmd.Execute(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
type GrafanaProcessor struct{}
func (urp *GrafanaProcessor) Process(rl *framework.ResourceList) error {
c := rl.FunctionConfig
data := c.GetDataMap()
db := Dashboard{
Revision: data["revision"],
ID: data["id"],
}
js, err := fetchDashboard(db)
if err != nil {
return fmt.Errorf("fetch dashboard: %v", err)
}
added, err := framework.TemplateProcessor{
TemplateData: map[string]string{
"Name": c.GetName() + "-dashboard",
"Namespace": c.GetNamespace(),
"Label": "grafana_dashboard",
"Dashboard": fmt.Sprintf("%q", js),
},
MergeResources: true,
ResourceTemplates: []framework.ResourceTemplate{{
Templates: parser.TemplateStrings(`
apiVersion: v1
kind: ConfigMap
metadata:
name: "{{.Name}}"
namespace: "{{.Namespace}}"
annotations:
config.kubernetes.io/path: "{{.Name}}.yaml"
labels:
"{{ .Label }}": "true"
data:
"{{.Name}}.json": {{.Dashboard}}
`)}},
}.Filter(rl.Items)
for _, a := range added {
s, _ := a.String()
fmt.Fprintln(os.Stderr, s)
}
rl.Items = added
return err
}
func fetchDashboard(db Dashboard) (string, error) {
url := fmt.Sprintf("https://grafana.com/api/dashboards/%s/revisions/%s/download", db.ID, db.Revision)
resp, err := http.Get(url)
if err != nil {
return "", err
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "", err
}
return string(b), nil
}Next I tried to backtrack, and instead use the typed ConfigMap. I realized I can use sigs.k8s.io/yaml instead of sigs.k8s.io/kustomize/kyaml/yaml and get proper marshaling; that worked. Now just needed to unmarshal it into a node
Ran into an interesting quirk of the library
This panics with obscure error:
var node *yaml.Node
if err := yaml.Unmarshal(b, node); err != nil {
return err
}
This works:
var node yaml.Node
if err := yaml.Unmarshal(b, &node); err != nil {
return err
}
Once that got moving, I found I was actually wrong - sigs.k8s.io/yaml does NOT automatically work, and I got an error that my output is not a KRM format. Oops. Since my function is so simple, I guess I will bite the bullet and just manually specify the type meta...:
TypeMeta: v1.TypeMeta{
Kind: "ConfigMap",
APIVersion: "v1",
},Next all I needed to do was get the merging TemplateProcessor was previously giving me for free. A quick look through the code led me to the MergeFilter. This might cause issues since I really want a replace, not a merge, but should be good enough for now:
rl.Items = append(rl.Items, yaml.NewRNode(&node))
rl.Items, err = filters.MergeFilter{}.Filter(rl.Items)
And finally things are working!
End code:
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
corev1 "k8s.io/api/core/v1"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/kustomize/kyaml/fn/framework"
"sigs.k8s.io/kustomize/kyaml/fn/framework/command"
"sigs.k8s.io/kustomize/kyaml/kio/filters"
"sigs.k8s.io/kustomize/kyaml/yaml"
kyaml "sigs.k8s.io/yaml"
)
type Dashboard struct {
Revision string `json:"revision"`
ID string `json:"id"`
}
func main() {
asp := GrafanaProcessor{}
cmd := command.Build(&asp, command.StandaloneEnabled, false)
if err := cmd.Execute(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
type GrafanaProcessor struct{}
func (urp *GrafanaProcessor) Process(rl *framework.ResourceList) error {
c := rl.FunctionConfig
data := c.GetDataMap()
db := Dashboard{
Revision: data["revision"],
ID: data["id"],
}
js, err := fetchDashboard(db)
if err != nil {
return fmt.Errorf("fetch dashboard: %v", err)
}
cm := &corev1.ConfigMap{
TypeMeta: v1.TypeMeta{
Kind: "ConfigMap",
APIVersion: "v1",
},
ObjectMeta: v1.ObjectMeta{
Name: c.GetName() + "-dashboard",
Namespace: c.GetNamespace(),
Annotations: map[string]string{
"config.kubernetes.io/path": c.GetName() + "-dashboard.yaml",
},
Labels: map[string]string{
"grafana_dashboard": "true",
},
},
Data: map[string]string{c.GetName() + ".json": js},
}
b, err := kyaml.Marshal(cm)
if err != nil {
return err
}
var node yaml.Node
if err := yaml.Unmarshal(b, &node); err != nil {
return err
}
rl.Items = append(rl.Items, yaml.NewRNode(&node))
rl.Items, err = filters.MergeFilter{}.Filter(rl.Items)
if err != nil {
return err
}
return nil
}
func fetchDashboard(db Dashboard) (string, error) {
url := fmt.Sprintf("https://grafana.com/api/dashboards/%s/revisions/%s/download", db.ID, db.Revision)
resp, err := http.Get(url)
if err != nil {
return "", err
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "", err
}
return string(b), nil
}Overall, there was no one thing that was broken/bad/hard - but rather at most steps I encountered small barriers that overtime led to a more frustrating experience than I expected. I hope over time as the ecosystem matures a bit this will get streamlined a bit - I think a large function catalog would be great for the project