🐛 Make root:compute export the rest of namespaced kube API

[MikeSpreitzer]

Summary

This PR does the following.

1. Makes kubestellar init extend the root:compute workspace with additional APIExports that cover the namespaced resources not already exported by kcp-dev/kcp and additional RBAC contents to make those additional APIExports usable.
2. Makes kubectl kubestellar ensure wmw --with-kube install corresponding APIBinding objects.

This PR also updates example1 to exercise the new functionality, using a ReplicaSet object instead of a Deployment object in the common workload. This PR also updates example1 to use kubestellar init because that now does a lot more than just create one workspace and do one kubectl apply.

Not exporting the cluster-scoped part yet, not sure about ability to support it in the future.

The CRDs and APIExports were produced as follows.

The resources were pulled from a kind cluster running Kubernetes release 1.25.2.

These were produced by the following bashery.

The following function converts a kubectl api-resources listing into a listing of arguments to the kcp crd-puller.

function rejigger() {
    if [[ $# -eq 4 ]]
    then gv="$2"
    else gv="$3"
    fi

    case "$gv" in
	(*/*) group=.$(echo "$gv" | cut -f1 -d/) ;;
	(*)   group=""
    esac

    echo "${1}$group"
}

With kubectl configured to manipulate a kcp workspace, the following command captures the listing of resources built into that kcp workspace.

kubectl api-resources | grep -v APIVERSION | while read line; do rejigger $line; done > /tmp/kcp-rgs.txt

With kubectl configured to manipulate a kind cluster, the following commands capture the resource listing split into namespaced and cluster-scoped.

kubectl api-resources | grep -v APIVERSION | grep -w true | while read line; do rejigger $line; done > /tmp/kind-ns-rgs.txt
kubectl api-resources | grep -v APIVERSION | grep -w false | while read line; do rejigger $line; done > /tmp/kind-cs-rgs.txt

With CWD=config/kube/exports/namespaced,

crd-puller --kubeconfig $KUBECONFIG $(grep -v -f /tmp/kcp-rgs.txt /tmp/kind-ns-rgs.txt)

With CWD=config/kube/exports/cluster-scoped,

crd-puller --kubeconfig $KUBECONFIG $(grep -v -f /tmp/kcp-rgs.txt /tmp/kind-cs-rgs.txt)

I manually deleted the four CRDs from https://github.com/kcp-dev/kcp/tree/v0.11.0/config/rootcompute/kube-1.24 .

Sadly, kubernetes/kubernetes#118698 is a thing.
So I manually hacked the CRD for jobs.

Sadly, the filenames produced by the crd-puller are not loved by apigen. The following function renames one file as needed.

function fixname() {
    rg=${1%%.yaml}
    case $rg in
	(*.*)
	    g=$(echo $rg | cut -d. -f2-)
	    r=$(echo $rg | cut -d. -f1);;
	(*)
	    g=core.k8s.io
	    r=$rg;;
    esac
    mv ${rg}.yaml ${g}_${r}.yaml
}

In each of those CRD directories,

for fn in *.yaml; do fixname $fn; done

Penultimately, with CWD=config/kube,

../../hack/tools/apigen --input-dir crds/namespaced --output-dir exports/namespaced
../../hack/tools/apigen --input-dir crds/cluster-scoped --output-dir exports/cluster-scoped

Finally, kubernetes/enhancements#1111 applies
to APIExport/APIBinding as well as to CRDs. And the CRD puller does
not know anything about this (not that it would help?). I manually
hacked the namespaced APIResource files that needed it to have an
api-approved.kubernetes.io annotation. It turns out that the
checking in the apiserver only requires that the annotation's value
parse as a URL (any URL will do).

Related issue(s)

This addresses part of #522 --- the part for namespaced resources, but not cluster-scoped ones.

/cc @waltforme
/cc @ezrasilvera

[ezrasilvera]

@MikeSpreitzer Is the production of the CRDs a one time thing that is done suing the manual instructions you listed in the PR description? Don't we need to automate this and add that script as part of the PR as well ?
In addition, looking forward, it means we need to pull in the crd-puller code into KubeStellar if we want to be KCP independent.

[MikeSpreitzer]

I initially planned to script all the stuff done to create the APIExports. First, on the principle that source code control should capture all the inputs; the vision is that there would be a script and/or Makefile target that gets the job done and some documentation would describe when, why, and how to use that. Second, note that this extraction was done for one particular release of Kubernetes. We probably need to get to the point of supporting an evolving set of Kubernetes releases and/or telling users how to support their chosen Kubernetes release.

If you read the description here of what was done, you will see that it would be non-trivial to script. I ran into some problems that I solved with manual editing. This could be scripted, and I even think should be scripted. But this PR does not include that.

We should consult with the kcp project to see if they plan to maintain the crd-puller and/or contribute it as part of something else. If they do then we can simply import and use that particular thing. It does not depend on a fork of Kubernetes.

[MikeSpreitzer]

This PR has a deficiency that should be addressed in follow-up. It is a consequence of a corner case in kcp's APIExport/APIBinding. Before the first object in it is created, an APIExport view behaves oddly: attempts to list and watch it are rebuffed with authentication errors (the correct behavior would be an empty list or watch response), even though kubectl api-resources says that those resources exist. In KubeStellar there is no special response or avoidance of this, so the log is continually spammed with logging these error responses.

[clubanderson]

please rebase this PR to get the latest workflow checks

[clubanderson]

/ok-to-test

[clubanderson]

/retest

[MikeSpreitzer]

The force-push to b75c1e3 adds a newline to the commit comment, trying to prod CI to run the docs executions.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[waltforme]

/lgtm

[kcp-ci-bot]

LGTM label has been added.

Git tree hash: 90b4e8d6670c14f822cfdd8578e2b6e4f302581f

[MikeSpreitzer]

/approve

[kcp-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MikeSpreitzer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [MikeSpreitzer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment