K3s support

[sr229]

Description

K3s is a Kubernetes distrbution intended for IoT, edge and those who wants a minimal Kubernetes experience (ala-HyperKube without the pain).

However, K3s removes the following:

• in tree vendor plugins.
• non v1 apiversion resources.

With this in mind, we need to have a seperate manifest for deploying in K3s as there are many resources we use that are not in the v1 API stability.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[amisevsk]

FWIW I don't believe we do anything outside the v1 API anymore, since deployments moved out of v1beta2

[sr229]

@amisevsk I believe we use Ingresses which isn't a v1 API. This'll beak K3s

[amisevsk]

@sr229 You are right, I missed that one.

[sr229]

I think at this rate we can have the same chart but we have to revise the Ingress resources to use a LoadBalancer Resource instead - I'm not exactly sure how that's parity on what we have

[petzsch]

It is indeed possible to setup Che on k3s.
You just need to make sure that you don't deploy traefik when setting up the manager/server:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --no-deploy traefik" sh

Otherwise I pretty much followed the Azure instructions here:
http://www.eclipse.org/che/docs/che-7/installing-eclipse-che-on-microsoft-azure/

So I deployed the ingress-nginx as described there.

One problem was the cert-manager for updating the domain. I had the domain on AzureDNS, but I'm pretty sure the free option (CloudFlare DNS) would have worked just as well. When using a more recent version of cert-manager (0.14.2 using this url with kubectl: https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml) some of the yml changed. Here are the examples I used:

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: che-certificate-issuer
spec:
  acme:
    email: markus@example.com
    privateKeySecretRef:
      name: letsencrypt
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        azuredns:
          # Service principal clientId (It's the value printed in the previous lines: echo ID=$AZURE_CERT_MANAGER_SP_APP_ID )
          clientID: <client-id>
          # A secretKeyRef to a service principal ClientSecret (password)
          # ref: https://docs.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-service-principal
          clientSecretSecretRef:
            name: azuredns-config
            key: CLIENT_SECRET
          # Azure subscription Id that can be obtained with command:
          # $ az account show  | jq -r '.id'
          subscriptionID: <subscription-id>
          # Azure AD tenant Id that can be obtained with command:
          # $ az account show  | jq -r '.tenantId'
          tenantID: <tenant-id>
          resourceGroupName: <ressource-group-you-put-your-ide-domain-into>
          # The DNS Zone to use
          hostedZoneName: ide.example.com
EOF

and for the cert:

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
 name: che-tls
 namespace: che
spec:
 secretName: che-tls
 issuerRef:
   name: che-certificate-issuer
   kind: ClusterIssuer
 dnsNames:
   - '*.ide.example.com'
EOF

Also for chectl to be able to deploy, I had to seperate the ca.crt from the tls.crt which seems to be a known bug:

kubectl patch secret \
  -n che che-tls \
  -p="{\"data\":{\"ca.crt\": \"$(kubectl get secret \
  -n che che-tls \
  -o json -o=jsonpath="{.data.tls\.crt}" \
  | base64 -d | awk 'f;/-----END CERTIFICATE-----/{f=1}' - | base64 -w 0)\"}}"

I will write together a more complete howto next weekend. I hope for now this helps anyone searching Che and k3s and finding this.

[amisevsk]

Thanks for trying this out @petzsch, it's great you were able to get up and running! If you have time, it would be great to contribute something to the Che documentation.

I haven't played with k3s so I'm kind of useless here.

[sr229]

I can review the document when it gets PRed. We would need a K3s guide.