Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gefyra run to set common Kubernetes env variables and ServiceAccount data (K8s cert and token) #319

Open
Schille opened this issue Jan 16, 2023 · 5 comments
Assignees
Labels
enhancement 🎉 New feature or request

Comments

@Schille
Copy link
Collaborator

Schille commented Jan 16, 2023

What is the new feature about?

At the least, those two are required in order to connect a locally running container with the remote K8s API through an internal path:
KUBERNETES_SERVICE_HOST - ip
KUBERNETES_SERVICE_PORT - port

In addition, it would be a nice to have to get/assign a service account for the local container.

Why would such a feature be important to you?

When writing applications that communicate with K8s API server, it would be important to make the address available to the locally running container.

Anything else we need to know?

No response

@Schille Schille added the enhancement 🎉 New feature or request label Jan 16, 2023
@schwobaseggl schwobaseggl self-assigned this Mar 6, 2023
@Schille
Copy link
Collaborator Author

Schille commented Mar 17, 2023

@schwobaseggl Here's my PoC with k3d.

  1. Create cluster
    k3d cluster create mycluster --agents 1 -p 8080:80@agent:0 -p 31820:31820/UDP@agent:0
  2. Run a Pod with Ubuntu in mycluster
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
    - name: ubuntu
      image: ubuntu
      command:
        - sleep
        - infinity
  1. Retrieve Kubernetes certificate and SA token from ubuntu
  • Create an archive within the Pod
cd /var
tar -cz run/secrets/kubernetes.io -f sa.tar.gz
  • Copy it to the current directory on the host
kubectl cp ubuntu:var/sa.tar.gz ./sa.tar.gz
  • install curl
apt update && apt install -y curl
  1. Setup a Gefyra container
  • gefyra up
  • gefyra run --image ubuntu --name ubuntu --rm -c "sleep infinity"
  1. Copy the Kubernetes certificate and SA token to Gefyra container from the current directory on the host
docker cp sa.tar.gz ubuntu:var/sa.tar.gz
  1. Prepare Gefyra container with curl and the copied data
  • docker exec -it ubuntu bash
  • Bash on the local Gefyra container
cd /var
tar -xzvf sa.tar.gz
apt update && apt install -y curl

In both shells (K3d running Pod and local Gefyra container), this should work:

APISERVER=https://kubernetes

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN     
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api

{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "172.20.0.2:6443"
    }
  ]
}

@Schille
Copy link
Collaborator Author

Schille commented Mar 17, 2023

Here's the PoC for a specific ServiceAccount (when given with gefyra run --sa mysa) after setting everything up from above.

  1. Create a service account, ClusterRole, and ClusterRoleBinding
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mysa
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-manager
rules:
- apiGroups: [""] 
  resources: ["pods"]
  verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: manage-pods
subjects: 
- kind: ServiceAccount
  name: mysa
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pod-manager
  1. Retrieve ServiceAccount token
SECRET=$(kubectl get serviceaccount mysa -o json | jq -Mr '.secrets[].name | select(contains("token"))')
TOKEN=$(kubectl get secret ${SECRET} -o json | jq -Mr '.data.token' | base64 -d)
  1. Put the token to the local Gefyra container
docker exec ubuntu bash -c "echo '`echo $TOKEN`' > /var/run/secrets/kubernetes.io/serviceaccount/token"
  1. curl the API server from local Gefyra container
  • docker exec -it ubuntu bash
  • run curl
TOKEN=$(cat ${SERVICEACCOUNT}/token)
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/default/pods/

{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "2498"
  },
  "items": [
    {
      "metadata": {
        "name": "ubuntu",
        "namespace": "default",
        "uid": "4cd343e6-7854-4d63-a21c-c4efe479341b",
        "resourceVersion": "877",
        "creationTimestamp": "2023-03-17T16:26:40Z",
        "annotations": {
[...]
        "startTime": "2023-03-17T16:26:40Z",
        "containerStatuses": [
          {
            "name": "ubuntu",
            "state": {
              "running": {
                "startedAt": "2023-03-17T16:27:09Z"
              }
            },
            "lastState": {
              
            },
            "ready": true,
            "restartCount": 0,
            "image": "docker.io/library/ubuntu:latest",
            "imageID": "docker.io/library/ubuntu@sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc295a0428a6d21",
            "containerID": "containerd://5c7bae9481439bd5e35f08caf7446e47b0f30a963ec06cc5d66f9c445a5b919a",
            "started": true
          }
        ],
        "qosClass": "BestEffort"
      }
    }
  ]
}

@Schille Schille changed the title gefyra run to set common Kubernetes env variables gefyra run to set common Kubernetes env variables and ServiceAccount data (K8s cert and token) Mar 17, 2023
@Schille
Copy link
Collaborator Author

Schille commented Mar 17, 2023

I'd say for getting the env from a Gefyra phantom Pod in the cluster when gefyra running, we should create a busybox Pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-phantom-run-1
spec:
  containers:
    - name: busybox
      image: busybox
      command:
        - sleep
        - infinity

And once the Pod is running:

kubectl exec my-phantom-run-1 -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=my-phantom-run-1
KUBERNETES_SERVICE_HOST=10.43.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
HOME=/root

While copying the environment to the local Gefyra container, we should modify the HOSTNAME value to match the local container's name.

@schwobaseggl
Copy link

schwobaseggl commented Mar 24, 2023

@Schille Reproduction with exact steps above.
ubuntu 20.04LTS amd
gefyra 1.0.4
Docker engine 23.0.1
api version 1.42
Go 1.19.5
containerd 1.6.18
runc 1.1.4
docker-init 0.19.0
k3d 5.4.3
k3s 1.23.6
kubectl client 1.24.1

  1. Create cluster
k3d cluster create mycluster --agents 1 -p 8080:80@agent:0 -p 31820:31820/UDP@agent:0

Cluster spins up w/o issues
2. Run pod with exact config from your comment

$ kubectl apply -f po.yaml
$ kubectl get po
NAME     READY   STATUS    RESTARTS   AGE
ubuntu   1/1     Running   0          53s

All good

.... all steps to 5. with no issues
6. API curl:

Output in k3d pod

{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "172.21.0.2:6443"
    }
  ]
}

Output in gefyra docker container

curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
  1. API curl with dedicated service account
    Same symptoms: in k3d pod, I get the PodList, in gefyra docker container, I get
    the error from above

  2. One notable difference seems to be that the api server sends different certificates to the pod and container respectively:
    In k3d pod:

root@ubuntu:/# echo | openssl s_client -showcerts -servername kubernetes -connect 10.43.0.1:443
CONNECTED(00000003)
depth=0 O = k3s, CN = k3s
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = k3s, CN = k3s
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 O = k3s, CN = k3s
verify return:1
---
Certificate chain
 0 s:O = k3s, CN = k3s
   i:CN = k3s-server-ca@1679651976
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Mar 24 09:59:36 2023 GMT; NotAfter: Mar 23 10:23:11 2024 GMT
-----BEGIN CERTIFICATE-----
MIICRDCCAeugAwIBAgIIfsJFLH1/1bcwCgYIKoZIzj0EAwIwIzEhMB8GA1UEAwwY
azNzLXNlcnZlci1jYUAxNjc5NjUxOTc2MB4XDTIzMDMyNDA5NTkzNloXDTI0MDMy
MzEwMjMxMVowHDEMMAoGA1UEChMDazNzMQwwCgYDVQQDEwNrM3MwWTATBgcqhkjO
PQIBBggqhkjOPQMBBwNCAAQHM/gpMUVyBlWd5lhQ+qPZFAIP6Ee1kMPt+FTP5UIT
GtUc7chqtFey/nnV3OMreDC8u+ys5D5CDXTCUep1XnyGo4IBDjCCAQowDgYDVR0P
AQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMB8GA1UdIwQYMBaAFObja+hv
hjOakT2BwP4n67draHAEMIHBBgNVHREEgbkwgbaCCWdudXBnLm9yZ4IWazNkLW15
Y2x1c3Rlci1zZXJ2ZXItMIIKa3ViZXJuZXRlc4ISa3ViZXJuZXRlcy5kZWZhdWx0
ghZrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjgiRrdWJlcm5ldGVzLmRlZmF1bHQuc3Zj
LmNsdXN0ZXIubG9jYWyCCWxvY2FsaG9zdIcEAAAAAIcECisAAYcEfwAAAYcErBUA
AocQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAgNHADBEAiANBup7pWDiuFwj
tfeeOB5aBWDJ0Qpk7pFCnGQXNRZPawIgc2s/LR/x2n5PkErMsC4/WUBRHkQ0ENIb
JX3YQ3zt4fo=
-----END CERTIFICATE-----
---
Server certificate
subject=O = k3s, CN = k3s
issuer=CN = k3s-server-ca@1679651976
---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1007 bytes and written 406 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 7BADB03C1BA4B4A861429A86A1BA5191CFDD3F8AE91EBB96F2C50F37905F3AE0
    Session-ID-ctx: 
    Resumption PSK: FB38CB1E6181712B29AB55A159283B5E4AFA06F8011C4E2321424A1D5BCC7DD0
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - fc a6 ec 25 45 77 5f 32-62 b8 92 45 7a 7c 6e 36   ...%Ew_2b..Ez|n6
    0010 - ae 1c 1f 27 d9 52 ea ab-be 6a 9f 92 90 59 3b 69   ...'.R...j...Y;i
    0020 - 9a 41 5b 87 38 c4 aa 75-55 fa ea 47 2c 27 64 9b   .A[.8..uU..G,'d.
    0030 - 49 ba e9 03 1a ce 0d 74-fe b4 48 d8 50 2e 33 0d   I......t..H.P.3.
    0040 - 6f 2f 0e cc a8 0d 33 c4-a6 fb c1 ec 58 63 5b 5e   o/....3.....Xc[^
    0050 - 3c f4 12 57 de 61 e9 d6-46 3c 5f f2 fc d7 63 f9   <..W.a..F<_...c.
    0060 - ae 6d 7c de 86 ac 76 69-09 a1 40 2e fc 15 4b 62   .m|...vi..@...Kb
    0070 - 1f                                                .

    Start Time: 1679653722
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
DONE

In gefyra docker container:

root@a37c034df362:/# echo | openssl s_client -showcerts -servername kubernetes -connect 10.43.0.1:443
CONNECTED(00000003)
depth=1 CN = k3s-server-ca@1656681228
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 CN = k3s-server-ca@1656681228
verify return:1
depth=0 O = k3s, CN = k3s
verify return:1
---
Certificate chain
 0 s:O = k3s, CN = k3s
   i:CN = k3s-server-ca@1656681228
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jul  1 13:13:48 2022 GMT; NotAfter: Mar 23 07:10:48 2024 GMT
-----BEGIN CERTIFICATE-----
MIICWDCCAf+gAwIBAgIIZ+uGCbjCVdAwCgYIKoZIzj0EAwIwIzEhMB8GA1UEAwwY
azNzLXNlcnZlci1jYUAxNjU2NjgxMjI4MB4XDTIyMDcwMTEzMTM0OFoXDTI0MDMy
MzA3MTA0OFowHDEMMAoGA1UEChMDazNzMQwwCgYDVQQDEwNrM3MwWTATBgcqhkjO
PQIBBggqhkjOPQMBBwNCAARIwNcU7SL2kS+L5wlp43bsXfJiHzNEXkvUgQfX4D94
o1eCUZ6QqxG/kqbkjKrPPJjARdGeSp9WPdrLs/c+UWpro4IBIjCCAR4wDgYDVR0P
AQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMB8GA1UdIwQYMBaAFGBhC1Id
CwJZCKfa14iF8rtp+IN2MIHVBgNVHREEgc0wgcqCCWdudXBnLm9yZ4IKa3ViZXJu
ZXRlc4ISa3ViZXJuZXRlcy5kZWZhdWx0ghZrdWJlcm5ldGVzLmRlZmF1bHQuc3Zj
giRrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWyCCWxvY2FsaG9z
dIISdmVpdC10aGlua3BhZC10NDgwhwQKKwABhwR/AAABhwTAqLJDhwTAqAJshwTA
qAJuhxAgAwDIHz94wroA5kPnkApshxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49
BAMCA0cAMEQCIDb1UoOkv5pw/u+c7T+2dc6WJgVU1g32QeltwrvY1OSHAiBdjohf
1dO0HUtyxfIIrYxKF9h/7RizAYoL+2Xg8v+UPg==
-----END CERTIFICATE-----
 1 s:CN = k3s-server-ca@1656681228
   i:CN = k3s-server-ca@1656681228
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jul  1 13:13:48 2022 GMT; NotAfter: Jun 28 13:13:48 2032 GMT
-----BEGIN CERTIFICATE-----
MIIBdzCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
dmVyLWNhQDE2NTY2ODEyMjgwHhcNMjIwNzAxMTMxMzQ4WhcNMzIwNjI4MTMxMzQ4
WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE2NTY2ODEyMjgwWTATBgcqhkjO
PQIBBggqhkjOPQMBBwNCAATYNXGbR6z8kGvyybn4w2eSP+VvU3bRJHKdOp2InSiT
dIhwD0h3+js1M6QpQujCiLOQujmEpkXLNqOGnmyz74TYo0IwQDAOBgNVHQ8BAf8E
BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUYGELUh0LAlkIp9rXiIXy
u2n4g3YwCgYIKoZIzj0EAwIDSAAwRQIhAOzNwzyqk2VGvhL7sOpz7eKT2qeWwI/l
6o0copjOS+kqAiB3z2hKB7GwtE/R8BJ7gLzlNAMLjO3Bjju5Xl/6dzr2cg==
-----END CERTIFICATE-----
---
Server certificate
subject=O = k3s, CN = k3s
issuer=CN = k3s-server-ca@1656681228
---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1410 bytes and written 406 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: CA378D595E4D76C414C28123694911AF76E0070CC228DAE3AA60F64F70E05FE9
    Session-ID-ctx: 
    Resumption PSK: 4CA4356C57C0A29F9367CAED2067A61EA16ACD211A827EA6978F04F3AB39FB44
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - a5 31 e1 7c 09 37 6a 8b-92 3c 3c 7c 27 94 8b d6   .1.|.7j..<<|'...
    0010 - 5e 51 96 4d 67 6b b3 c7-36 f1 da c5 83 fe 2c c5   ^Q.Mgk..6.....,.
    0020 - bd c9 a9 93 b7 e6 cb 2c-aa fa 38 19 59 d8 66 57   .......,..8.Y.fW
    0030 - 4f da 79 85 3c 3e 0b fc-7e cb 35 ba 5a a9 92 4a   O.y.<>..~.5.Z..J
    0040 - 54 f2 23 e0 ab e1 f5 a7-42 93 25 d9 41 3f 06 70   T.#.....B.%.A?.p
    0050 - c6 10 7f 53 cc 23 f5 b1-f2 13 f9 91 84 bf 01 8f   ...S.#..........
    0060 - 9f 83 c8 9a 90 fb d0 61-20 bf 6c f7 20 23 7a e2   .......a .l. #z.
    0070 - e8                                                .

    Start Time: 1679653803
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
DONE

@schwobaseggl
Copy link

@Schille
With the new Laptop and a clean environment, it now works for me, too. Shall we keep this shelved or move forward with implementing it along this way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement 🎉 New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants