Spire deployment on kubernetes and certificate retrieval

[ConstantRohmer]

I did not found any other forum about spire so sorry if this is not the right place for this...

I am currently trying to deploy spire on my k8s cluster (3 worker nodes). My objective is to successfully deploy spire and use the certificates generated by the spire-server to encrypt the exchanges between 2 pods : a webclient and a webserver.

I followed the official tutorial https://spiffe.io/docs/latest/try/getting-started-k8s/
but have encountered some issues without being able to find the answers in the docs :

1. Although the spire-server is running fine, I am struggling to understand if the spire-agents deployed on each of the k8s nodes are working. The pods are Running 1/1, but I don't know how to get the SVIDs and therefore verify if everything works well (e.g if the spire-agents are registered by the server)

2. I deployed my 2 test pods (webclient and webserver) and added registration entries on the spire-server to select the tags client and webserver of my 2 pods and to register them on spire with the following entries :

kubectl exec -n spire spire-server-0 -- \
   /opt/spire/bin/spire-server entry create \
   -spiffeID spiffe://test.example.com/ns/spire/sa/default/app/spire-agent \
   -parentID spiffe://test.example.com/ns/spire \
   -selector k8s:ns:spire \
   -selector k8s:sa:default \
   -selector k8s:pod-label:app:spire-agent


kubectl exec -n spire spire-server-0 -- \
   /opt/spire/bin/spire-server entry create \
   -spiffeID spiffe://test.example.com/ns/spire/sa/default/app/client \
   -parentID spiffe://test.example.com/ns/spire/sa/default/app/spire-agent \
   -selector k8s:ns:spire \
   -selector k8s:sa:default \
   -selector k8s:pod-label:app:client


kubectl exec -n spire spire-server-0 -- \
   /opt/spire/bin/spire-server entry create \
   -spiffeID spiffe://test.example.com/ns/spire/sa/default/app/webserver \
   -parentID spiffe://test.example.com/ns/spire/sa/default/app/spire-agent \
   -selector k8s:ns:spire \
   -selector k8s:sa:default \
   -selector k8s:pod-label:app:webserver

From the tutorial, I know something like below should help me verify that these pods have been issued a SVID and certificate :

/spire-1.8.7/bin/spire-agent api fetch -socketPath /run/spire/sockets/agent.sock

However, the only way I managed to execute this command was by installing spire binaries on the client and webserver directly which seems odd because I already have agents running. In fact, I can't open a shell in the spire-agents because spire changed their images to be minimal and it doesn't include /bin/sh or anything (from what I've seen).

Do you know a way to check for SVID issuance without installing spire on the workloads or is it the right way ?

Following this idea, I tried to build my own spire-agent pods by adding the following commands in the yml to install spire on an alpine pod and therefore be able to connect to the spire-agent pods and debug from there :

kubectl -n spire exec -it spire-agent pod -- /bin/sh
wget https://github.com/spiffe/spire/releases/download/v1.8.7/spire-1.8.7-linux-amd64-musl.tar.gz
tar zvxf spire-1.8.7-linux-amd64-musl.tar.gz
cp -r spire-1.8.7/. /opt/spire/
ln -s /opt/spire/bin/spire-server /usr/bin/spire-server
ln -s /opt/spire/bin/spire-agent /usr/bin/spire-agent

3. Although I put the same path for the socket agent.sock and mounted it to the pods, it is sometimes not found by the pods, how should I work with this socket ?

4. Once everything will work as expected (I hope so at least ;) ), how should I retrieve the certificates issued by my workloads ? Are they even available from these pods so I could be able to create an https session for example ?

Thanks for anything you can add or correct in what I've done, any help is really appreciated !

[MarcosDY]

Although the spire-server is running fine, I am struggling to understand if the spire-agents deployed on each of the k8s nodes are working. The pods are Running 1/1, but I don't know how to get the SVIDs and therefore verify if everything works well (e.g if the spire-agents are registered by the server)

you can list attested agent using spire-server agent list:

$ ./bin/spire-server agent list
found 1 attested agent:

SPIFFE ID         : spiffe://example.org/spire/agent/k8s_psat/cluster1/33b2323d-ebe8-4c2a-9ac8-e1ef6bee3f4f
Attestation type  : k8s_psat
Expiration time   : 2024-03-21 16:11:34 +0000 UTC
Serial number     : 109596583725639493826327346552467884462
Can re-attest     : true

To see if SVIDs are propagated you will need to rely on logs:

time="2024-03-21T15:11:44Z" level=debug msg="Entry created" entry=2a5eaa9a-c956-4007-92aa-fd3faa437ab8 selectors_added=1 spiffe_id="spiffe://example.org/workload" subsystem_name=cache_manager
time="2024-03-21T15:11:44Z" level=info msg="Creating X509-SVID" entry_id=2a5eaa9a-c956-4007-92aa-fd3faa437ab8 spiffe_id="spiffe://example.org/workload" subsystem_name=manager

or metrics

another option is to use debug endpoint

but you can rely on metrics or logs to see SVID propagation.

However, the only way I managed to execute this command was by installing spire binaries on the client and webserver directly which seems odd because I already have agents running. In fact, I can't open a shell in the spire-agents because spire changed their images to be minimal and it doesn't include /bin/sh or anything (from what I've seen).

there is an know issue that prevents to use k8s attestor for inside agent it self, you may rely on using unix attestor if you want to attest agent as a workload
something like:

kubectl exec -n spire spire-server-0 -- \
   /opt/spire/bin/spire-server entry create \
   -spiffeID spiffe://test.example.com/workload \
   -parentID spiffe://test.example.com/ns/spire/sa/default/app/spire-agent \
   -selector unix:user:roor

As you mentioned we are not including sh or another commands, because we are using scratch images that contains only our binaries, you will need to use kubectl exec to communicate only with spire binaries (spire-server or spire-agent)

Do you know a way to check for SVID issuance without installing spire on the workloads or is it the right way ?
as I mentioned before you can verify logs.

But is your workload able to fetch SVIDs using go-spiffe? or you want to have a product to fetch SVIDs and store that on disk?
You dont need to install spire agent on each workload, but add capabilities to your workload to fetch SVIDs using go-spiffe, or depends on proxies (envoy, ghosttunnel) to fetch SVIDs for you and use that for authentication, or you can use spiffe-helper to store SVIDs on disk.

Although I put the same path for the socket agent.sock and mounted it to the pods, it is sometimes not found by the pods, how should I work with this socket ?

In K8s you can use spiffe-csi, to inject spire-agent socket, you can see this example

Once everything will work as expected (I hope so at least ;) ), how should I retrieve the certificates issued by my workloads ? Are they even available from these pods so I could be able to create an https session for example ?

You can take a look to go-spiffe examples to see how to use your SVIDs.

[ConstantRohmer]

Thank you for your answer !

you can list attested agent using spire-server agent list:

1. I checked on the server, my 3 agents are attested and their attestation is renewed every hour.

To see if SVIDs are propagated you will need to rely on logs:

2. The logs of the server show messages like "Signed X509 SVID", "Renewing agent SVID" and "Rotating server SVID", so my agents and application pods are attested. However, when I run the command :

kubectl exec -it -n spire spire-agent-sj8qk -- spire-1.8.7/bin/spire-agent api fetch -socketPath /run/spire/sockets/agent.sock

I get the error :
Defaulted container "spire-agent" out of: spire-agent, init (init) rpc error: code = PermissionDenied desc = no identity issued command terminated with exit code 1

Shouldn't I be able to verify the issuance of the SVIDs of my workloads with this command ?

3. Thank you for the last parts, I will go check go-spiffe, because I never encountered this name in the spire documentation 😢
   However, I believe this is the kind of library that I need to establish mTLS using the generated certificates from spire.

[MarcosDY]

The logs of the server show messages like "Signed X509 SVID", "Renewing agent SVID" and "Rotating server SVID", so my agents and application pods are attested. However, when I run the command :

what attestor are you using? unfortunately there is an known issue where you can't attest agent (as a workload) using k8s attestor, if you want to attest your agent as a workload you will need to use another attestor (like unix attestor)

Shouldn't I be able to verify the issuance of the SVIDs of my workloads with this command ?

Nop, your workload must attest successfully to verify SVID issuance, so, your workload must be the one calling,
but as you mentioned if you take a look to agent logs, you will see logs about entries propagation (and SVIDs propagation) that you can use to verify that your entries are propagated to agent.

However, I believe this is the kind of library that I need to establish mTLS using the generated certificates from spire

Yes, you have java-spiffe and py-spiffe too, if you need another language,
or you can rely in proxies like envoy or ghostunnel

[MarcosDY]

@ConstantRohmer could you figure it out?
we can talk on slack if this can help you

[ConstantRohmer]

Hi @MarcosDY, I would like to thank you for your help and follow up on this subject, which was really appreciated !

I won't be able to continue working on this subject since it was a school project and we reached the end of the deadlines. However, it is possible that another group of students will resume from where we stopped and all the information you gave will probably point them in the right direction.

See you on another subject one day maybe ;)

[azdagron]

Sounds good. We'll close for now.