Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HostProcess Container Configuration for k8s #864

Merged
merged 1 commit into from
Feb 7, 2022

Conversation

jsturtevant
Copy link
Contributor

Kubernetes now has support for HostProcess Containers so this is a continuation of #581 using HostProcess containers

Requirements:

  • Kuberentes 1.22+
  • containerd 1.6 Beta+
  • WindowsHostProcessContainers feature-gate (currently in alpha in 1.22 going beta in 1.23) turned on for kube-apiserver and kubelet if using Kubernetes 1.22

@jsturtevant jsturtevant requested a review from a team as a code owner November 5, 2021 19:54
@jsturtevant
Copy link
Contributor Author

I know the requirements are hard to come by so for anyone that wants to try this out there are few options available currently:

kubernetes/kubernetes.md Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
kubernetes/kubernetes.md Outdated Show resolved Hide resolved
kubernetes/kubernetes.md Outdated Show resolved Hide resolved
kubernetes/windows-exporter-daemonset.yaml Outdated Show resolved Hide resolved
kubernetes/windows-exporter-daemonset.yaml Outdated Show resolved Hide resolved
@jsturtevant jsturtevant mentioned this pull request Nov 10, 2021
kubernetes/windows-exporter-daemonset.yaml Outdated Show resolved Hide resolved
kubernetes/kubernetes.md Outdated Show resolved Hide resolved
kubernetes/windows-exporter-daemonset.yaml Outdated Show resolved Hide resolved
kubernetes/windows-exporter-daemonset.yaml Outdated Show resolved Hide resolved
kubernetes/windows-exporter-servicemonitor.yaml Outdated Show resolved Hide resolved
kubernetes/windows-exporter-servicemonitor.yaml Outdated Show resolved Hide resolved
@jsturtevant jsturtevant force-pushed the use-hostprocess branch 8 times, most recently from d0c0da6 to 0a4431e Compare December 1, 2021 23:27
@jsturtevant
Copy link
Contributor Author

@carlpett I've addressed all the feedback. Ready for a final review and to sort out the image pushing mechanism. We only need Linux machine and Container Registry since we are using buildx to create the image.

Copy link
Collaborator

@carlpett carlpett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me, thanks a lot!
We still need the credentials, though, and looks like SuperQ is busy. Perhaps @RichiH has an idea on who could help out?

@jsturtevant
Copy link
Contributor Author

Let me know if can help in anyway.

@jsturtevant
Copy link
Contributor Author

@carlpett anything we can do to move this along? Is there a community meeting I can add it to the agenda? or maybe start a conversation in slack?

@carlpett
Copy link
Collaborator

carlpett commented Jan 6, 2022

Hi @jsturtevant,
We actually just completed a migration from AppVeyor to Github Actions, which means we should now be able to utilize the builtin container registry here without account management headaches.

So I think the next step would be amending the release step to build and push the image. Something you'd be comfortable doing?

@jsturtevant
Copy link
Contributor Author

sweet! I'll take a look tomorrow and see if I have questions. Thanks!

@jsturtevant
Copy link
Contributor Author

Looked into it a bit and looks like the github action jobs are running on Windows. Buildx doesn't work on windows so I will need to adjust this a bit to build containers on Windows. Shouldn't be a big deal.

I was also building the image in the container but you have the image on the build machine so I could just copy that into the container. This way its the same executable that is being released.

@breed808
Copy link
Contributor

breed808 commented Jan 8, 2022

It's worth noting that the OS can be set at the job level, so in theory you could add another job running Linux and use the artifacts from the Build job.

@jsturtevant jsturtevant force-pushed the use-hostprocess branch 5 times, most recently from f3e9a34 to 9caa917 Compare January 11, 2022 01:16
@jsturtevant
Copy link
Contributor Author

It's worth noting that the OS can be set at the job level, so in theory you could add another job running Linux and use the artifacts from the Build job.

I think it makes sense to have the build work on Windows since rest of the tooling is working there.

I've pushed the changes to make this work on Windows github action runner as separate commit. I made minimal changes to make it work but let me know if you want a different approach. Can squash if you want me too.

Note to build the multi arch image needed to use the WS2022 runners, which apparently don't have the gitversion tool installed by default.

@jsturtevant jsturtevant force-pushed the use-hostprocess branch 2 times, most recently from 8849fbd to 6836d2a Compare January 11, 2022 18:40
@carlpett
Copy link
Collaborator

This looks good to me, great work! 👏
Did you have the possibility to test it out end-to-end somewhere? Otherwise I think the best we can do is merge, make a draft release and see what happens?

@jsturtevant
Copy link
Contributor Author

I built an image with this code here: https://hub.docker.com/repository/docker/jsturtevant/windows-exporter if you want to try it out (requires k8s 1.22+ and containerd 1.6+)

I will run a quick validation this afternoon.

@jsturtevant
Copy link
Contributor Author

I validated it is working with that image. I will double check once there is a released image

k get nodes -o wide                                                                                                                                                                                                                                                 
NAME                                       STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME                                                                                                                                                                                        
win-conta-qzl6z                            Ready    <none>                 32m   v1.22.2   10.1.0.5      <none>        Windows Server 2019 Datacenter   10.0.17763.2237    containerd://1.6.0-beta.0                                                                                                                                                                                
win-conta-snwfp                            Ready    <none>                 32m   v1.22.2   10.1.0.7      <none>        Windows Server 2019 Datacenter   10.0.17763.2237    containerd://1.6.0-beta.0                                                                                                                                                                                
win-containerd-32766-control-plane-9lltd   Ready    control-plane,master   35m   v1.22.2   10.0.0.4      <none>        Ubuntu 20.04.3 LTS               5.8.0-1041-azure   containerd://1.5.5                                                                                                                                                                                       
win-containerd-32766-md-0-fclv9            Ready    <none>                 33m   v1.22.2   10.1.0.4      <none>        Ubuntu 20.04.3 LTS               5.8.0-1041-azure   containerd://1.5.5                                                                                                                                                                                       
win-containerd-32766-md-0-m4zhn            Ready    <none>                 33m   v1.22.2   10.1.0.6      <none>        Ubuntu 20.04.3 LTS               5.8.0-1041-azure   containerd://1.5.5

                                                                                                                                                                                       
[win-containerd-32766-admin@win-containerd-32766|monitoring]  ~/projects/windows_exporter   use-hostprocess  kgp                                                                                                                                                                                                                                                                 
NAME                                  READY   STATUS    RESTARTS   AGE    IP                NODE                                       NOMINATED NODE   READINESS GATES                                                                                                                                                                                                             
alertmanager-main-0                   2/2     Running   0          21m    192.168.24.130    win-containerd-32766-md-0-fclv9            <none>           <none>                                                                                                                                                                                                                      
alertmanager-main-1                   2/2     Running   0          21m    192.168.178.194   win-containerd-32766-md-0-m4zhn            <none>           <none>                                                                                                                                                                                                                      
alertmanager-main-2                   2/2     Running   0          21m    192.168.24.129    win-containerd-32766-md-0-fclv9            <none>           <none>                                                                                                                                                                                                                      
blackbox-exporter-6798fb5bb4-phvlt    3/3     Running   0          21m    192.168.24.131    win-containerd-32766-md-0-fclv9            <none>           <none>                                                                                                                                                                                                                      
grafana-767fcb6796-tgds5              1/1     Running   0          21m    192.168.178.195   win-containerd-32766-md-0-m4zhn            <none>           <none>                                                                                                                                                                                                                      
kube-state-metrics-6c699dfb8-c8r45    3/3     Running   0          21m    192.168.24.132    win-containerd-32766-md-0-fclv9            <none>           <none>                                                                                                                                                                                                                      
node-exporter-bcgmj                   2/2     Running   0          21m    10.1.0.6          win-containerd-32766-md-0-m4zhn            <none>           <none>                                                                                                                                                                                                                      
node-exporter-hpm78                   2/2     Running   0          21m    10.0.0.4          win-containerd-32766-control-plane-9lltd   <none>           <none>                                                                                                                                                                                                                      
node-exporter-sk9vf                   2/2     Running   0          21m    10.1.0.4          win-containerd-32766-md-0-fclv9            <none>           <none>                                                                                                                                                                                                                      
prometheus-adapter-7dc46dd46d-twgwp   1/1     Running   0          21m    192.168.178.196   win-containerd-32766-md-0-m4zhn            <none>           <none>                                                                                                                                                                                                                      
prometheus-adapter-7dc46dd46d-zh86k   1/1     Running   0          21m    192.168.24.133    win-containerd-32766-md-0-fclv9            <none>           <none>                                                                                                                                                                                                                      
prometheus-k8s-0                      2/2     Running   0          21m    192.168.24.134    win-containerd-32766-md-0-fclv9            <none>           <none>                                                                                                                                                                                                                      
prometheus-k8s-1                      2/2     Running   0          21m    192.168.178.197   win-containerd-32766-md-0-m4zhn            <none>           <none>                                                                                                                                                                                                                      
prometheus-operator-5c875b748-h98gq   2/2     Running   0          22m    192.168.178.193   win-containerd-32766-md-0-m4zhn            <none>           <none>                                                                                                                                                                                                                      
windows-exporter-5c5l7                1/1     Running   0          91s    10.1.0.7          win-conta-snwfp                            <none>           <none>                                                                                                                                                                                                                      
windows-exporter-drjp9                1/1     Running   0          107s   10.1.0.5          win-conta-qzl6z                            <none>           <none>    

                                                                                                                                                                                                                  
[win-containerd-32766-admin@win-containerd-32766|monitoring]  ~/projects/windows_exporter   use-hostprocess  kubectl get --raw=/api/v1/namespaces/monitoring/pods/windows-exporter-5c5l7/proxy/metrics                                                                                                                                                                           
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.                                                                                                                                                                                                                                                                                         
# TYPE go_gc_duration_seconds summary                                                                                                                                                                                                                                                                                                                                               
go_gc_duration_seconds{quantile="0"} 0                                                                                                                                                                                                                                                                                                                                              
go_gc_duration_seconds{quantile="0.25"} 0                                                                                                                                                                                                                                                                                                                                           
go_gc_duration_seconds{quantile="0.5"} 0                                                                                                                                                                                                                                                                                                                                            
go_gc_duration_seconds{quantile="0.75"} 0                                                                                                                                                                                                                                                                                                                                           
go_gc_duration_seconds{quantile="1"} 0.0012611                                                                                                                                                                                                                                                                                                                                      
go_gc_duration_seconds_sum 0.0029943                                                                                                                                                                                                                                                                                                                                                
go_gc_duration_seconds_count 21                                                                                                                                                                                                                                                                                                                                                     
# HELP go_goroutines Number of goroutines that currently exist.                                                                                                                                                                                                                                                                                                                     
# TYPE go_goroutines gauge                                                                                                                                                                                                                                                                                                                                                          
go_goroutines 9                                                                                                                                                                                                                                                                                                                                                                     
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
....

@jsturtevant
Copy link
Contributor Author

do you want me to squash the commits?

- args:
- --config.file=%CONTAINER_SANDBOX_MOUNT_POINT%/config.yml
name: windows-exporter
image: jsturtevant/windows-exporter:latest
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carlpett I need to update this but currently not pushing a latest tag. We won't have a tag until the git repository is tagged and released. Thoughts?

the current format is ghcr.io/prometheus-community/windows-exporter:<versionnumber>

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on other unreleased changes, the next version is going to be 0.18.
I'm not sure if we want a latest tag? On the other hand, documenting it as a fixed version, I doubt we'll remember to update it...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking some more, I think we'll want a "bleeding edge" image. For example, the node_exporter has a master tag. They also have a latest, pointing to latest release. I'm personally not a fan, but I suppose there's probably high demand for it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a "bleeding" edge image. let me know if you want me to squash the commits

@jsturtevant
Copy link
Contributor Author

e2e test failed, It might have been a flake? I don't seem to be able to re-triger the job

@jsturtevant
Copy link
Contributor Author

e2e test failed, It might have been a flake? I don't seem to be able to re-triger the job

git commit --amend --no-edit -s and git push -force worked

@jsturtevant
Copy link
Contributor Author

must be something else. I don't see anything in the directly in the output

@jsturtevant
Copy link
Contributor Author

@carlpett @breed808 Looking at a few other open prs which have this error and this looks like it might be broken on the main branch?

@breed808
Copy link
Contributor

Yes, you'll need to rebase this PR on master to pick up the CI fix. See #924 for context.

@breed808
Copy link
Contributor

breed808 commented Feb 5, 2022

@jsturtevant are you happy for this to be merged? I can't see any open threads besides the bleeding edge image, which has been implemented.

Co-authored-by: Brian Redmond <brianisrunning@gmail.com>
Signed-off-by: Brian Redmond <brianisrunning@gmail.com>
Signed-off-by: James Sturtevant <jstur@microsoft.com>
@jsturtevant
Copy link
Contributor Author

Yes let's do it 🥳 I just rebased to clean up the commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants