-
Notifications
You must be signed in to change notification settings - Fork 9.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CoreDNS settings are not appropriate when using fargate profile for kube-system namespace #11327
Comments
This is definitely a problem. |
Please see my response to a similar issue here: #11383 (comment) Most importantly:
For additional context, this comment is referring to the available EKS APIs, which the Terraform AWS Provider interacts with the EKS service. Interactions with Kubernetes can be handled via the mentioned Terraform Kubernetes Provider, or potentially solutions such as using the Since this type of functionality is out of scope of this particular codebase unless the service adds API functionality, I'm going to close this issue. The maintainers here can only suggest that you reach out to AWS via https://github.com/aws/containers-roadmap, a feature request AWS Support case, or talking to your AWS account managers to have the EKS service automatically remove that annotation, implement API functionality to update/remove annotations, or some other mechanism to improve the handling of this problem for automated deployments. If you do feel there is a deficiency in the Terraform AWS Provider documentation or an actual bug with the Terraform AWS Provider's interaction with the available EKS APIs however, please do create another issue and we'll take another look. 👍 |
I disagree with this assessment. This issue is not related to kubernetes management. This issue is highlighting a bug that states that the annotation that aws_eks_fargate_profile attaches to the resultant pod definitions is wrong. It is attaching: This is a bug in the provider, and not a Kubernetes management issue. |
Hi again, @IanMoroney The Terraform AWS Provider is issuing the EKS CreateFargateProfile API call here: Any logic for the Kubernetes annotations within the Kubernetes deployments of the EKS Cluster is managed by the EKS service itself. The Terraform AWS Provider can only provide what is available in the API and the API parameters do not seem to give any ability to configure those annotations. If you have a bug report to file with EKS, you'll need to reach out to AWS Support (the Terraform AWS Provider is unaffiliated with AWS except for business relationships behind the scenes). If I am missing something that we can control with the EKS API and AWS Go SDK, please do let us know. |
eksctl handles this on their side in order to ensure that coredns gets scheduled correctly: https://github.com/weaveworks/eksctl/blob/master/pkg/fargate/coredns/coredns.go They set |
This file references the Kubernetes Go client (e.g. Terraform Providers are generally designed around managing one underlying "API client". We enforce this separation of concerns so each Terraform Provider can be separately maintained on its own development cycle with experts in that particular functionality (see also the list of available Terraform Providers). The Terraform AWS Provider boundary is the AWS API functionality provided by the AWS service APIs. We explicitly draw boundaries on the functionality provided by the Terraform AWS Provider since AWS' abilities cover many varying infrastructure components (e.g. many types of databases with RDS, Kafka with MSK, Kubernetes with EKS, etc.) and we do not intend to support all the underlying infrastructure management (e.g. MySQL client, Kafka client, Kubernetes client, etc.) within this one Terraform Provider as it would be unsustainable and cause a sub-par experience. The Terraform Kubernetes Provider is one of the canonical Terraform Providers designed and maintained against the Kubernetes Go client. That codebase and its published provider should have the functionality being requested to connect to Kubernetes and configure it appropriately. If not, please create a feature request in the Terraform Kubernetes Provider repository. |
Ah ok, I see what you're saying now. I understand why this wouldn't want to be baked into the provider. I was under the impression that because the launch-type was being set to ec2 in the first place, that it was something the provider or the API had control over, but it seems like that's not the case. Thank you for the explanation @bflad |
FWIW if you REALLY want to do this in terraform and you know
I didn't promise it'd be pretty 😝, but it does work and also uses the config directly from the cluster you're creating without relying on a local kubeconfig. |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
This issue was originally opened by @sflaherty2009 as hashicorp/terraform#23692. It was migrated here as a result of the provider split. The original body of the issue is below.
Terraform Version
Expected Behavior
When spinning up a cluster with AWS EKS with Fargate profiles for kube-system CoreDNS pods appropriately builds and starts appropriately
Actual Behavior
When spinning up a cluster with AWS EKS with Fargate profiles for kube-system CoreDNS pods go to a pending state due to a node not being available with appropriate labels. The label that it is looking for is
eks.amazonaws.com/compute-type : ec2
. This label does not exist with Fargate instances and must be removed prior to CoreDNS working appropriately. This can be removed withkubectl patch deployment coredns -n kube-system --type json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'
as outlined in AWS documentation. https://docs.aws.amazon.com/eks/latest/userguide/fargate-getting-started.htmlSteps to Reproduce
Spin up Fargate profile for the kube-system namespace.
Additional Context
References
The text was updated successfully, but these errors were encountered: