Elastic CI agent fails to start when AWS Linux kernel is updated

[gueletk-affirm]

Describe the bug
When using an AMI where all the security fixes have been run (goes up to 4.14.262-200.489), the template fails during the AgentAutoScaleGroup creation.

Steps To Reproduce
Steps to reproduce the behavior:

1. Launch an EC2 instance using the BuildKite AMI (with a public IP)
2. SSH into the server and follow these instructions to update the kernel with the security patches
3. Make a new AMI from that server
4. Launch the Elastic CI stack using the new AMI

Expected behavior
The stack is created successfully

Actual behaviour
The stack fails during the AgentAutoScaleGroup creation.

Stack parameters (please complete the following information):

• AWS Region: us-east-1
• Version: Buildkite stack v5.7.2

Additional context

• Instance screenshot from the failing AutoScalingGroup

• Output of dmesg | grep docker:

  • [ 63.925260] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready

• Version of buildkite-agent inside instance: version 3.33.3, build 4013

• have tried running the BK template with stable, beta, and edge values for BuildkiteAgentRelease

• the closest thing I could find to this error was here, but I'm not sure how to change those settings in the AMI/template

Would really appreciate any guidance here.

[moskyb]

hi there @gueletk-affirm! there's a chance you're running into the issue we came across in #995 - there's something funky with the latest versions where on EC2 instances backed by Nitro, cloud-init fails to run for some unknown reason. when i generate instance screenshots from my affected instances in our sandbox account, they look pretty much identical to yours posted above.

to help me diagnose, would you be able to send through:

• the instance type you're using to launch your stack
• the output of the hostnamectl command on the affected instance(s) - mine looks like:

$ hostnamectl
   Static hostname: ip-<snip>.ec2.internal
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 84039ce <snip>
           Boot ID: 9db4fe<snip>
    Virtualization: amazon
  Operating System: Amazon Linux 2
       CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2
            Kernel: Linux 4.14.262-200.489.amzn2.x86_64
      Architecture: x86-64

• The output of the command cat /var/log/cloud-init.log | grep "/user-data" on the affected instance

The key symptom of this issue for us was having ec2 instances that appeared to be running okay, but weren't actually registering agents, and weren't picking up any work for running buildkite pipelines.

[moskyb]

if this is the issue that's affecting you, we're going to be releasing a new version of the elastic stack either today or tomorrow (New Zealand time) that includes an AMI update to use the most recent version of Amazon Linux not affected by this bug. Once AWS releases a fixed AMI, we'll release a patch for that too.

[moskyb]

hi @gueletk-affirm - i think we've fixed this in the latest version release, v5.8.0 - only 24 days late! I'm going to close this issue out for now, but feel free to reopen it if this issue rears its ugly head again :)