Support multiple regions/availabilty zones if an instance type cannot be started

[gilShin]

We are using instances types that require lots of reasources. Because of, from time to time it is impossible to start an instance of some type in a certain region and a certain AZ.
It'll be great if this plugin will be able to run over list of regions/AZs (subnets and security groups) and start the instance where its type is available.

[stas00]

I just came to file the same request, getting 95% of the time this:

Error: AWS EC2 instance starting error
Error: InsufficientInstanceCapacity: We currently do not have sufficient p3.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get p3.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b.
Error: We currently do not have sufficient p3.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get p3.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b.

and this tool fails if I remove aws-region from the config file.

Needing this for a CI, so this is a big issue.

Thank you!

[philschmid]

To support multi-AZ We could make config.input.subnetId a list and try to iterate randomly over it here and stop when an instance started.
I think multi-Region is hard to implement since Security groups, AMIs, etc are always bound to a Region.

[gilShin]

Sounds like a very good idea Thanks!

…

On Fri, Aug 27, 2021, 09:55 Philipp Schmid ***@***.***> wrote: We could make config.input.subnetId a list and try to iterate randomly over it here <https://github.com/machulav/ec2-github-runner/blob/8911518f6d7d3cbfc7de6cf9ecac0931f8025798/src/aws.js#L35> and stop when an instance started. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE5TX6QKIUH46I5DDRLT64ZFLANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jpalomaki]

FWIW, when using EC2 launch templates, one can apparently omit the explicit selection of AZ/subnet.

It might be worth it to investigate/test if this facility would imply intelligent automatic selection of AZ (taking capacity into account).

[jpalomaki]

Turns out one can only omit subnet ID from run parameters, if you are using the default VPC. Per related AWS docs:

[EC2-VPC] If you don't specify a subnet ID, we choose a default subnet from your default VPC for you. If you don't have a default VPC, you must specify a subnet ID in the request.

[gilShin]

If I do use the default vpc, will it try in one AZ or will keep on trying in all AZs? Anyhow I would like that to be across regions as well. I did solve it using a very ugly workaround but seems like working. In the start job, I have added a step that if it fails, it triggers the workflow again with different subnet ID. The subnet is of the start step uses an env variable that uses an input parameter.

…

On Mon, Aug 30, 2021, 21:39 Jukka Palomäki ***@***.***> wrote: Turns out one can only omit subnet ID from run parameters, if you are using the default VPC. [EC2-VPC] If you don't specify a subnet ID, we choose a default subnet from your default VPC for you. If you don't have a default VPC, you must specify a subnet ID in the request. See AWS docs <https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE2MPVD3PKELGNZXOITT7PF6RANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jpalomaki]

If I do use the default vpc, will it try in one AZ or will keep on trying
in all AZs?

I don't know, but I suspect it would just pick one. What I'd like to know is whether the automatic subnet selection algorithm is AZ capacity-aware or not.

[gilShin]

I doubt that and I'll explain why. Capacity of what? Vcpu? Mem? GPU? You can have enough vcpus in a certain AZ but no GPUs. I think it just picks up a subnet randomly and hope for the best. For us it is even worst because we are using mostly GPUs and not all AZs in a certain region has that type that we are looking for.

…

On Mon, Aug 30, 2021, 22:18 Jukka Palomäki ***@***.***> wrote: If I do use the default vpc, will it try in one AZ or will keep on trying in all AZs? I don't know, but I suspect it would just pick one. What I'd like to know is if the automatic selection algorithm is AZ capacity-aware. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE3G443Y42KOWGTW5D3T7PKR3ANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jpalomaki]

All needed capacity. AWS must be checking them anyhow, at instance launch time, to be able to distribute/optimize load.

Looking at https://aws.amazon.com/premiumsupport/knowledge-center/ec2-insufficient-capacity-errors/ and then https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-launch.html#troubleshooting-launch-capacity, the latter link does mention "not specifying AZ" as one solution. 🤷‍♂️

That said, one option could be to abstract have the runner start job in a composite action that tries to try ec2-github-runner start with different region/AZ settings (in consecutive, conditional steps) until it succeeds, returning the start outputs from the first successful step?

EDIT: amended workaround proposal because it looks like composite actions do not support if conditions

[gilShin]

Can be a solution but I think it'll be best if the code can go over list of regions and AZs and try in each and every one of them. I did something similar in Jenkins pipeline

…

On Mon, Aug 30, 2021, 23:31 Jukka Palomäki ***@***.***> wrote: All needed capacity. AWS must be checking them anyhow, at instance launch time, to be able to distribute/optimize load. Looking at https://aws.amazon.com/premiumsupport/knowledge-center/ec2-insufficient-capacity-errors/ and then https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-launch.html#troubleshooting-launch-capacity, the latter link does mention "not specifying AZ" as one solution. 🤷‍♂️ That said, one option could be to abstract the runner start job in a composite action <https://docs.github.com/en/actions/creating-actions/creating-a-composite-action> that tries ec2-github-runner start with different region/AZ settings (in consecutive, conditional steps) until it succeeds, returning the start mode outputs from the successful step? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE4QPMTAGC4AXAZP56DT7PTADANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[stas00]

And while we are at it, when it fails to start because of the region not having an instance I get a while bunch of errors on stopping:

[jpalomaki]

And while we are at it, when it fails to start because of the region not having an instance I get a while bunch of errors on stopping

That can probably be fixed by amending the if condition on the stop job (to also check the result of the start job)

[jpalomaki]

@stas00 would something like this work? (NOTE: untested and some details omitted for brevity):

jobs:
  start-runner:
    runs-on: ubuntu-20.04
    outputs:
      label: ${{ steps.result.outputs.label }}
      ec2-instance-id: ${{ steps.result.outputs.ec2-instance-id }}
    steps:
      ...
      - id: try-us-east-2a
        uses: machulav/ec2-github-runner@v2
        continue-on-error: true
        with:
          mode: start
          subnet-id: subnet-on-2a
          ...
      - id: try-us-east-2b
        if: steps.try-us-east-2a.outcome == 'failure'
        uses: machulav/ec2-github-runner@v2
        continue-on-error: true
        with:
          mode: start
          subnet-id: subnet-on-2b
          ...
      - id: try-us-east-2c
        if: steps.try-us-east-2a.outcome == 'failure' && steps.try-us-east-2b.outcome == 'failure'
        uses: machulav/ec2-github-runner@v2
        with:
          mode: start
          subnet-id: subnet-on-2c
          ...
      - id: result
        run: |
          if [ "${{ steps.try-us-east-2a.outcome }}" = "success" ]; then
            echo "::set-output name=label::${{ steps.try-us-east-2a.outputs.label }}"
            echo "::set-output name=ec2-instance-id::${{ steps.try-us-east-2a.outputs.ec2-instance-id }}"
          fi
          if [ "${{ steps.try-us-east-2b.outcome }}" = "success" ]; then
            echo "::set-output name=label::${{ steps.try-us-east-2b.outputs.label }}"
            echo "::set-output name=ec2-instance-id::${{ steps.try-us-east-2b.outputs.ec2-instance-id }}"
          fi
          if [ "${{ steps.try-us-east-2c.outcome }}" = "success" ]; then
            echo "::set-output name=label::${{ steps.try-us-east-2c.outputs.label }}"
            echo "::set-output name=ec2-instance-id::${{ steps.try-us-east-2c.outputs.ec2-instance-id }}"
          fi

Also, if we supported launch templates (see #65), then multi-region would be less awkward as we could have a template per region (also less inputs per ec2-github-runner step)?

EDIT: fixed the if condition on the try-us-east-2c step to only run if both previous tries failed.

[stas00]

Thank you for writing out a concrete example, @jpalomaki! I will try and report back.

So it's the same sub-region we are using the same AMI /security group, but only rotating though the subnet-ids. Got it.