VMs not being deleted

[ssilva-soap]

After some time using it, I noticed that some VMs were not deleted after the task ended.
I don't know what information I need to provide to help solve it. I don't even know if it's related to you code or some GCP problem.

[ssilva-soap]

I checked and two VMs are being created for each workflow.
So when the workflow finish just one VM is being deleted

I'm trying to understand it better

[Tereius]

A new VM should be created for each job in a workflow. The VM then processes the assigned job and is deleted when the job is completed. This is the basic procedure. In the github workflow view you, can understand this as follows:

Each newly started job (within a workflow) has a yellow dot (with no spinner around it) - in the screenshot we see two newly started jobs running in parallel.

The two VMs are now provisioned in the GCP by the github-runner-autoscaler CloudRun. Have a look into the logs of the github-runner-autoscaler CloudRun - you should see two 'queued' webhook messages:

It takes about 1 minute until each VM is up and running. Each VM then takes exactly one job (a VM will never take more than one job). As soon as a VM has taken a job, the yellow spinner is displayed, which rotates around the yellow dot - in the screenshots, we can see that both jobs are processed in parallel by two VMs.

Once the jobs have been processed, the github-runner-autoscaler CloudRun will receive two 'completed' webhook messages:


Both VMs are then deleted by the github-runner-autoscaler CloudRun.

Please try this minimal workflow to determine if the error persists:

name: test
on: [push]
jobs:
  FirstJob:
    runs-on: self-hosted
    name: "FirstJob"
    steps:
      - run: echo "Hello from VM one"; sleep 60
  SecondJob:
    runs-on: self-hosted
    name: "SecondJob"
    steps:
      - run: echo "Hello from VM two"; sleep 60

Also check if you see any error messages in the github-runner-autoscaler CloudRun logs. You can find more detailed documentation on how it works here: https://github.com/Privatehive/gcp-hosted-github-runner/tree/master/runner-autoscaler

[ssilva-soap]

I discovered that two VMs are being created when one workflow with only one task is started.
So only the VM that is used to run the task is being deleted.
That's the evidence:

[ssilva-soap]

I don't know why Github is sending 3 POST to the webhook.
It sends an "action:queued" after sends "actions:waiting" and then "actions:queued" again.
That's why 2 different VMs are being created

[Tereius]

Did you maybe register the webhook twice? Maybe one time on the Organization level and one time on the Repository level?

Does your Cloud Run allow concurrency > 1?

It is important that each webhook is acknowledged within 10 seconds. Otherwise the webhook will be sent again, resulting in duplicates.

In your logs, I see a workflow job action “waiting” that I have never seen before. You are probably using requried reviews in the workflow. I have never tested this case - which may cause the action 'queued' to be sent twice. Once before the review and once after the review. If this is the case, I should be able to reproduce this case and maybe fix it.

[ssilva-soap]

I'm using this config:

on:
push:
branches:
- development
pull_request:
branches:
- development

I think Github is triggering the workflow twice under the hood, I don't know.
I will test it only with on trigger to understand if it's the problem and I'll let you know

[Tereius]

Okay, I can reproduce it now. When I insert a deployment environment (that requires a review) into the workflow, a “queued” action webhook is sent before the review has actually taken place. Then a second “queued” action webhook is sent after the review. The problem is that we have to ignore the first “queued” webhook, but there is nothing in the webhook payload to detect that a review is pending. The only indicator is that a “waiting” action webhook is sent a few milliseconds after the first “queued” webhook. I'll have to think about a workaround - maybe it's even a GitHub webhook bug that we get a “queued” action webhook before the review.

[ssilva-soap]

But I'm not using review in this workflow, which is strange.
I'll test some other thing here, and I'll let you know.

[Tereius]

I implemented a workaround: if a "waiting" webhook event shortly follows (within 10 seconds) the "queued" event the create_vm cloud task callback will be canceled thus no VM will be created for the first queued event. Please run terraform init -upgrade to update the module to the latest version.

[Tereius]

Is it working for you? Can we close the issue?

[ssilva-soap]

I'm still seeing some VMs being created and turned off after because no job was assigned.
And now I see that VMs are getting longer to be ready when I start more jobs simultaneously, because of the timer you implemented, I think.

But now I understand what you did, and I can personalize the code to meet what I need.
Thanks for the help!

[Tereius]

I am closing this issue for now. Please let me know if you have any improvements that should be merged.