Timeout when trying to provision new workers on GCP

[ashea-code]

I know this repo is becoming out of date, bit I'm trying to re-run the concourse-up command on an existing deployment in GCP. I have exported WORKERS=2 to add an extra worker to the pool.

However, I get this as an error:

Task 1175

Task 1175 | 23:02:09 | Preparing deployment: Preparing deployment (00:02:16)
                    L Error: worker/ed0abfe3-0867-49e1-9092-d242f833bd74: Timed out sending 'get_state' to instance: 'worker/ed0abfe3-0867-49e1-9092-d242f833bd74', agent-id: 'be09322d-de63-4f1a-9d55-54925a64270a' after 45 seconds
Task 1175 | 23:04:26 | Error: worker/ed0abfe3-0867-49e1-9092-d242f833bd74: Timed out sending 'get_state' to instance: 'worker/ed0abfe3-0867-49e1-9092-d242f833bd74', agent-id: 'be09322d-de63-4f1a-9d55-54925a64270a' after 45 seconds

Task 1175 Started  Mon Jun  3 23:02:09 UTC 2019
Task 1175 Finished Mon Jun  3 23:04:26 UTC 2019
Task 1175 Duration 00:02:17
Task 1175 error

Updating deployment:
  Expected task '1175' to succeed but state is 'error'

Exit code 1

Any idea on what is timing out here? GCP is known to be a bit slow on provisioning.

[DanielJonesEB]

Hmm, I'm not sure. We've seen the GCP CPI time out regularly and intermittently (we need to bump the version of the CPI to fix it) but normally with different errors:

Task 10 | 12:28:32 | Error: CPI error 'Bosh::Clouds::CloudError' with message 'Creating vm: Failed to find Google Image 'stemcell-e5d99deb-c5b4-4f5f-53ad-87ef7e71d15a': Get https://www.googleapis.com/compute/v1/projects/ps-amcginlay/global/images/stemcell-e5d99deb-c5b4-4f5f-53ad-87ef7e71d15a?alt=json: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp 108.177.111.84:443: i/o timeout' in 'create_vm' CPI method (CPI request ID: 'cpi-653711')

Is the issue intermittent for you?

[ashea-code]

This issue isn't intermittent, and trying to make a fresh install also presents me with:

Task 10

Task 10 | 21:46:23 | Preparing deployment: Preparing deployment (00:00:01)
Task 10 | 21:46:24 | Preparing deployment: Rendering templates (00:00:02)
Task 10 | 21:46:26 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 10 | 21:46:26 | Creating missing vms: web/ecade835-5d6e-416e-830c-fb8d648e99ef (0)
Task 10 | 21:46:26 | Creating missing vms: worker/1115b42a-0163-487f-8938-0e23fd19f6c8 (0)
Task 10 | 21:46:26 | Creating missing vms: worker/83502149-1c15-41eb-8838-0af800d0a49f (2)
Task 10 | 21:46:26 | Creating missing vms: worker/dc3fdd8f-26d6-4cbf-88a5-678e21a5dddf (1)
Task 10 | 21:47:28 | Creating missing vms: web/ecade835-5d6e-416e-830c-fb8d648e99ef (0) (00:01:02)
Task 10 | 21:47:49 | Creating missing vms: worker/1115b42a-0163-487f-8938-0e23fd19f6c8 (0) (00:01:23)
Task 10 | 21:47:50 | Creating missing vms: worker/dc3fdd8f-26d6-4cbf-88a5-678e21a5dddf (1) (00:01:24)
Task 10 | 21:47:50 | Creating missing vms: worker/83502149-1c15-41eb-8838-0af800d0a49f (2) (00:01:24)
Task 10 | 21:47:51 | Updating instance web: web/ecade835-5d6e-416e-830c-fb8d648e99ef (0) (canary)
Task 10 | 21:47:51 | Updating instance worker: worker/1115b42a-0163-487f-8938-0e23fd19f6c8 (0) (canary) (00:01:07)
Task 10 | 21:48:58 | Updating instance worker: worker/83502149-1c15-41eb-8838-0af800d0a49f (2)
Task 10 | 21:48:58 | Updating instance worker: worker/dc3fdd8f-26d6-4cbf-88a5-678e21a5dddf (1) (00:01:02)
Task 10 | 21:50:01 | Updating instance worker: worker/83502149-1c15-41eb-8838-0af800d0a49f (2) (00:01:03)
Task 10 | 21:59:49 | Updating instance web: web/ecade835-5d6e-416e-830c-fb8d648e99ef (0) (canary) (00:11:58)
                   L Error: 'web/ecade835-5d6e-416e-830c-fb8d648e99ef (0)' is not running after update. Review logs for failed jobs: atc, grafana
Task 10 | 21:59:49 | Error: 'web/ecade835-5d6e-416e-830c-fb8d648e99ef (0)' is not running after update. Review logs for failed jobs: atc, grafana

Task 10 Started  Tue Jun  4 21:46:23 UTC 2019
Task 10 Finished Tue Jun  4 21:59:49 UTC 2019
Task 10 Duration 00:13:26
Task 10 error

Updating deployment:
  Expected task '10' to succeed but state is 'error'

Exit code 1

Has something changed on GCP?