BOSH Agent not reachable due to issues with blobstore settings

[anshrupani]

We identified some issues with the recent CPI version bumps, where the agent is not able to start due to missing blobstore properties. We guess these issues might have been triggered by #147.

On the outside, we see the following while deploying the director:

Deploying:
Creating instance 'bosh/0':
  Waiting until instance is ready:
    Post "https://mbus:<redacted>@<director-hostname>:6868/agent": dial tcp <director-ip>:6868: connect: connection refused

Here's what we observed in the agent logs on the director VM:

2023-04-27_14:38:07.96970 [main] 2023/04/27 14:38:07 ERROR - App setup Getting blobstore: Getting blobstore: Validating blobstore: executable bosh-blobstore- not found in PATH
2023-04-27_14:38:07.96971 [main] 2023/04/27 14:38:07 ERROR - Agent exited with error: Getting blobstore: Getting blobstore: Validating blobstore: executable bosh-blobstore- not found in PATH

The settings.json, on the other hand consists the following, which might be expected:
{{"blobstore":{"provider":"","options":null} }}

We observe this on AWS and GCP bosh-cpi releases (version 98 and 47 respectively). Azure (v44) also most probably has the issue, but we did not test it yet.

[beyhan]

Some context/documentation about the blobstore settings is available in https://bosh.io/docs/agent-cpi-interactions/#agent-settings-format. @anshrupani is there no env with blobstore configuration in the settings.json? Like:

...
"env": {
    "bosh": {
      "authorized_keys": [
        "--AUTHORIZED KEY--"
      ],
      "blobstores": [
        {
...

[beyhan]

I think what happens is that in create-env the bosh-cli is using this properties to configure the blobstore. The env in the settings.json is available for the VMs created by bosh director see. I think that create-env doesn't populate the env that is why after the removal in the blobstore properties are missing.

[ystros]

You need to set the blobstore settings under the env.bosh section of the manifest. Passing agent settings under this section has been standard for a while now. Previously, because the CPIs had their own set of blobstore properties, it would go through the code path @beyhan points out. Now that they have been removed, you must set them under env.bosh.

For bosh create-env environments, this is under resource_pools, whereas setting the properties for the CPI were under cloud_provider_section. An example in bosh-deployment:
https://github.com/cloudfoundry/bosh-deployment/blob/c62c8bd8265cef7534ba03d6c012601476304d87/bosh.yml#L158-L163

I'll add more obvious breaking change notes to each of the CPI releases

[beyhan]

Ok, somehow I missed the reasource_pools option for bosh create-env. Good to know. In this case we should clean up the bosh.yml manifest in bosh-deployment. I created this pr. Or is there any other reason to keep blobstore properties under cloud_provider?

[beyhan]

@anshrupani is this issue solved for you?

[anshrupani]

Oh, thanks @beyhan and @ystros for pointing out the issue :)
We have indeed not updated our manifests. We will test it out and provide feedback soon.

[ramonskie]

@anshrupani can we close this issue?

[anshrupani]

Hi @ramonskie, @beyhan
We were blocked in our testing due to other issues, but we can confirm that suggestions provided by @ystros help.
We'll close the issue now.