Consul ACL bootstrap error

[const-tmp]

Describe the bug
Consul secret engine ACL bootstrap doesn't work as described in documentation
After vault write consul/config/access address="127.0.0.1:8500" we got an error:

➜  ~ vault write consul/config/access \
    address="127.0.0.1:8500"

Error writing data to consul/config/access: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/consul/config/access
Code: 400. Errors:

* Token not provided and failed to bootstrap ACLs

To Reproduce
Steps to reproduce the behavior:

1. vault server -dev
2. vault secrets enable consul
3. vault write consul/config/access address="127.0.0.1:8500"
4. See error

Expected behavior
Vault bootstraps Consul ACL as described in docs

Environment:

• Vault Server Version (retrieve with vault status):

➜  ~ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.12.2
Build Date      2022-11-23T12:53:46Z
Storage Type    inmem
Cluster Name    vault-cluster-3798f231
Cluster ID      045672f5-3092-7d42-5a2d-503ea731f969
HA Enabled      false

• Vault CLI Version (retrieve with vault version):

➜  ~ vault version
Vault v1.12.2 ('415e1fe3118eebd5df6cb60d13defdc01aa17b03+CHANGES'), built 2022-11-23T12:53:46Z

• Server Operating System/Architecture: MacOS 13.0.1 M1

Vault server configuration file(s):
vault dev server or production one

[tomhjp]

This is probably either because the ACLs have already been bootstrapped on the consul cluster, or because ACL support is disabled in consul. Config reference here.

You can try starting up a dev agent with consul agent -dev -hcl "acl { enabled = true }", and that should enable Vault to bootstrap the ACLs without providing a token.

I've opened #18750 to help make this clearer in the docs. LMK if you have any feedback on that!

[const-tmp]

I have exactly the same issue right now. Was trying during couple of hours, what am I doing wrong?
ACL was NOT bootstrapped:

➜  ~ vault secrets enable consul
Success! Enabled the consul secrets engine at: consul/
➜  ~ vault write consul/config/access \
    address="https://consul.service.consul:8501"
Error writing data to consul/config/access: Error making API request.

URL: PUT https://70.34.246.49:8200/v1/consul/config/access
Code: 400. Errors:

* Token not provided and failed to bootstrap ACLs
➜  ~ vault secrets disable consul
Success! Disabled the secrets engine (if it existed) at: consul/
➜  ~ vault secrets enable consul
Success! Enabled the consul secrets engine at: consul/
➜  ~ vault write consul/config/access \
    address="consul.service.consul:8501"
Error writing data to consul/config/access: Error making API request.

URL: PUT https://70.34.246.49:8200/v1/consul/config/access
Code: 400. Errors:

* Token not provided and failed to bootstrap ACLs
➜  ~ vault secrets disable consul
Success! Disabled the secrets engine (if it existed) at: consul/
➜  ~ vault secrets enable consul
Success! Enabled the consul secrets engine at: consul/
➜  ~ vault write consul/config/access \
    address="consul.service.consul:8501" \
>   scheme="https" \
>   client_cert="$(cat $CONSUL_CLIENT_CERT)" \
>   client_key="$(cat $CONSUL_CLIENT_KEY)"
Error writing data to consul/config/access: Error making API request.

URL: PUT https://70.34.246.49:8200/v1/consul/config/access
Code: 400. Errors:

* Token not provided and failed to bootstrap ACLs
➜  ~ vault write consul/config/access \
    address="consul.service.consul:8501" \
  scheme="https" \
  client_cert="$(cat $CONSUL_CLIENT_CERT)" \
  client_key="$(cat $CONSUL_CLIENT_KEY)"
Error writing data to consul/config/access: Error making API request.

URL: PUT https://70.34.246.49:8200/v1/consul/config/access
Code: 400. Errors:

* Token not provided and failed to bootstrap ACLs
➜  ~ vault write consul/config/access \
    address="consul.service.consul:8501" \
  scheme="https" \
  client_cert="$(cat $CONSUL_CLIENT_CERT)" \
  client_key="$(cat $CONSUL_CLIENT_KEY)" \
> ca_cert="$(cat tls/root-ca.pem)"
Error writing data to consul/config/access: Error making API request.

URL: PUT https://70.34.246.49:8200/v1/consul/config/access
Code: 400. Errors:

* Token not provided and failed to bootstrap ACLs
➜  ~ consul acl token list
Failed to retrieve the token list: Unexpected response code: 403 (Permission denied: anonymous token lacks permission 'acl:read'. The anonymous token is used implicitly when a request does not specify a token.)
➜  ~ consul acl bootstrap
AccessorID:       b76a95b3-2617-df66-8c96-20d704e8a727
SecretID:         f3d8151d-c14c-a6ef-9b4a-e6d9ee5be93b
Description:      Bootstrap Token (Global Management)
Local:            false
Create Time:      2023-05-29 17:33:39.035347913 +0000 UTC
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

Vault debug logs say pretty much nothing:

2023-05-29T17:26:43.603Z [INFO]  core: successful mount: namespace="" path=consul/ type=consul version=""
2023-05-29T17:26:43.741Z [DEBUG] core.secrets.deletion: clearing view: namespace=root path=consul/ total_keys=0
2023-05-29T17:26:43.741Z [DEBUG] core.secrets.deletion: view cleared: namespace=root path=consul/

Consul logs only logged DNS queries, but logs were rotated, just trust me :)

[maxb]

I have exactly the same issue right now. Was trying during couple of hours, what am I doing wrong?

As all of your attempts involved talking to a Consul cluster over HTTPS, my guess would be an issue with the TLS certificate in use by the Consul server not being accepted by Vault.

I suggest trying to connect to Consul over plain HTTP, as this would be an easy way to confirm/deny this idea.

[const-tmp]

@maxb Thanks for the reply! You were right, TLS is the issue, just tried plain HTTP connection and it worked.

[tomhjp]

Thanks for the follow up - the error from the Consul API is currently being swallowed, so I opened a PR to fix that in #20891