Upgrade from v3.8.1 to v3.9.0 causes "could not load AWS config" error

[heydonovan]

I've upgraded our sops binary from v3.8.1 to v3.9.0. Without changing any encrypted file, our pipeline jobs started failing due to this new error. I've put together a few steps to replicate the issue. Let me know if you need any more info!

$ cat Dockerfile
FROM python:latest

RUN wget -q -O /usr/local/bin/sops381 https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1.linux.amd64 \
  && chmod +x /usr/local/bin/sops381 \
  && sops381 --version

RUN wget -q -O /usr/local/bin/sops390 https://github.com/getsops/sops/releases/download/v3.9.0/sops-v3.9.0.linux.amd64 \
  && chmod +x /usr/local/bin/sops390 \
  && sops390 --version

COPY dev.env dev.env

$ cat dev.env
FOO=ENC[AES256_GCM,data:YN03,iv:KSsqQL28j+Vpt0YZuBa+XXjCfxhPSHEMKXaMmfLcMLE=,tag:zM/6m3Xw2Gy7hRe/PXwT+w==,type:str]
sops_kms__list_0__map_arn=arn:aws:kms:REDACTED:REDACTED:key/REDACTED
sops_kms__list_0__map_aws_profile=REDACTED
sops_kms__list_0__map_created_at=2023-09-28T00:28:38Z
sops_kms__list_0__map_enc=REDACTED
sops_kms__list_1__map_arn=arn:aws:kms:REDACTED:REDACTED:key/REDACTED
sops_kms__list_1__map_aws_profile=REDACTED
sops_kms__list_1__map_created_at=2023-09-28T00:28:38Z
sops_kms__list_1__map_enc=REDACTED
sops_lastmodified=2024-07-02T17:29:39Z
sops_mac=ENC[AES256_GCM,data:REDACTED,type:str]
sops_unencrypted_suffix=_unencrypted
sops_version=3.8.1

$ docker build -t sops .

$ docker run -it --entrypoint=/bin/bash sops

$ sops381 --verbose --decrypt dev.env
[AWSKMS]	 time="2024-07-02T17:21:36Z" level=info msg="Decryption succeeded" arn="arn:aws:kms:REDACTED"
[SOPS]	 time="2024-07-02T17:21:36Z" level=info msg="Data key recovered successfully"
[SOPS]	 time="2024-07-02T17:21:36Z" level=debug msg="Decrypting tree"

$ sops390 ---verbose --decrypt dev.env
[AWSKMS]	 time="2024-07-02T17:21:36Z" level=info msg="Decryption failed" arn="arn:aws:kms:REDACTED"
[AWSKMS]	 time="2024-07-02T17:21:36Z" level=info msg="Decryption failed" arn="arn:aws:kms:REDACTED"
Failed to get the data key required to decrypt the SOPS file.
Group 0: FAILED
  arn:aws:kms:REDACTED: FAILED
    - | could not load AWS config: failed to get shared config
      | profile, dev
  
  arn:aws:kms:REDACTED: FAILED
    - | could not load AWS config: failed to get shared config
      | profile, dev
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

[jonasbadstuebner]

Hey, can you test something for me please? Are you able to build your own sops version? If so, could you look at the following comment and change

sops/config/config.go

Line 193 in 8b5cafe

key := fmt.Sprintf("%T/%v", v, v.ToString())

to

// keytype + toMap make a unique combination per key type
valueMap := v.ToMap()
delete(valueMap, "created_at")
delete(valueMap, "enc")

key := fmt.Sprintf("%T/%v", v, valueMap)

and report if this fixes your problem?

It could be that the "make keys unique" is causing your issue.
(I know this is not the clean solution we would wish for in the long term, I just want to know if this is the problem)

[felixfontein]

Since the encrypted file contains two keys, and the 3.9.0 error message also mentions two keys, I don't think this is related to making keys unique since both keys seem to be tried.

My guess is that this is related to some updated dependency. I don't use any KMS (and have access to none) so I have no way to test this myself...

[elopsod]

hi
got same error(