Logs and events may contain provider credentials

[turkenh]

What happened?

Trying to create an elasticache cluster and it failed for some reason I am still debugging, but realized another problem which is terraform logs and event on cr contains access_key/secret_key:

See the following (intentionally replaced with REDACTED):

  conditions:
  - lastTransitionTime: "2021-10-08T17:02:05Z"
    message: |-
      observe failed: cannot run refresh: cannot refresh: {"@level":"info","@message":"Terraform 1.0.4","@module":"terraform.ui","@timestamp":"2021-10-08T20:02:04.398569+03:00","terraform":"1.0.4","type":"version","ui":"0.1.0"}
      {"@level":"info","@message":"\nInterrupt received.\nPlease wait for Terraform to exit or data loss may occur.\nGracefully shutting down...\n","@module":"terraform.ui","@timestamp":"2021-10-08T20:02:05.046946+03:00","type":"log"}
      {"@level":"error","@message":"Error: Missing required argument","@module":"terraform.ui","@timestamp":"2021-10-08T20:02:05.943625+03:00","diagnostic":{"severity":"error","summary":"Missing required argument","detail":"The argument \"cluster_id\" is required, but no definition was found.","range":{"filename":"main.tf.json","start":{"line":1,"column":859,"byte":858},"end":{"line":1,"column":860,"byte":859}},"snippet":{"context":"resource.aws_elasticache_cluster.sample-cluster","code":"{\"provider\":{\"tf-provider\":{\"access_key\":\"REDACTED\",\"region\":\"us-east-1\",\"secret_key\":\"REDACTED\",\"token\":\"\"}},\"resource\":{\"aws_elasticache_cluster\":{\"sample-cluster\":{\"apply_immediately\":null,\"availability_zone\":null,\"az_mode\":null,\"cluster_identifier\":\"sample-cluster\",\"engine\":\"memcached\",\"engine_version\":null,\"final_snapshot_identifier\":null,\"lifecycle\":{\"prevent_destroy\":true},\"maintenance_window\":null,\"node_type\":\"cache.t2.micro\",\"notification_topic_arn\":null,\"num_cache_nodes\":1,\"parameter_group_name\":\"default.memcached1.4\",\"port\":11211,\"preferred_availability_zones\":null,\"replication_group_id\":null,\"security_group_ids\":null,\"security_group_names\":null,\"snapshot_arns\":null,\"snapshot_name\":null,\"snapshot_retention_limit\":null,\"snapshot_window\":null,\"subnet_group_name\":null,\"tags\":null,\"tags_all\":null}}},\"terraform\":{\"required_providers\":{\"tf-provider\":{\"source\":\"hashicorp/aws\",\"version\":\"3.56.0\"}}}}","start_line":1,"highlight_start_offset":858,"highlight_end_offset":859,"values":[]}},"type":"diagnostic"}

How can we reproduce it?

Trigger a problem and observe terraform logs.

[turkenh]

We have a similar problem with main.tf.json file in workspaces and prevents us from asking for workspace directory content for troubleshooting purposes.
As we discussed offline, passing credentials as env vars could be a solution for both cases but this wouldn't work if we want to move towards a shared provider approach. The other alternative we discussed, which is masking sensitive information in the logs would solve the issue with logs and events but not with workspace directory content.

Considering the severity of this issue, I am still wondering whether we should fix this with an env var solution asap, and consider finding an alternative one once/if we decide to use a shared provider approach.