Elasticsearch domain: ValidationException for instance type

[nirsv]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Affected Resource(s)

• aws_elasticsearch_domain

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Debug Output

Panic Output

Expected Behavior

Create a cluster with instance type of i3.2xlarge.search (which is the type when you manually create an amazon opensearch cluster)

Actual Behavior

When I choose instance type of type i3.2xlarge.search (which is the type when you manually create an amazon opensearch cluster) I get the following error:
rror creating Elasticsearch domain: ValidationException: 2 validation errors detected: Value 'i3.2xlarge.search' at 'elasticsearchClusterConfig.instanceType' failed to satisfy constraint: Member must satisfy enum value set: [r6gd.12xlarge.elasticsearch, i3.2xlarge.elasticsearch, ultrawarm1.xlarge.elasticsearch, m5.4xlarge.elasticsearch, t3.xlarge.elasticsearch, m6g.xlarge.elasticsearch, i3.4xlarge.elasticsearch, m3.large.elasticsearch, m6g.12xlarge.elasticsearch, r4.16xlarge.elasticsearch, t2.micro.elasticsearch, m4.large.elasticsearch, r6gd.16xlarge.elasticsearch, d2.2xlarge.elasticsearch, t3.micro.elasticsearch, m5.large.elasticsearch, i3.8xlarge.elasticsearch, i3.large.elasticsearch, d2.4xlarge.elasticsearch, t2.small.elasticsearch, c4.2xlarge.elasticsearch, t3.small.elasticsearch, c5.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c6g.2xlarge.elasticsearch, d2.8xlarge.elasticsearch, c5.4xlarge.elasticsearch, t4g.medium.elasticsearch, c6g.4xlarge.elasticsearch, c6g.xlarge.elasticsearch, m3.medium.elasticsearch, c6g.12xlarge.elasticsearch, c4.8xlarge.elasticsearch, c4.large.elasticsearch, c5.xlarge.elasticsearch, c5.large.elasticsearch, t4g.small.elasticsearch, c4.xlarge.elasticsearch, c5.9xlarge.elasticsearch, c6g.8xlarge.elasticsearch, c6g.large.elasticsearch, d2.xlarge.elasticsearch, ultrawarm1.medium.elasticsearch, t3.nano.elasticsearch, t3.medium.elasticsearch, m6g.2xlarge.elasticsearch, t2.medium.elasticsearch, t3.2xlarge.elasticsearch, c5.18xlarge.elasticsearch, m6g.4xlarge.elasticsearch, i3.xlarge.elasticsearch, r6gd.2xlarge.elasticsearch, i2.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r4.2xlarge.elasticsearch, m5.xlarge.elasticsearch, m4.10xlarge.elasticsearch, r6gd.4xlarge.elasticsearch, r6g.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r5.2xlarge.elasticsearch, m5.12xlarge.elasticsearch, m4.xlarge.elasticsearch, m6g.8xlarge.elasticsearch, m6g.large.elasticsearch, r4.4xlarge.elasticsearch, m5.24xlarge.elasticsearch, m3.xlarge.elasticsearch, r6g.4xlarge.elasticsearch, i3.16xlarge.elasticsearch, t3.large.elasticsearch, r5.4xlarge.elasticsearch, ultrawarm1.large.elasticsearch, m3.2xlarge.elasticsearch, r6gd.8xlarge.elasticsearch, r6gd.large.elasticsearch, r6g.xlarge.elasticsearch, r3.8xlarge.elasticsearch, r3.large.elasticsearch, r5.xlarge.elasticsearch, m4.2xlarge.elasticsearch, r6g.12xlarge.elasticsearch, r4.8xlarge.elasticsearch, r4.xlarge.elasticsearch, r4.large.elasticsearch, r5.12xlarge.elasticsearch, m5.2xlarge.elasticsearch, r6gd.xlarge.elasticsearch, r6g.8xlarge.elasticsearch, r6g.large.elasticsearch, i2.2xlarge.elasticsearch, r3.xlarge.elasticsearch, r5.24xlarge.elasticsearch, r5.large.elasticsearch, m4.4xlarge.elasticsearch]; Value 'm5.large.search' at 'elasticsearchClusterConfig.dedicatedMasterType' failed to satisfy constraint: Member must satisfy enum value set: [r6gd.12xlarge.elasticsearch, i3.2xlarge.elasticsearch, ultrawarm1.xlarge.elasticsearch, m5.4xlarge.elasticsearch, t3.xlarge.elasticsearch, m6g.xlarge.elasticsearch, i3.4xlarge.elasticsearch, m3.large.elasticsearch, m6g.12xlarge.elasticsearch, r4.16xlarge.elasticsearch, t2.micro.elasticsearch, m4.large.elasticsearch, r6gd.16xlarge.elasticsearch, d2.2xlarge.elasticsearch, t3.micro.elasticsearch, m5.large.elasticsearch, i3.8xlarge.elasticsearch, i3.large.elasticsearch, d2.4xlarge.elasticsearch, t2.small.elasticsearch, c4.2xlarge.elasticsearch, t3.small.elasticsearch, c5.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c6g.2xlarge.elasticsearch, d2.8xlarge.elasticsearch, c5.4xlarge.elasticsearch, t4g.medium.elasticsearch, c6g.4xlarge.elasticsearch, c6g.xlarge.elasticsearch, m3.medium.elasticsearch, c6g.12xlarge.elasticsearch, c4.8xlarge.elasticsearch, c4.large.elasticsearch, c5.xlarge.elasticsearch, c5.large.elasticsearch, t4g.small.elasticsearch, c4.xlarge.elasticsearch, c5.9xlarge.elasticsearch, c6g.8xlarge.elasticsearch, c6g.large.elasticsearch, d2.xlarge.elasticsearch, ultrawarm1.medium.elasticsearch, t3.nano.elasticsearch, t3.medium.elasticsearch, m6g.2xlarge.elasticsearch, t2.medium.elasticsearch, t3.2xlarge.elasticsearch, c5.18xlarge.elasticsearch, m6g.4xlarge.elasticsearch, i3.xlarge.elasticsearch, r6gd.2xlarge.elasticsearch, i2.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r4.2xlarge.elasticsearch, m5.xlarge.elasticsearch, m4.10xlarge.elasticsearch, r6gd.4xlarge.elasticsearch, r6g.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r5.2xlarge.elasticsearch, m5.12xlarge.elasticsearch, m4.xlarge.elasticsearch, m6g.8xlarge.elasticsearch, m6g.large.elasticsearch, r4.4xlarge.elasticsearch, m5.24xlarge.elasticsearch, m3.xlarge.elasticsearch, r6g.4xlarge.elasticsearch, i3.16xlarge.elasticsearch, t3.large.elasticsearch, r5.4xlarge.elasticsearch, ultrawarm1.large.elasticsearch, m3.2xlarge.elasticsearch, r6gd.8xlarge.elasticsearch, r6gd.large.elasticsearch, r6g.xlarge.elasticsearch, r3.8xlarge.elasticsearch, r3.large.elasticsearch, r5.xlarge.elasticsearch, m4.2xlarge.elasticsearch, r6g.12xlarge.elasticsearch, r4.8xlarge.elasticsearch, r4.xlarge.elasticsearch, r4.large.elasticsearch, r5.12xlarge.elasticsearch, m5.2xlarge.elasticsearch, r6gd.xlarge.elasticsearch, r6g.8xlarge.elasticsearch, r6g.large.elasticsearch, i2.2xlarge.elasticsearch, r3.xlarge.elasticsearch, r5.24xlarge.elasticsearch, r5.large.elasticsearch, m4.4xlarge.elasticsearch]

Steps to Reproduce

1. set instance type of i3.2xlarge.search
2. terraform apply

Important Factoids

The enum should include the type i3.2xlarge.search rather than i3.2xlarge.elasticsearch

References

• #0000

[ivoros]

The same goes to t3.medium.search. Terraform is clearly not prepared for OpenSearch yet.

Error: Error creating Elasticsearch domain: ValidationException: 1 validation error detected: Value 't3.medium.search' at 'elasticsearchClusterConfig.instanceType' failed to satisfy constraint: Member must satisfy enum value set: ....

[tonymayflower]

please, can we have update on the issue please ?

[ghdiska]

opensearch is a separate service in aws-sdk-go ( service opensearchservice ) but terraform use elastictsearch service only

[nirsv]

AWS managed sercice is only opensearch.
Terraform must support it.
Yiu cannot use aws_elasticsearch_domain without actually create opensearch domain

[snazzyfox]

Even though opensearch has separate APIs and shows up as a separate service in the SDK, it is a full replacement of elasticsearch. The existing elasticsearch APIs appear to simply translate requests to opensearch instead.

• Using a new instance type name (e.g. m6g.large.search) causes an error in terraform as shown in the issue.
• Using an old instance type name (e.g. m6g.large.elasticsearch) passes terraform's validation, but the SDK throws InvalidTypeException: Invalid instance type: m6g.large.elasticsearch since it's expecting the new instance type names instead.

This completely breaks the aws_elasticsearch_domain resource in terraform as it's impossible to create new domains or update any existing domains (created before aws's rebranding).

[raffek]

Is there any update on this? Any workaround recommendation?

[hannes-ucsc]

I could swear the old .elasticsearch types worked four days ago. As of today they stopped working. I get

InvalidTypeException: Invalid instance type: r6gd.xlarge.elasticsearch

during terraform apply. plan and validate pass.

$ terraform version
Terraform v0.12.24
+ provider.aws v2.70.0
+ provider.google v2.20.3
+ provider.null v2.1.2
+ provider.template v2.2.0

[hannes-ucsc]

It turns out that creating a domain with cluster_config.instance_type set to an .elasticsearch type still works if the elasticsearch_version is set to 7.10. Updating an existing domain whose elasticsearch_version is 6.8 fails, even if the update attempts to set elasticsearch_version to 7.10 while also setting cluster_config.instance_type to say r6gd.xlarge.elasticsearch. Updating cluster_config.instance_type on an existing domain whose elasticsearch_version is 7.10 also works.

This might just be a bug in AWS.

[sobi3ch]

It turns out that creating a domain with cluster_config.instance_type set to an .elasticsearch type still works if the elasticsearch_version is set to 7.10. Updating an existing domain whose elasticsearch_version is 6.8 fails, even if the update attempts to set elasticsearch_version to 7.10 while also setting cluster_config.instance_type to say r6gd.xlarge.elasticsearch. Updating cluster_config.instance_type on an existing domain whose elasticsearch_version is 7.10 also works.

This might just be a bug in AWS.

Confirming, with version 7.10 it works (accidentally has set to 7.1) and extension must be *.elasticsearch.

[zatr]

It turns out that creating a domain with cluster_config.instance_type set to an .elasticsearch type still works if the elasticsearch_version is set to 7.10. Updating an existing domain whose elasticsearch_version is 6.8 fails, even if the update attempts to set elasticsearch_version to 7.10 while also setting cluster_config.instance_type to say r6gd.xlarge.elasticsearch. Updating cluster_config.instance_type on an existing domain whose elasticsearch_version is 7.10 also works.

This might just be a bug in AWS.

This is a workaround using an old version of Elasticsearch. This issue was submitted because Terraform cannot be used to create an OpenSearch domain.