resource(vpc_firewall_rules): provider produced invalid plan

[sudomateo]

A customer reported that the following Terraform configuration resulted in an error when running Terraform v1.5.0 and provider version v0.10.0.

resource "oxide_vpc_firewall_rules" "default_rules" {
  vpc_id = data.oxide_vpc.default_vpc.id
  rules = toset([
    {
      action      = "allow"
      description = "Allow HTTPS."
      name        = "allow-https"
      direction   = "inbound"
      priority    = 50
      status      = "enabled"
      filters = {
        hosts = [
          {
            type  = "vpc"
            value = "default"
          }
        ]
        ports     = ["443"]
        protocols = ["TCP"]
      },
      targets = [
        {
          type  = "subnet"
          value = "default"
        }
      ]
    },
    {
      action      = "allow"
      description = "Allow SSH."
      name        = "allow-ssh"
      direction   = "inbound"
      priority    = 50
      status      = "enabled"
      filters = {
        hosts = [
          {
            type  = "vpc"
            value = "default"
          }
        ]
        ports     = ["22"]
        protocols = ["TCP"]
      },
      targets = [
        {
          type  = "subnet"
          value = "default"
        }
      ]
    }
  ])
}

Error: Provider produced invalid plan

Provider "registry.terraform.io/oxidecomputer/oxide" planned an invalid value
for
module.oxide_host["dc15-testl1"].oxide_vpc_firewall_rules.default_rules.rules:
count in plan (cty.UnknownVal(cty.Number)) disagrees with count in config
(cty.NumberIntVal(2)).

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

[sudomateo]

I tested the following configuration on the provider built from main and Terraform v1.12.2.

resource "oxide_vpc" "example" {
  project_id  = "5476ccc9-464d-4dc4-bfc0-5154de1c986f"
  description = "Example VPC."
  name        = "example"
  dns_name    = "example"
}

resource "oxide_vpc_firewall_rules" "example" {
  vpc_id = oxide_vpc.example.id
  rules = toset([
    {
      action      = "allow"
      description = "Allow HTTPS."
      name        = "allow-https"
      direction   = "inbound"
      priority    = 50
      status      = "enabled"
      filters = {
        hosts = [
          {
            type  = "vpc"
            value = oxide_vpc.example.name
          }
        ]
        ports     = ["443"]
        protocols = ["TCP"]
      },
      targets = [
        {
          type  = "subnet"
          value = oxide_vpc.example.name
        }
      ]
    },
    {
      action      = "allow"
      description = "Allow SSH."
      name        = "allow-ssh"
      direction   = "inbound"
      priority    = 50
      status      = "enabled"
      filters = {
        hosts = [
          {
            type  = "vpc"
            value = oxide_vpc.example.name
          }
        ]
        ports     = ["22"]
        protocols = ["TCP"]
      },
      targets = [
        {
          type  = "subnet"
          value = oxide_vpc.example.name
        }
      ]
    }
  ])
}

The create, read, destroy flow works fine.

The create, read, update flow logs the following error. The update was commenting out the second element in rules.

oxide_vpc_firewall_rules.example: Modifying... [id=c75d1e61-9737-4abd-be16-8837920e2344]
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to oxide_vpc_firewall_rules.example, provider "provider[\"registry.terraform.io/oxidecomputer/oxide\"]" produced an unexpected new value: .rules:
│ planned set element cty.ObjectVal(map[string]cty.Value{"action":cty.StringVal("allow"), "description":cty.StringVal("Allow HTTPS."), "direction":cty.StringVal("inbound"),
│ "filters":cty.ObjectVal(map[string]cty.Value{"hosts":cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"type":cty.StringVal("vpc"),
│ "value":cty.StringVal("example")})}), "ports":cty.SetVal([]cty.Value{cty.StringVal("443")}), "protocols":cty.SetVal([]cty.Value{cty.StringVal("TCP")})}),
│ "id":cty.StringVal("1ded3e80-2af4-4caa-9a7c-156c444f2000"), "name":cty.StringVal("allow-https"), "priority":cty.NumberIntVal(50), "status":cty.StringVal("enabled"),
│ "targets":cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"type":cty.StringVal("subnet"), "value":cty.StringVal("example")})}),
│ "time_created":cty.StringVal("2025-06-20 13:04:11.352229 +0000 UTC"), "time_modified":cty.StringVal("2025-06-20 13:04:11.352229 +0000 UTC")}) does not correlate with any
│ element in actual.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

[sudomateo]

I spent some time understanding what's happening here so I wanted to write it up for others.

In #432 we removed the requires replace directive from the vpc_firewall_rules resource to support in-place modification. The intent was to prevent unnecessary network connectivity issues while VPC firewall rules are deleted and created anew (e.g., when the delete succeeds and the create fails). With this change, Update is now called rather than Delete followed by Create.

However, this change was also a bit of a workaround since the Oxide API does not support in-place updates of VPC firewall rules, at least not in the manner than Terraform expects. Instead, when you replace VPC firewall rules using the Oxide API each rule will be given a unique ID.

> curl --silent --request PUT --header "Authorization: Bearer $OXIDE_TOKEN" --header "Content-Type: application/json" $OXIDE_HOST/v1/vpc-firewall-rules?project=matthewsanabria&vpc=example --data @/tmp/rules.json
{"rules":[{"id":"0852c8c8-e1d4-4dd4-84a7-19946eae0313","name":"allow-https","description":"Allow HTTPS.","time_created":"2025-06-20T19:40:22.514499Z","time_modified":"2025-06-20T19:40:22.514499Z","status":"enabled","direction":"inbound","targets":[{"type":"subnet","value":"example"}],"filters":{"hosts":[{"type":"vpc","value":"example"}],"protocols":["TCP"],"ports":["443"]},"action":"allow","priority":50,"vpc_id":"b7b7b444-1c64-4cb5-82d0-f1e4404a100c"},{"id":"bac0f44f-1835-4943-9944-08e6c328bbea","name":"allow-ssh","description":"Allow SSH (new).","time_created":"2025-06-20T19:40:22.514501Z","time_modified":"2025-06-20T19:40:22.514501Z","status":"enabled","direction":"inbound","targets":[{"type":"subnet","value":"example"}],"filters":{"hosts":[{"type":"vpc","value":"example"}],"protocols":["TCP"],"ports":["22"]},"action":"allow","priority":50,"vpc_id":"b7b7b444-1c64-4cb5-82d0-f1e4404a100c"}]}

> curl --silent --request GET --header "Authorization: Bearer $OXIDE_TOKEN" --header "Content-Type: application/json" $OXIDE_HOST/v1/vpc-firewall-rules?project=matthewsanabria&vpc=example | jq -r '.rules[] | pick(.id, .name)'
{
  "id": "0852c8c8-e1d4-4dd4-84a7-19946eae0313",
  "name": "allow-https"
}
{
  "id": "bac0f44f-1835-4943-9944-08e6c328bbea",
  "name": "allow-ssh"
}

Even if you issue another PUT request with the exact same request body Oxide will still generate new IDs for the rules.

> curl --silent --request PUT --header "Authorization: Bearer $OXIDE_TOKEN" --header "Content-Type: application/json" $OXIDE_HOST/v1/vpc-firewall-rules?project=matthewsanabria&vpc=example --data @/tmp/rules.json
{"rules":[{"id":"74658b3c-f3f9-4e36-985c-f9259c81c98a","name":"allow-https","description":"Allow HTTPS.","time_created":"2025-06-20T19:40:47.171866Z","time_modified":"2025-06-20T19:40:47.171866Z","status":"enabled","direction":"inbound","targets":[{"type":"subnet","value":"example"}],"filters":{"hosts":[{"type":"vpc","value":"example"}],"protocols":["TCP"],"ports":["443"]},"action":"allow","priority":50,"vpc_id":"b7b7b444-1c64-4cb5-82d0-f1e4404a100c"},{"id":"91b353f1-8025-4c89-9d28-760c2c5719dd","name":"allow-ssh","description":"Allow SSH (new).","time_created":"2025-06-20T19:40:47.171868Z","time_modified":"2025-06-20T19:40:47.171868Z","status":"enabled","direction":"inbound","targets":[{"type":"subnet","value":"example"}],"filters":{"hosts":[{"type":"vpc","value":"example"}],"protocols":["TCP"],"ports":["22"]},"action":"allow","priority":50,"vpc_id":"b7b7b444-1c64-4cb5-82d0-f1e4404a100c"}]}

> curl --silent --request GET --header "Authorization: Bearer $OXIDE_TOKEN" --header "Content-Type: application/json" $OXIDE_HOST/v1/vpc-firewall-rules?project=matthewsanabria&vpc=example | jq -r '.rules[] | pick(.id, .name)'
{
  "id": "74658b3c-f3f9-4e36-985c-f9259c81c98a",
  "name": "allow-https"
}
{
  "id": "91b353f1-8025-4c89-9d28-760c2c5719dd",
  "name": "allow-ssh"
}

Let's walk through why this is a problem for Terraform.

I used Terraform to create two VPC firewall rules as shown below.

> jq -r '.resources[1].instances[].attributes.rules[] | pick(.id, .name)' terraform.tfstate
{
  "id": "12bbb9ca-3360-4f9f-ad5a-ada3cbb736c4",
  "name": "allow-https"
}
{
  "id": "5d83771f-76fd-421f-a785-3726b6310389",
  "name": "allow-ssh"
}

I then updated the port for the allow-ssh rule from 22 to 23. The Terraform plan shows that the resource will be updated in place by deleting the allow-ssh rule with ID 5d83771f-76fd-421f-a785-3726b6310389 and creating a new one. It will not touch the allow-https rule at all.

> terraform apply
oxide_vpc.example: Refreshing state... [id=b7b7b444-1c64-4cb5-82d0-f1e4404a100c]
oxide_vpc_firewall_rules.example: Refreshing state... [id=c75d1e61-9737-4abd-be16-8837920e2344]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # oxide_vpc_firewall_rules.example will be updated in-place
  ~ resource "oxide_vpc_firewall_rules" "example" {
      ~ id     = "c75d1e61-9737-4abd-be16-8837920e2344" -> (known after apply)
      ~ rules  = [
          - {
              - action        = "allow" -> null
              - description   = "Allow SSH." -> null
              - direction     = "inbound" -> null
              - filters       = {
                  - hosts     = [
                      - {
                          - type  = "vpc" -> null
                          - value = "example" -> null
                        },
                    ] -> null
                  - ports     = [
                      - "22",
                    ] -> null
                  - protocols = [
                      - "TCP",
                    ] -> null
                } -> null
              - id            = "5d83771f-76fd-421f-a785-3726b6310389" -> null
              - name          = "allow-ssh" -> null
              - priority      = 50 -> null
              - status        = "enabled" -> null
              - targets       = [
                  - {
                      - type  = "subnet" -> null
                      - value = "example" -> null
                    },
                ] -> null
              - time_created  = "2025-06-20 20:30:10.992084 +0000 UTC" -> null
              - time_modified = "2025-06-20 20:30:10.992084 +0000 UTC" -> null
            },
          + {
              + action        = "allow"
              + description   = "Allow SSH."
              + direction     = "inbound"
              + filters       = {
                  + hosts     = [
                      + {
                          + type  = "vpc"
                          + value = "example"
                        },
                    ]
                  + ports     = [
                      + "23",
                    ]
                  + protocols = [
                      + "TCP",
                    ]
                }
              + id            = (known after apply)
              + name          = "allow-ssh"
              + priority      = 50
              + status        = "enabled"
              + targets       = [
                  + {
                      + type  = "subnet"
                      + value = "example"
                    },
                ]
              + time_created  = (known after apply)
              + time_modified = (known after apply)
            },
            # (1 unchanged element hidden)
        ]
        # (1 unchanged attribute hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Applying this plan causes the error noted in this issue. Looking closer at the error text we see Terraform is referring to the allow-https rule with ID 12bbb9ca-3360-4f9f-ad5a-ada3cbb736c4, not the allow-ssh rule.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

oxide_vpc_firewall_rules.example: Modifying... [id=c75d1e61-9737-4abd-be16-8837920e2344]
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to oxide_vpc_firewall_rules.example, provider "provider[\"registry.terraform.io/oxidecomputer/oxide\"]"
│ produced an unexpected new value: .rules: planned set element
│ cty.ObjectVal(map[string]cty.Value{"action":cty.StringVal("allow"), "description":cty.StringVal("Allow HTTPS."),
│ "direction":cty.StringVal("inbound"),
│ "filters":cty.ObjectVal(map[string]cty.Value{"hosts":cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"type":cty.StringVal("vpc"),
│ "value":cty.StringVal("example")})}), "ports":cty.SetVal([]cty.Value{cty.StringVal("443")}),
│ "protocols":cty.SetVal([]cty.Value{cty.StringVal("TCP")})}), "id":cty.StringVal("12bbb9ca-3360-4f9f-ad5a-ada3cbb736c4"),
│ "name":cty.StringVal("allow-https"), "priority":cty.NumberIntVal(50), "status":cty.StringVal("enabled"),
│ "targets":cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"type":cty.StringVal("subnet"),
│ "value":cty.StringVal("example")})}), "time_created":cty.StringVal("2025-06-20 20:07:00.207241 +0000 UTC"),
│ "time_modified":cty.StringVal("2025-06-20 20:07:00.207241 +0000 UTC")}) does not correlate with any element in actual.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Why does Terraform care about the allow-https rule here at all if the plan was never meant to touch it?

It's because before the plan Terraform had the following in its state.

> jq -r '.resources[1].instances[].attributes.rules[] | pick(.id, .name)' terraform.tfstate
{
  "id": "12bbb9ca-3360-4f9f-ad5a-ada3cbb736c4",
  "name": "allow-https"
}
{
  "id": "9c723c19-16ce-489e-9c6d-23daf2ebeb1c",
  "name": "allow-ssh"
}

Terraform expected after the plan was applied the state would look like this. The allow-https rule would remain untouched and the allow-ssh rule would be completely new with a new ID.

> jq -r '.resources[1].instances[].attributes.rules[] | pick(.id, .name)' terraform.tfstate
{
  "id": "12bbb9ca-3360-4f9f-ad5a-ada3cbb736c4",
  "name": "allow-https"
}
{
  "id": "935e19c0-9036-419b-b615-d68665aed048",
  "name": "allow-ssh"
}

Instead, the Oxide API generated new IDs for all the rules, so Terraform got back the following state.

> jq -r '.resources[1].instances[].attributes.rules[] | pick(.id, .name)' terraform.tfstate
{
  "id": "32f2e026-f258-4512-bd68-79b613a8b5de",
  "name": "allow-https"
}
{
  "id": "935e19c0-9036-419b-b615-d68665aed048",
  "name": "allow-ssh"
}

Now Terraform was left wondering what happened to the original allow-https rule with ID 12bbb9ca-3360-4f9f-ad5a-ada3cbb736c4. Since Terraform couldn't find that original rule from the plan in the new state it returned an error.

[sudomateo]

Outside of adding the requires replace directive back I'm not sure what to do to fix this. At first I though moving to schema.ListNestedAttribute would work but even though it fixes this issue it creates a new one where elements being reordered ruins the plan. I'll think about it some more next week.

[karencfv]

Instead, the Oxide API generated new IDs for all the rules, so Terraform got back the following state.

Ugh, of course. Because it's recreating them all.

I think there are two possible courses of action here.

1. We put back the requires replace directive. This has the drawback of potentially losing connectivity if something goes wrong when applying the terraform plan. Additionally, We are taking an additional step of deleting all rules which isn't great either. The positive side is that the user has transparency in what is going on.
2. We don't record each individual rule's id in the state. It would be nice to have them, but practically speaking there is no use for them today. You can't target a rule individually through the API so there's really no use for those IDs for now. The drawback here is that it won't be obvious to the user that we're not "updating" the rule set in the strict sense, and we're recreating all rules.

I think I'm leaning towards solution 2. It'll be a smoother workflow. We should update the documentation to explain how the API is handling this flow, so our customers have transparency on how this all works.

[sudomateo]

Thank you for the rubber ducking! I like 2 as well. I'll code it up and see if works as expected.