redfish_bios: attributes not applied on reboot

[dkaser]

Terraform CLI and Terraform Redfish Provider Version

Terraform 1.9.5
Redfish Provider Version 1.4.0

Server(s) details and firmware version

Dell R760

iDRAC 9
BIOS Version 2.2.8
Firmware Version 7.10.50.10

Affected Resource(s)

resource "redfish_bios"

Terraform Configuration Files

resource "redfish_bios" "bios" {
  for_each = var.rack1

  redfish_server {
    user         = each.value.user
    password     = each.value.password
    endpoint     = each.value.endpoint
    ssl_insecure = each.value.ssl_insecure
  }

  // Bios attributes to be altered
  attributes = {
    "SerialComm" = "Off"
  }

  /* Reset parameters to be applied after bios settings are applied
     list of possible value:
      [ ForceRestart, GracefulRestart, PowerCycle]
  */
  reset_type    = "ForceRestart"
  reset_timeout = "120"
  // The maximum amount of time to wait for the bios job to be completed
  bios_job_timeout = "1200"

  // by default, the resource uses the first system
  # system_id = "System.Embedded.1"
}

Expected Behavior

Attributes should be applied to BIOS.

Actual Behavior

The attributes appear as "pending" settings in the iDRAC web interface, and the server reboots, but the settings are not applied. A "BIOS.Setup" job does not appear in the job queue.

Resource then times out after bios_job_timeout.

Steps to Reproduce

1. terraform apply

[dkaser]

redfish_bios.bios["name"]: Creating...
2024-08-21T15:13:25.525-0400 [INFO]  Starting apply for redfish_bios.bios["name"]
2024-08-21T15:13:25.526-0400 [DEBUG] redfish_bios.bios["name"]: applying the planned Create change
2024-08-21T15:13:26.387-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:13:26 Connection with the redfish endpoint https://server.ip/ was sucessful
2024-08-21T15:13:26.387-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:13:26 [DEBUG] Locking https://server.ip/
2024-08-21T15:13:26.387-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:13:26 [DEBUG] Locked https://server.ip/
2024-08-21T15:13:27.009-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: resetTimeout is set to 120 and Bios Config Job timeout is set to 1200: @module=redfish tf_resource_type=redfish_bios tf_rpc=ApplyResourceChange @caller=terraform-provider-redfish/redfish/provider/resource_redfish_bios.go:353 tf_provider_addr=registry.terraform.io/dell/redfish tf_req_id=2fa3996b-c749-fbba-de84-f1da37b2cb86 timestamp=2024-08-21T15:13:27.008-0400
2024-08-21T15:13:27.009-0400 [INFO]  provider.terraform-provider-redfish_v1.4.0: Submitting patch request for bios attributes: @caller=terraform-provider-redfish/redfish/provider/resource_redfish_bios.go:357 @module=redfish tf_provider_addr=registry.terraform.io/dell/redfish tf_req_id=2fa3996b-c749-fbba-de84-f1da37b2cb86 tf_resource_type=redfish_bios tf_rpc=ApplyResourceChange timestamp=2024-08-21T15:13:27.008-0400
2024-08-21T15:13:29.794-0400 [INFO]  provider.terraform-provider-redfish_v1.4.0: Submitting patch request for bios attributes completed successfully: tf_provider_addr=registry.terraform.io/dell/redfish tf_resource_type=redfish_bios tf_rpc=ApplyResourceChange @caller=terraform-provider-redfish/redfish/provider/resource_redfish_bios.go:364 @module=redfish tf_req_id=2fa3996b-c749-fbba-de84-f1da37b2cb86 timestamp=2024-08-21T15:13:29.793-0400
2024-08-21T15:13:29.794-0400 [INFO]  provider.terraform-provider-redfish_v1.4.0: rebooting the server: @caller=terraform-provider-redfish/redfish/provider/resource_redfish_bios.go:365 tf_req_id=2fa3996b-c749-fbba-de84-f1da37b2cb86 tf_resource_type=redfish_bios @module=redfish tf_provider_addr=registry.terraform.io/dell/redfish tf_rpc=ApplyResourceChange timestamp=2024-08-21T15:13:29.794-0400
redfish_bios.bios["name"]: Still creating... [10s elapsed]
redfish_bios.bios["name"]: Still creating... [20s elapsed]
redfish_bios.bios["name"]: Still creating... [30s elapsed]
redfish_bios.bios["name"]: Still creating... [40s elapsed]
2024-08-21T15:14:09.997-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: system.Reset successful: @caller=terraform-provider-redfish/redfish/provider/common.go:306 @module=redfish tf_provider_addr=registry.terraform.io/dell/redfish tf_rpc=ApplyResourceChange tf_resource_type=redfish_bios tf_req_id=2fa3996b-c749-fbba-de84-f1da37b2cb86 timestamp=2024-08-21T15:14:09.996-0400
2024-08-21T15:14:09.997-0400 [INFO]  provider.terraform-provider-redfish_v1.4.0: rebooting the server completed successfully: tf_req_id=2fa3996b-c749-fbba-de84-f1da37b2cb86 tf_rpc=ApplyResourceChange @caller=terraform-provider-redfish/redfish/provider/resource_redfish_bios.go:375 tf_provider_addr=registry.terraform.io/dell/redfish tf_resource_type=redfish_bios @module=redfish timestamp=2024-08-21T15:14:09.996-0400
2024-08-21T15:14:09.997-0400 [INFO]  provider.terraform-provider-redfish_v1.4.0: Waiting for the bios config job to finish: tf_provider_addr=registry.terraform.io/dell/redfish tf_req_id=2fa3996b-c749-fbba-de84-f1da37b2cb86 tf_resource_type=redfish_bios @caller=terraform-provider-redfish/redfish/provider/resource_redfish_bios.go:376 @module=redfish tf_rpc=ApplyResourceChange timestamp=2024-08-21T15:14:09.996-0400
redfish_bios.bios["name"]: Still creating... [50s elapsed]
2024-08-21T15:14:20.158-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:14:20 [DEBUG] - Attempting one more time... Job state is
redfish_bios.bios["name"]: Still creating... [1m0s elapsed]
2024-08-21T15:14:30.213-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:14:30 [DEBUG] - Attempting one more time... Job state is
redfish_bios.bios["name"]: Still creating... [1m10s elapsed]
2024-08-21T15:14:40.223-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:14:40 [DEBUG] - Attempting one more time... Job state is
redfish_bios.bios["name"]: Still creating... [1m20s elapsed]
2024-08-21T15:14:50.165-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:14:50 [DEBUG] - Attempting one more time... Job state is
redfish_bios.bios["name"]: Still creating... [1m30s elapsed]
2024-08-21T15:15:00.127-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:15:00 [DEBUG] - Attempting one more time... Job state is
redfish_bios.bios["name"]: Still creating... [1m40s elapsed]
2024-08-21T15:15:10.217-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/21 15:15:10 [DEBUG] - Attempting one more time... Job state is

[gokul-srivathsan]

Thanks @dkaser for raising this , will get back to you soon.

[gokul-srivathsan]

@dkaser , did you get any job timeout error at the end of the log?

[dkaser]

@gokul-srivathsan Yes, I'll get a copy of the message shortly.

[dkaser]

redfish_bios.bios["Name"]: Still creating... [20m40s elapsed]
2024-08-22T13:49:50.185-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/22 13:49:50 [DEBUG] - Attempting one more time... Job state is
2024-08-22T13:49:50.185-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/22 13:49:50 [DEBUG] - Error. Timeout reached
2024-08-22T13:49:50.185-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/22 13:49:50 [DEBUG] Unlocking https://server.ip/
2024-08-22T13:49:50.185-0400 [DEBUG] provider.terraform-provider-redfish_v1.4.0: 2024/08/22 13:49:50 [DEBUG] Unlocked https://server.ip/
2024-08-22T13:49:50.411-0400 [ERROR] provider.terraform-provider-redfish_v1.4.0: Response contains error diagnostic: diagnostic_detail="timeout waiting for the job to finish" diagnostic_severity=ERROR diagnostic_summary="error waiting for Bios config monitor task to be completed" tf_proto_version=6.4 @caller=github.com/hashicorp/terraform-plugin-go@v0.19.1/tfprotov6/internal/diag/diagnostics.go:58 tf_provider_addr=registry.terraform.io/dell/redfish tf_resource_type=redfish_bios @module=sdk.proto tf_req_id=468ceb74-1f4d-a648-0077-6293db157955 tf_rpc=ApplyResourceChange timestamp=2024-08-22T13:49:50.410-0400
2024-08-22T13:49:50.470-0400 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2024-08-22T13:49:50.471-0400 [ERROR] vertex "redfish_bios.bios[\"Name\"]" error: error waiting for Bios config monitor task to be completed
╷
│ Error: error waiting for Bios config monitor task to be completed
│
│   with redfish_bios.bios["Name"],
│   on main.tf line 38, in resource "redfish_bios" "bios":
│   38: resource "redfish_bios" "bios" {
│
│ timeout waiting for the job to finish
╵
2024-08-22T13:49:50.508-0400 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-08-22T13:49:50.512-0400 [INFO]  provider: plugin process exited: plugin=.terraform/providers/registry.terraform.io/dell/redfish/1.4.0/linux_amd64/terraform-provider-redfish_v1.4.0 id=565800
2024-08-22T13:49:50.512-0400 [DEBUG] provider: plugin exited

[dkaser]

I was able to pin down a more specific cause of this failure after I did some testing using the same attributes, but using Ansible instead of Terraform.

I had an invalid value for one of the attributes, which the Ansible task flagged as "Invalid value for enumeration". Fixing this attribute allows the Terraform provider to run successfully, including creating the Configure BIOS task.

I think the root problem is that the provider can't detect this failure (either by examining the Redfish response, or by independently validating the attributes like how the Ansible logic does). There was nothing in the Terraform output to suggest that anything was wrong with the provided attributes, and the unproductive restarts add extra confusion :(

[anupamaloke]

@dkaser, thank you for providing this information. This is very helpful.
You are right. As opposed to Ansible modules, Terraform BIOS resource does not check the attribute-value keypair against the BIOS message registry. All Bios configuration is an async operation that requires a server restart. A Bios configuration request is queued up in Job queue which requires a server reboot to be started. In case the job fails due to a incorrect attribute value, then, unfortunately, this is not recorded in the Job response. So, ideally, the provider should check agaist message registry and flag the incorrect values before submitting a job to iDRAC similar to Ansible modules. This is an enhancement that I have added to the backlog to be picked up in the next release.

Meanwhile, could you please confirm whether the terraform apply command succeeded once you entered the correct value for the Bios resource?

[gokul-srivathsan]

@dkaser your input will be much appreciated.

[dkaser]

The apply did work successfully.