add esg support for set rules

[camrossi]

No description provided.

[camrossi]

I just realised there is an issue in my code but I am not sure how to fix it. As with #293 APIC will not let you add a set rule to an import_route_control if the set rule is referencing a non-existing ESG/EPG and will throw this error Set policy tag to ESg not permitted to L3out

So in the code we need to have the update the import-rule after the ESGs/EPGs are created but I am not sure what is the best way to do so... I tried adding module.aci_endpoint_security_group, in the depends_on list of the module "aci_l3out" but that fails with Error: Cycle

I would need some guidance on how to do this.

[juchowan]

and simply adding a dependency on ESG for set rules module wouldn't help?

[camrossi]

@juchowan I tried and I get Error: Cycle as mentioned in my comment I am really not an expert in netascode so maybe I am missing something basic?

[juchowan]

but in the comment you mention you set depends_on in the l3out module, i'm talking about dependency between set_rules and esg modules. Is it causing the same error?

[camrossi]

I will try tomorrow (almost midnight for me) and will put here the result of the test I do!

[camrossi]

@juchowan

Let me first explain the issue a bit better:

APIC will block the configuration if the ESG does not exist yet
In my terrafrom plan am creating ESGs and the import-route maps at the same time and it fails with this error:

• Missing EGS:

module.aci.module.aci_l3out["terraform-pr-tests/terraform-pr-tests"].aci_rest_managed.rtctrlRsScopeToAttrP_import["ns1"]: Creating...
╷
│ Error: The post rest request failed
│ 
│   with module.aci.module.aci_l3out["terraform-pr-tests/terraform-pr-tests"].aci_rest_managed.rtctrlRsScopeToAttrP_import["ns1"],
│   on .terraform/modules/aci/modules/terraform-aci-l3out/main.tf line 96, in resource "aci_rest_managed" "rtctrlRsScopeToAttrP_import":
│   96: resource "aci_rest_managed" "rtctrlRsScopeToAttrP_import" {
│ 
│ Response Status Code: 400, Error Code: 1, Error Message: Invalid ESg (uni/tn-terraform-pr-tests/ap-cluster/esg-pods): Set policy tag to ESg not permitted to L3out
│ uni/tn-terraform-pr-tests/out-terraform-pr-tests.

The issue happens when you add the set_rule in the route-map context i.e.:

So the actual code in netascode that should wait for the ESG to exist is the l3out code specifically the below code:

resource "aci_rest_managed" "rtctrlCtxP" {
  for_each   = { for context in local.route_map_contexts : "${context.route_map_name}-${context.name}" => context }
  dn         = "${aci_rest_managed.rtctrlProfile[each.value.route_map_name].dn}/ctx-${each.value.name}"
  class_name = "rtctrlCtxP"
  content = {
    name   = each.value.name
    descr  = each.value.description
    action = each.value.action
    order  = each.value.order
  }
}
resource "aci_rest_managed" "rtctrlScope" {
  for_each   = { for context in local.route_map_contexts : "${context.route_map_name}-${context.name}" => context if context.set_rule != null && context.set_rule != "" }
  dn         = "${aci_rest_managed.rtctrlCtxP["${each.value.route_map_name}-${each.value.name}"].dn}/scp"
  class_name = "rtctrlScope"
}

resource "aci_rest_managed" "rtctrlRsScopeToAttrP" {
  for_each   = { for context in local.route_map_contexts : "${context.route_map_name}-${context.name}" => context if context.set_rule != null && context.set_rule != "" }
  dn         = "${aci_rest_managed.rtctrlScope["${each.value.route_map_name}-${each.value.name}"].dn}/rsScopeToAttrP"
  class_name = "rtctrlRsScopeToAttrP"
  content = {
    tnRtctrlAttrPName = each.value.set_rule
  }
}

But if I add a dependency in the aci_tenants.tf for the l3out module to wait for the ESG it fails

module "aci_l3out" {

  depends_on = [
    module.aci_tenant,
    module.aci_vrf,
    module.aci_ospf_interface_policy,
    module.aci_eigrp_interface_policy,
    module.aci_bfd_interface_policy,
    module.aci_set_rule,
    module.aci_match_rule,
    module.aci_endpoint_security_group,
  ]
}

as it creates a circular dependency and also is very inefficient to do so. What I think the solution is is unfortunately to take out from the L3out module the route-maps part and create a dedicated module that can depend on the l3out and the ESG creation but is a major breaking change and maybe there is a better way ?

[robvand]

The l3out module has a dependency on aci_set_rule so it creates a cycle.

rtctrlRsSetPolicyTagToESg validates the ESG exists and is in the same VRF as the L3out. This creates an implicit dependency on the ESG. The ESG might be created out of sequence or by a different plan so fixing this in the module is not feasible. To work around this issue you could create a new module for this feature with a dependency on ESG.

CSCwq86357 has been opened to address this.

[camrossi]

Thanks for confirming this @robvand it did feel was a complicated issue :)