[Feature Req] code-generator: needs to support custom `find_operation`

[nnbu]

Is your feature request related to a problem?
In case 'Find' api is not available for a resource, I see that code-generator returns return nil, ackerr.NotImplemented in sdkFind() function. There doesn't seem to be a way to write custom find function.

I see that there is custom function available for update method using update_operation tag in generator.yaml. But I dont see find_operation tag where we can specify custom function for find operation.

As an example, there is not find method available for https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateNetworkAclEntry.html
But I can write custom code using https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeNetworkAcls.html and extract network_acl_entries from network_acl.

But I don't see an easy way to add this custom find function

Describe the solution you'd like
The solution would allow us to implement custom find function. The name could be written in generator.yaml as find_operation: <name-of-function> similar to update_operation. And generated code for sdkFind() would call this custom find function. The find function itself could be written in hooks.go.

Describe alternatives you've considered
My first try was to use hooks (sdk_read_one_pre_build_request, sdk_read_many_pre_build_request, etc) to get around this problem. But when find function is not available, then find related hooks are not effective. Because code-generator takes templates/pkg/resource/sdk_find_not_implemented.go.tpl path (where hooks logic is not available), instead of templates/pkg/resource/sdk_find_read_many.go.tpl, where hooks logic is available.

[jaypipes]

@nnbu you need to put the following in your generator.yaml file:

operations:
  DescribeNetworkAcls:
    operation_type: READ_MANY
    resource_name: NetworkACL

and the code-generator should be able to figure things out from there.

[nnbu]

@jaypipes Thanks for reviewing.

However, there are 2 different set of apis for 2 resources networkAcl and networkAclEntry.
There is create, describe, delete apis for networkAcl.

But there is only create and delete apis for networkAclEntry. It does not have describe/find api.

The concern here is for supporting networkAclEntry resource. Of course, that will be done after supporting networkAcl resource, which should not have any problem.

The way to verify if networkAclEntry resource is existing or not, is to check describeNetworkAcl resource and iterate over networkAclEntry list inside it and figure out if the networkAclEntry is existing or not. This can not done using generated code of describeNetworkAcl api. That is why it needs a custom find function.

[jaypipes]

@nnbu OK, yeah, I think it would be best to only create a NetworkACL resource and have a Spec.Entries custom field on that NetworkACL resource that corresponds to the CreateNetworkAclEntry/DeleteNetworkAclEntry calls. This is how I did things for the IAM controller's Role, User and Group resources which each have a Spec.Policies field. This Spec.Policies field is defined in the generator.yaml file and populated with custom code hooks

[nnbu]

@jaypipes I see. Let me check this way of handling this using custom hooks.

Btw, irrespective of this case, do you think that we should have a way to write custom find function similar to custom update function?

[nnbu]

@jaypipes I reviewed your suggestions.

There are a few differences between IAM's group, policy groupPolicy related change and EC2's networkACL, networkACLEntry.

1. In case of IAM, group and policy are the actual resources, whereas groupPolicy provides association between group and policy. i.e. user creates group and policy independently. and then associates them together using groupPolicy. So, in this case, we are able to have group and policy in generator.yaml under resources section.

In case of networkAcl and networkAclEntry, networkAcl is purely a resource. But networkAclEntry is a resource as
well as association. i.e. We cant create networkAclEntry without having networkAcl. So, we must have networkAcl
and networkAclEntry in resources section in generator.yaml.

2. For all the resources in generator.yaml, there needs to be find/describe operation available. Otherwise IUC, find-not-implemented kicks in and thereby stalls reconciler's progress.

If this understanding is correct, then networkAclEntry needs to have find function available.

In IAM's case, in fact, we have find function available for all group, policy and groupPolicy. So, we did not face this issue. But networkAclEntry does not have find available. I feel that is the problem here. wdyt?

[As a side note, in IAM's case, policy has arn to uniquely identify it which you used in generator.yaml. But networkAclEntry does not have arn. This is secondary concern at this time because the main issue, IMO, is the need to have find function for resources defined in generator.yaml. ]