Extract references in a single place

[muvaf]

What problem are you facing?

With #27 , we're inferring the references by traversing through the HCL code and store them in the provider metadata YAML in the following format together with the example they're inferred from:

examples:
    - manifest: |-
        {
          "api_management_name": "${data.azurerm_api_management_api.example.api_management_name}",
          "api_name": "${data.azurerm_api_management_api.example.name}",
          "description": "This can only be done by the logged in user.",
          "display_name": "Delete User Operation",
          "method": "DELETE",
          "operation_id": "user-delete",
          "resource_group_name": "${data.azurerm_api_management_api.example.resource_group_name}",
          "response": [
            {
              "status_code": 200
            }
          ],
          "url_template": "/users/{id}/delete"
        }
      references:
        api_management_name: data.api_management_name
        api_name: data.name
        resource_group_name: data.resource_group_name

Then in #12 we try to resolve the references from other examples of the resource and if that fails, we add a config.Reference entry, which allows us to generate references in the later stages.

How could Terrajet help solve your problem?

This two-steps approach does not cause any immediate problems so we don't need to change it at the moment. However, there might be room for improvement here by keeping the whole reference stuff in one place. We could possibly have the scraper generate only the examples and then do the traversal of those examples together with resolving and adding config.Reference entries. All those operations could be a configurable config.ResourceOption that providers can use by default.

This would let us keep the current cognitive image of how Upjet works same with TF schema, which lets us add more features easily. With TF schema, we take the raw input, produce some default configurations and then let user add their override config.ResourceOptions and then run the code generation. If we treat the provider metadata just as raw as schema where we have basic information, we can say that it works the same way with schema; the only difference would be that instead of terraform schema command, it's our scraper command line tool producing it.