Add metadata scraper

[ulucinar]

Description of your changes

Related to https://github.com/upbound/official-providers/issues/19
Depends on #19

Adds a metadata scraper, which is invoked during provider code generation to generate a provider metadata file.

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable to ensure this PR is ready for review.
• Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Enabled the example generation pipelines in this PR.

[muvaf]

Thanks a lot for this PR @ulucinar ! It's quite challenging to follow recursive traversal like most functions here, would you mind adding unit tests so that we don't break it accidentally and see immediately what we expect them to do?

[muvaf]

This seems to be intended as per-resource but in practice, we set it to the same for all resources. I wonder how it'd look if we had ScrapeConfiguration include existing fields as well as XPath configs and used as input only for the ScrapeRepo function call. This way registry.ProviderMetadata and registry.Resource would be free of scraping-methodology-specific details. What do you think?

[ulucinar]

Removed scraper configuration fields from registry.Resource and registry.ProviderMetadata. Thanks @muvaf!

[muvaf]

Opened a couple issues for stuff that may require refactoring since we need to wrap up metadata work given our timelines and we can iterate on it later. Dropped a few comments that are non-blocking but LGTM! Thanks!

[muvaf]

Do we need this separate require block?

[ulucinar]

Done. Looks like go mod tidy populated those dependencies in a separate block. Manually merged those two blocks for required dependencies and ran go mod tidy again.

[muvaf]

Seems like all scrape functions require the doc and an x-path string. Do you think there is a room for a common interface with Scrape(r *Resource, doc, xpath string) error function which would let us write unit tests focused on single units?

[ulucinar]

I have not included unit tests for the scrape functions as they are not exported (only the registry.ProviderMetadata.ScrapeRepo and registry.ProviderMetadata.Store methods are currently exported). Let's consider this as a future enhancement.

[muvaf]

nitpick: seems like a chance to return early here.

[ulucinar]

Done.

[muvaf]

doc seems to be already used.

[ulucinar]

Done. Renamed as docStr.

[muvaf]

nitpick: does this deserve its own function given it's used only here?

[ulucinar]

Done. Embedded the function.

[muvaf]

I still think that the metadata document should be as raw as it can be and we can do everything references-related in a single place, which is Upjet itself in this PR. But changing it won't have much effect on authoring experience at the moment even though I believe it'd have been simpler to reason about how Upjet works. Opened this issue to have a place to drop feedback about it.

[muvaf]

From our chat, I remember that having a separate dependencies array doesn't make much difference for the later stages so we could possibly remove it to reduce the complexity here. Opened this issue to track the discussion.

[muvaf]

In https://github.com/upbound/official-providers/pull/312 , I see that all entries have the same value for name, such as aws_acm_certificate_validation in provider-aws. Is that intentional?

[ulucinar]

From the markdown files, we scrape the resource name first from the title of the documentation page, and then if any example manifests are available, use what's available in that HCL configuration's block label. I think in the past I have seen some cases where the title did not correctly reflect the resource name. But for most resources title name and the extracted resource name match. And when/if we generate per-resource metadata files, we can get rid of the duplicated map keys.

[ulucinar]

Thanks @muvaf.