Add Metal3 Custom Resources Documentation #559

pratik-mahalle · 2025-07-26T07:04:42Z

Add Metal3 Custom Resources Documentation

Summary

This PR adds comprehensive documentation for Metal3 custom resources in the CAPM3 (Cluster API Provider Metal3) user guide. The documentation covers three key custom resources essential for managing bare metal infrastructure in Kubernetes clusters.

Changes Made

New Documentation Files Added

custom_resources.md - Overview and introduction to Metal3 custom resources
metal3data.md - Documentation for the Metal3Data resource
metal3datatemplate.md - Comprehensive guide for Metal3DataTemplate resource

Updated Files

SUMMARY.md - Added navigation links for the new documentation sections

Documentation Coverage

Metal3DataTemplate

Complete API reference with YAML examples
Metadata specifications (strings, object names, indexes, IP pool references)
Network data specifications following Nova network_data.json format
Template reference management for versioning
Usage guidelines and examples

Metal3Data

Resource lifecycle and management
Template rendering process
Integration with Metal3DataTemplate

Custom Resources Overview

Introduction to Metal3 custom resources
Relationship between different resources
Common use cases and patterns

Benefits

Provides comprehensive reference for Metal3 users
Includes practical examples for real-world scenarios
Improves discoverability through proper navigation structure

Related Issue:
Fixes #398

metal3-io-bot · 2025-07-26T07:04:52Z

Hi @pratik-mahalle. Thanks for your PR.

I'm waiting for a metal3-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lentzi90 · 2025-07-30T05:38:52Z

/ok-to-test

lentzi90

Thank you for the PR. I have gone through it and added comments below. Unfortunately it feels quite AI generated with redundant information, useless links to the front page and confident "best practices" out of nowhere. There is still a lot of good here, but you will need to work on it before we merge.
This was supposed to be a pretty straight forward task, just copy and paste from the links in the issue description and clean it up. Now it is both harder to review (because it is hard to compare with the old docs) and more work to clean it up because of all the extra stuff that I assume came with the AI.

lentzi90 · 2025-07-30T05:45:04Z

docs/user-guide/src/capm3/custom_resources.md

+## Key Concepts
+
+### Data Templates and Instances
+
+CAPM3 uses a template-based approach for generating host-specific configuration
+data:
+
+1. **Metal3DataTemplate**: Contains templates for metadata and network
+   configuration that will be rendered for each host
+2. **Metal3Data**: Represents the actual rendered data for a specific host,
+   created from a template
+
+### Metadata and Network Data
+
+- **Metadata**: Contains host-specific information like hostnames, labels, and
+  custom key-value pairs
+- **Network Data**: Defines the network configuration including interfaces, IP
+  addresses, routes, and DNS settings
+
+### Index Management
+
+CAPM3 automatically manages indexes for hosts to ensure unique identification and
+proper resource allocation. Each Metal3Data instance gets a unique index that is
+used in naming and resource allocation.


This feels redundant to me (everything from key concepts). Some of this is already in the overview and the details are better covered on the separate pages.

lentzi90 · 2025-07-30T05:50:21Z

docs/user-guide/src/capm3/custom_resources.md

+  resource
+- [Metal3DataTemplate](metal3datatemplate.md) - Detailed documentation for the
+  Metal3DataTemplate resource
+- [Cluster API Documentation](https://cluster-api.sigs.k8s.io/) - General


I think we can leave this out. It is not really relevant here, and we are so deep in the docs that any user getting here should already have come across CAPI.

lentzi90 · 2025-07-30T05:59:39Z

docs/user-guide/src/capm3/metal3data.md

+- [Metal3Machine](../introduction.md) - Machine management
+- [BareMetalHost](../bmo/introduction.md) - Host provisioning
+- [IP Address Manager](../ipam/introduction.md) - IP pool management


Delete these. The Metal3Machine link just goes to the front page... BareMetalHost goes to BMO intro...

lentzi90 · 2025-07-30T06:02:09Z

docs/user-guide/src/capm3/metal3data.md

+## Error Handling
+
+### Common Error Scenarios
+
+1. **Template Not Found**: The referenced `Metal3DataTemplate` doesn't exist
+2. **Index Conflicts**: Multiple controllers try to use the same index
+3. **Secret Creation Failure**: Unable to create the required secrets
+4. **Template Rendering Errors**: Invalid template configuration
+
+### Error Status
+
+When errors occur, the status fields are updated:
+
+```yaml
+status:
+  ready: false
+  error: true
+  errorMessage: "Failed to create metadata secret: template validation failed"
+```
+
+## Best Practices
+
+1. **Monitor Status**: Check the `ready` and `error` status fields
+2. **Handle Errors**: Implement proper error handling for failed data
+   generation
+3. **Use Indexing**: Leverage the automatic indexing for consistent naming
+4. **Template Validation**: Validate templates before deployment
+5. **Secret Management**: Ensure proper RBAC for secret access
+
+## Troubleshooting
+
+### Common Issues
+
+**Issue**: Metal3Data stuck in "not ready" state
+
+- **Check**: Template configuration and validation
+- **Solution**: Verify template syntax and referenced resources
+
+**Issue**: Index conflicts
+
+- **Check**: Existing Metal3Data objects and their indexes
+- **Solution**: Clean up orphaned objects or adjust indexing strategy
+
+**Issue**: Secret creation failures
+
+- **Check**: RBAC permissions and namespace access
+- **Solution**: Ensure proper permissions for secret creation
+
+### Debugging Commands
+
+```bash
+# Check Metal3Data status
+kubectl get metal3data -n <namespace>
+
+# View Metal3Data details
+kubectl describe metal3data <name> -n <namespace>
+
+# Check generated secrets
+kubectl get secrets -n <namespace> | grep <machine-name>
+
+# View secret content
+kubectl get secret <secret-name> -n <namespace> -o yaml
+```


Delete this (from Error handling). This does not seem useful to me and it should anyway not be on this page. We have a separate troubleshooting page where it may fit. But even then, we should not include things like index conflicts... that is handled by the controllers and not something users should have to think about.

lentzi90 · 2025-07-30T06:07:32Z

docs/user-guide/src/capm3/metal3data.md

+  errorMessage: "<error-message>"
+```
+
+## Lifecycle


This part is missing an important thing:

If the Metal3DataTemplate object is updated, the generated secrets will not be updated, to allow for reprovisioning of the nodes in the exact same state as they were initially provisioned. Hence, to do an update, it is necessary to do a rolling upgrade of all nodes.

https://github.com/metal3-io/cluster-api-provider-metal3/blob/686d7e69531b21eb9370edaf0f72d017d3e037e9/docs/api.md?plain=1#L1001-L1004

This is not addressed either?

lentzi90 · 2025-07-30T06:42:28Z

docs/user-guide/src/capm3/metal3datatemplate.md

+## Best Practices
+
+1. **Use Template References**: Always set `templateReference` for better
+   template management
+2. **Plan Indexing**: Consider your indexing strategy for consistent naming
+3. **Validate Network Config**: Test network configurations before deployment
+4. **Use IP Pools**: Leverage IP pools for dynamic address allocation
+5. **Document Templates**: Keep templates well-documented for team
+   collaboration


Remove this section. Especially the template reference is going to be removed and should not be used.

lentzi90 · 2025-07-30T06:43:54Z

docs/user-guide/src/capm3/metal3datatemplate.md

+- [Metal3Data](metal3data.md) - The rendered data instances
+- [IP Address Manager](https://github.com/metal3-io/ip-address-manager) - IP pool
+  management
+- [Nova Network Data Format](https://docs.openstack.org/nova/latest/) - Network


Use the same link as earlier or remove it.

lentzi90 · 2025-07-30T06:45:27Z

docs/user-guide/src/capm3/metal3datatemplate.md

+## Template Reference Management
+
+The `templateReference` field enables template versioning and updates:
+
+- **Immutable Templates**: Data template parts are immutable since BareMetalHost
+  references the secrets
+- **Update Process**: Updates require creating a new template and referencing it
+  in the Metal3MachineTemplate
+- **Backward Compatibility**: Supports transition from old templates without
+  `templateReference` to new ones
+
+### Template Linking
+
+Metal3Data objects are linked to Metal3DataTemplate by:
+
+1. Direct reference in the `template` field
+2. Matching `templateReference` key
+3. Template's `templateReference` matching the Metal3Data's template name
+   (backward compatibility)


I suggest removing everything about template references as we are going to remove that field soon anyway.

lentzi90 · 2025-07-30T06:48:38Z

docs/user-guide/src/capm3/metal3datatemplate.md

+- `fromAnnotation`: MAC from object annotation
+- `fromHostInterface`: MAC from BareMetalHost interface
+
+### Networks Configuration


This section could benefit from some sub-sections like Bond Modes and Ethernet Types above. For example the fields under networks and routes

tuminoid · 2025-08-05T07:49:34Z

/retitle Add Metal3 Custom Resources Documentation
More descriptive title for the PR.

pratik-mahalle · 2025-08-10T04:34:29Z

Hey @lentzi90, I have updated and improve it. Please let me know if anything needs to revise or check

lentzi90

Thank you, it looks better!
I still have a few comments below. Could you also check the test failures?
We are using a markdown linter that requires a specific format for some things. For example it is complaining if code blocks or titles are not surrounded by blank lines.

lentzi90 · 2025-08-11T05:59:43Z

docs/user-guide/src/capm3/metal3data.md

+## Template Reference Management
+
+The `templateReference` field enables linking to specific template versions:
+
+- **Backward Compatibility**: If not set, the controller matches by template
+  name
+- **Version Control**: When set, enables template versioning and updates
+- **Transition Support**: Allows migration from old templates to new ones


Please remove this.

lentzi90 · 2025-08-11T06:00:01Z

docs/user-guide/src/capm3/metal3data.md

+    kind: Metal3DataTemplate
+    name: worker-template
+spec:
+  templateReference: worker-v1


Suggested change

templateReference: worker-v1

lentzi90 · 2025-08-11T06:01:27Z

docs/user-guide/src/capm3/metal3data.md

+## Template Update Behavior
+
+**Important**: If the `Metal3DataTemplate` object is updated, the generated
+secrets will not be updated automatically. This behavior is intentional to
+allow for reprovisioning of the nodes in the exact same state as they were
+initially provisioned.
+
+To apply template updates to existing nodes, it is necessary to perform a
+rolling upgrade of all nodes that reference the updated template. 


Could you move this down and make it a subsection under "Lifecycle"?

lentzi90 · 2025-08-11T06:03:27Z

docs/user-guide/src/capm3/custom_resources.md

+## Key Concepts
+
+### Data Templates and Instances
+
+CAPM3 uses a template-based approach for generating host-specific configuration
+data:
+
+1. **Metal3DataTemplate**: Contains templates for metadata and network
+   configuration that will be rendered for each host
+2. **Metal3Data**: Represents the actual rendered data for a specific host,
+   created from a template
+
+### Metadata and Network Data
+
+- **Metadata**: Contains host-specific information like hostnames, labels, and
+  custom key-value pairs
+- **Network Data**: Defines the network configuration including interfaces, IP
+  addresses, routes, and DNS settings
+
+### Index Management
+
+CAPM3 automatically manages indexes for hosts to ensure unique identification and
+proper resource allocation. Each Metal3Data instance gets a unique index that is
+used in naming and resource allocation.


Suggested change

## Key Concepts

### Data Templates and Instances

CAPM3 uses a template-based approach for generating host-specific configuration

data:

1. **Metal3DataTemplate**: Contains templates for metadata and network

configuration that will be rendered for each host

2. **Metal3Data**: Represents the actual rendered data for a specific host,

created from a template

### Metadata and Network Data

- **Metadata**: Contains host-specific information like hostnames, labels, and

custom key-value pairs

- **Network Data**: Defines the network configuration including interfaces, IP

addresses, routes, and DNS settings

### Index Management

CAPM3 automatically manages indexes for hosts to ensure unique identification and

proper resource allocation. Each Metal3Data instance gets a unique index that is

used in naming and resource allocation.

pratik-mahalle · 2025-08-11T10:54:33Z

Hey @lentzi90, I have updated the changes you suggest. Please have a look at it when you got time

lentzi90

Ok I think this is good enough to merge now.
Could you squash the commits? Then I am happy to approve

pratik-mahalle · 2025-08-19T12:30:54Z

Ok I think this is good enough to merge now. Could you squash the commits? Then I am happy to approve

Hey @lentzi90 I have squash the commit, Please go ahead

lentzi90 · 2025-08-20T07:26:52Z

One more thing. Could you include some more details in the commit description? It would also be nice to update the PR description. It still includes things like best practices and what tests have been done (which is quite irrelevant since the automated tests run on the PR). Could you instead please link to the issue (#398) you are fixing?

pratik-mahalle · 2025-08-20T07:55:55Z

Hey @lentzi90, please check. I have updated that

lentzi90

Thanks!
/approve

tuminoid · 2025-08-20T08:07:13Z

docs/user-guide/src/capm3/data_sources.md

+    kind: KubeadmControlPlane
+    name: test1
+    namespace: metal3
+````


There is four ticks, breaks formatting.

tuminoid · 2025-08-20T08:07:28Z

docs/user-guide/src/capm3/data_sources.md

+The `Cluster` resource is **CAPI resource** and includes a reference to the control
+plane via the `controlPlaneRef` field:
+
+``` yaml


Suggested change

``` yaml

```yaml

No spaces

tuminoid · 2025-08-20T08:08:26Z

docs/user-guide/src/capm3/data_sources.md

+an infrastructure provider for cluster API (CAPI), it necessarily references
+also other CAPI resources, however, this document focuses on metal3 resources.
+
+For more details about CAPI resources and to get a big picture, refer to


Suggested change

For more details about CAPI resources and to get a big picture, refer to

For more details about CAPI resources and to get the big picture, refer to

tuminoid · 2025-08-20T08:09:47Z

docs/user-guide/src/capm3/custom_resources.md

+1. Create a Metal3DataTemplate with your desired metadata and network
+   configuration
+2. Reference the template in your Metal3MachineTemplate
+3. CAPM3 automatically creates Metal3Data instances and renders the
+   configuration


Use backticks around CRD names like elsewhere in this PR for consistency.

Right, Done

I don't see any backticks?

tuminoid · 2025-08-20T08:10:31Z

docs/user-guide/src/capm3/custom_resources.md

+
+### Advanced Usage
+
+- Use IP pools for dynamic IP address allocation


Specific resource names mentioned here, preferably with links, would go long way for user friendly documentation.

This is still open.

docs/user-guide/src/capm3/data_sources.md

tuminoid

Hi, thanks for putting in the work. Some feedback:

Can you please fix all of the misspellings of metal3 to Metal3, also cluster API to Cluster API etc
These new files besides data_sources.md are not linked from anywhere, how is user supposed to find them in user-guide?
Let's not use backticks in titles. It is not enforced but it is not good practice
Maybe leave out the design doc URL fix into separate PR as it requires different set of approvers than the rest and is not related to CRD documenation
Some comments left here and there

With some polishing this will make nice addition to our user guide.

/cc @elfosardo @dtantsur
for some more eyes

pratik-mahalle · 2025-08-21T14:16:24Z

Hi, thanks for putting in the work. Some feedback:

1. Can you please fix all of the misspellings of `metal3` to `Metal3`, also `cluster API` to `Cluster API` etc

2. These new files besides data_sources.md are not linked from anywhere, how is user supposed to find them in user-guide?

3. Let's not use backticks in titles. It is not enforced but it is not good practice

4. Maybe leave out the design doc URL fix into separate PR as it requires different set of approvers than the rest and is not related to CRD documenation

5. Some comments left here and there

With some polishing this will make nice addition to our user guide.

/cc @elfosardo @dtantsur for some more eyes

Hey @tuminoid, need to clarify some thing.
As you mentioned about - These new files besides data_sources.md are not linked from anywhere, how is user supposed to find them in user-guide?
Where should i mentioned this?

And also -Maybe leave out the design doc URL fix into separate PR as it requires different set of approvers than the rest and is not related to CRD documenation
What should i do now for this?

tuminoid · 2025-08-22T05:42:30Z

Hey @tuminoid, need to clarify some thing. As you mentioned about - These new files besides data_sources.md are not linked from anywhere, how is user supposed to find them in user-guide? Where should i mentioned this?

These new documentation should be linked somewhere in the main menu, or linked from related existing documentation, so the user's can reach them.

And also -Maybe leave out the design doc URL fix into separate PR as it requires different set of approvers than the rest and is not related to CRD documenation What should i do now for this?

Remove that part from this PR, and open a separate PR for the URL fix.

pratik-mahalle · 2025-08-22T06:06:51Z

Hey @tuminoid, As you mentioned, I have updated things. Please let me know if there is any changes.

tuminoid · 2025-08-22T07:25:00Z

/Cc @lentzi90
also to take another look

tuminoid

There are some open comments left.

Also, squash the commits when done fixing them up.

tuminoid · 2025-09-03T06:58:25Z

docs/user-guide/src/capm3/data_sources.md

-Visualization of relationships between metal3 resources can be found for example
-[here](https://github.com/metal3-io/cluster-api-provider-metal3/issues/1358).
+Visualization of relationships between Metal3 resources can be found for example
+[cluster provider Metal3](https://github.com/metal3-io/cluster-api-provider-metal3/issues/1358).


Suggested change

[cluster provider Metal3](https://github.com/metal3-io/cluster-api-provider-metal3/issues/1358).

in this [CAPM3 isssue](https://github.com/metal3-io/cluster-api-provider-metal3/issues/1358).

tuminoid · 2025-09-03T06:58:41Z

docs/user-guide/src/capm3/custom_resources.md

+1. Create a Metal3DataTemplate with your desired metadata and network
+   configuration
+2. Reference the template in your Metal3MachineTemplate
+3. CAPM3 automatically creates Metal3Data instances and renders the
+   configuration


I don't see any backticks?

tuminoid · 2025-09-03T06:58:49Z

docs/user-guide/src/capm3/custom_resources.md

+
+### Advanced Usage
+
+- Use IP pools for dynamic IP address allocation


This is still open.

tuminoid · 2025-09-03T06:59:49Z

docs/user-guide/src/capm3/metal3data.md

+  errorMessage: "<error-message>"
+```
+
+## Lifecycle


This is not addressed either?

docs/user-guide/src/capm3/custom_resources.md

pratik-mahalle · 2025-09-29T05:04:33Z

Hey @lentzi90 @tuminoid Any news on this??

tuminoid · 2025-09-29T06:00:38Z

There are some open comments left.

Also, squash the commits when done fixing them up.

I don't see updates on the PR after these comments, so they're still open on your side.

lentzi90 · 2025-10-24T11:42:59Z

I don't see much progress here. Are you still working on this?
There are now again multiple commits that should be squashed. Please address the comments!
/approve cancel

pratik-mahalle · 2025-11-24T13:26:19Z

Hey @lentzi90, sorry for the delay, I will update the pr and squash the commit

metal3-io-bot · 2025-11-24T13:44:14Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kashifest for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS
design/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

metal3-io-bot · 2025-11-24T13:44:35Z

@pratik-mahalle: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
spellcheck	`c08f7d3`	link	true	`/test spellcheck`
markdownlint	`c08f7d3`	link	true	`/test markdownlint`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

pratik-mahalle · 2025-11-24T13:48:48Z

Hey @lentzi90 As this pr is going big, I am creating the new pr for this one to understand the changes quickly. I would really appreciate your feedback on that
#598

metal3-io-bot requested review from elfosardo and s3rj1k July 26, 2025 07:04

metal3-io-bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 26, 2025

pratik-mahalle mentioned this pull request Jul 30, 2025

Document Metal3Data and Metal3DataTemplate #398

Open

metal3-io-bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 30, 2025

lentzi90 reviewed Jul 30, 2025

View reviewed changes

metal3-io-bot added the needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. label Jul 31, 2025

metal3-io-bot changed the title ~~add a section in metal3 docs~~ Add Metal3 Custom Resources Documentation Aug 5, 2025

metal3-io-bot removed the needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. label Aug 10, 2025

lentzi90 reviewed Aug 11, 2025

View reviewed changes

lentzi90 reviewed Aug 19, 2025

View reviewed changes

pratik-mahalle force-pushed the doc branch 2 times, most recently from 137982a to aba03ce Compare August 19, 2025 12:30

pratik-mahalle force-pushed the doc branch from d6e6450 to 7fede07 Compare August 20, 2025 07:55

lentzi90 reviewed Aug 20, 2025

View reviewed changes

tuminoid reviewed Aug 20, 2025

View reviewed changes

docs/user-guide/src/capm3/data_sources.md Outdated Show resolved Hide resolved

tuminoid reviewed Aug 20, 2025

View reviewed changes

docs/user-guide/src/capm3/data_sources.md Outdated Show resolved Hide resolved

tuminoid reviewed Aug 20, 2025

View reviewed changes

metal3-io-bot requested a review from dtantsur August 20, 2025 08:14

metal3-io-bot requested a review from lentzi90 August 22, 2025 07:25

tuminoid reviewed Sep 3, 2025

View reviewed changes

metal3-io-bot added needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 24, 2025

docs(capm3): add Metal3 custom resources docs

c08f7d3

pratik-mahalle force-pushed the doc branch from 97c4928 to c08f7d3 Compare November 24, 2025 13:44

metal3-io-bot removed the needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. label Nov 24, 2025

	For more details about CAPI resources and to get a big picture, refer to
	For more details about CAPI resources and to get the big picture, refer to


		### Advanced Usage

		- Use IP pools for dynamic IP address allocation

	[cluster provider Metal3](https://github.com/metal3-io/cluster-api-provider-metal3/issues/1358).
	in this [CAPM3 isssue](https://github.com/metal3-io/cluster-api-provider-metal3/issues/1358).

Add Metal3 Custom Resources Documentation #559

Are you sure you want to change the base?

Add Metal3 Custom Resources Documentation #559

Uh oh!

Conversation

pratik-mahalle commented Jul 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!