docs: Streamline pallet documentation exposed in the metadata

[lexnv]

Context

In metadata V15, a new docs field has been introduced.
The filed contains pallet documentation valuable for end-users (ie polkadot-js, subxt).

Problem

However, pallets are not documented in a way that exposes the documentation in the metadata.
For instance, the system::pallet lacks associated documentation:

polkadot-sdk/substrate/frame/system/src/lib.rs

Lines 200 to 202 in 122086d

	#[frame_support::pallet]
	pub mod pallet {

Proposal

To address this issue, substrate offers several methods for propagating documentation to the metadata:

polkadot-sdk/substrate/frame/support/procedural/src/lib.rs

Lines 266 to 280 in 122086d

	/// ## Runtime Metadata Documentation
	///
	/// The documentation added to this pallet is included in the runtime metadata.
	///
	/// The documentation can be defined in the following ways:
	///
	/// ```ignore
	/// #[pallet::pallet]
	/// /// Documentation for pallet 1
	/// #[doc = "Documentation for pallet 2"]
	/// #[doc = include_str!("../README.md")]
	/// #[pallet_doc("../doc1.md")]
	/// #[pallet_doc("../doc2.md")]
	/// pub mod pallet {}
	/// ```

Adding a #[pallet_doc("../README.md")] attribute on pallets is the most straight-forward solution to propagate documentation.
However, this should be treated on a per-pallet basis being mindful that the documentation will also propagate when publishing crates.

There are three levels of documentation that we could leverage to streamline this process:

• the README.md level
• the src/lib.rs file-level
• the pallet pub mod pallet level

I would propose to share the README.md documentation at the file and pallet levels.
Then, at the file level we could include extra details about pallets, or implementation details that might not be worth exposing in the metadata.

#[doc = include_str!("../README.md")]
//! Extra implementation details, edge cases to raise awareness for developers.

And at the pallet level respectively, the README.md documentation is propagated solely to the metadata.

/// Extra documentation to be propagated on crates.io.
///
#[pallet_doc("../README.md")] (not propagated on crates.io)
pub mod pallet {

Would love to get your thoughts on this 🙏
cc @kianenigma @bkchr @paritytech/subxt-team

[kianenigma]

In the spirit of using things like docify, and #991, I am generally a fan of doing the opposite of:

I would propose to share the README.md documentation at the file and pallet levels.
Then, at the file level we could include extra details about pallets, or implementation details that might not be worth exposing in the metadata.

I think everything should be in rust-docs. Then, we should remove all these markdown files and replace them with a CI job that will re-generate them upon release or some other frequency. See #1260

As for the metadata, I think we should avoid a new attribute and continue on the same path as the rest of the Rust items in a pallet and how their document will lead to the metadata. If you put some doc on top of mod pallet, it would go to the metadata. If none is provided, none should go to the metadata.

Also, we are injecting a default doc for mod pallet does not exist. Is this not intercepted in the metadata generation part? Possibly not because the metadata is generated in the same macro process.

---

In general, it might be useful to have a way to distinguish between what rust-docs go to metadata and what don't. So something like your #[pallet_doc("../README.md")] but more general. Given that normal rust docs are already #[doc()] attribute under the hood, possibly we could have something like #[metadata_onlt_doc(..)]. Truth is that we optimize rust-docs for readability and linking in docs.rs, not when you see them eg. in your Polkadot Valut app when you sign something.

[jsdw]

Then, we should remove all these markdown files and replace them with a CI job that will re-generate them upon release or some other frequency.

Another option here perhaps is to just check that the readme aligns with the rust docs in CI, but not try to generate it (expecting users to run a script to regenerate it if it changes in some PR, as they would with eg cargo fmt to fix CI errors about formatting).

[xlc]

do we really want user docs to be included in metadata, which is included in wasm?