Deduplicate plugin documentation

[psss]

Currently we store detailed plugin documentation at two places:

• In the specification in the form of a user story
• As part of the python module as docs string

The first is used to generate web docs, the second for the --help message. The content is very similar though. Would be nice to find a way how to maintain this at once place only. Quick brainstorm:

• Extract the specification story content and feed it to click?
• Or generate story based on the plugin class docstring?

The specification should always be a starting point so I'd say the first approach probably makes more sense. Any other way? Thoughts?

[lukaszachy]

My PoV is that I don't like stories :)

So docstring for me is better, it is closer to the code.

[lukaszachy]

Since we have "dreams" and "ideas" in specification - it could be messy in --help or require additional logic to remove not implemented but planned stuff happening in steps/tmt

[lukaszachy]

On the other hand - we are likely to have images, tables, links and extended examples in 'docs' while having more concise --help.
So story -> click is easier to implement right in this regard.

[lukaszachy]

Another thing to improve is structure of docs - some plugins are huge, specification is mixing plugins for how with step properties, etc.

[psss]

Yeah, we could also make use of the title attribute to make the headings more readable.

[happz]

Currently we store detailed plugin documentation at two places:

• In the specification in the form of a user story
• As part of the python module as docs string

The first is used to generate web docs, the second for the --help message. The content is very similar though. Would be nice to find a way how to maintain this at once place only. Quick brainstorm:

• Extract the specification story content and feed it to click?
• Or generate story based on the plugin class docstring?

The specification should always be a starting point so I'd say the first approach probably makes more sense. Any other way? Thoughts?

I somehow missed this issue, but I believe that I've been working towards solving it for a couple of months already. I agree with the points discussed above, namely the mixture of specification and plugin docs. My approach is this:

• separate plugin docs from the specification. L2 docs focus on L2 attributes, steps, how they work together, etc. but does not cover plugins and their fields. In other words, the prepare page, built from specs/, would mention how and where keys and it'd speak about where the step sits in the pipeline, but it would not discuss the ansible plugin and its keys - instead, we'd have "Plugins" section in docs, with a subtree for each step where would docs for each plugin live.
• plugin docs would live in plugin (class) docstrings and plugin data class help texts. Unfortunately, dataclass fields cannot have docstrings (it's possible to use #: foo... comment, but that's consumed by Sphinx, it does not work with pure Python). We can collect steps, and plugins, from plugins we can reach their data classes that define fields accepted by plugins.
• with a Jinja template, we can render an HTML page for every plugin.
• with a ReST parser and renderer, we can render CLI help for every plugin, honoring ReST directives.
• taking advantage of ReST - it makes HTML better with Sphinx, and for CLI we would have a parser to make it look as good as the HTML version. Raw docstrings would contain ReST directives, and the parser/renderer duo would convert them to nice CLI content.

What I have so far:

• Move code-related pages under new code directory #2545 - to make plugins name available for "Plugins" section in docs
• Generate plugin documentation from their sources #2549 - bootstraps the HTML docs for plugins
• Simple ReST renderer for CLI help texts #2574 - ReST rendering for CLI
• and follow-up PRs improving docs for individual plugins, e.g. adding code-block, proper lists, literals, metavars, references, etc. would start flowing in.