Introduce free-form text sections in docs

[ebeahan]

Summary

Introduce a concept into the ECS repo which captures extended usage examples. These examples would appear in the ECS documentation as a dedicated page per field set.

Motivation:

Real-world examples are powerful tools to help users better grasp ECS concepts and how users can better map their custom data sources. In addition to the field specifications, ECS also provides guidance on the preferred design patterns for modeling particular events.

A wealth of examples and past discussion is captured in the repo and on Discuss, but not in the ECS docs. Having such examples incorporated into the docs would provide a single, authoritative location. Restructuring could also allow us to include diagrams, code snippets, and other visual aids.

Issues touching on or providing such examples:

• Is ECS confusing to you? #516
• Add ECS categorization examples #860
• [ECS] Multiple users in an event proposal #809
• Populating of source/destination if both source/client or server/destination is being used #300

Design Proposal:

• Refactor the existing schemas directory

  • As of ECS 1.5.0, the schemas directory contains a collection of YAML files and a README.
  • create subdirectories named after the fieldset
  • Each subdirectory contains the schema file for that fieldset. It can also contain an AsciiDoc formatted usage or example file.

• Add a new page dedicated to the free-form usage and examples docs. The page will be nested underneath the field set reference page.

• The AsciiDoc generator will be updated to support this refactoring in the docs

To-do:

• Add discrete attribute to field details page headers #989 Add [discrete] attribution for field details section headers.
  • [discrete] will need to be added to the previous ECS version branches to ensure the legacy formatting is retained after changing the chunking level.
• Change ECS chunking level docs#1970 docs PR to increase chunking level for ECS book from 1 to 2
• Add usage docs section #988 add free-form usage asciidoc per fieldset

[ebeahan]

I've pushed a branch that can be pulled down and viewed with make docs: https://github.com/ebeahan/ecs/tree/free-text

I had to modify the make docs command arguments setting --chunk=2. We can investigate the correct way to handle within the docs build, but this adjustment unblocked me enough to get a locally working build.

Here, I focused on the hierarchy structure and each usage subpage is a placeholder. The three mockups are viewed under the base, agent, and autonomous system field reference sections.

1. Fieldset details page remains unchanged. One sub-page with usage details:

• Agent Fields
  • Agent Fields Usage

2. Fieldset introduction page with two sub-pages for details and usage:

• Base Fields
  • Base Fields Details
  • Base Fields Usage

3. All fieldset details and usage on same page (added more for completeness and curiosity - feels far too congested):

• Autonomous System Fields containing field descriptions, reuse, and usage.

cc @webmat

[webmat]

Thanks for providing that branch to check out. I have to say, the Star Wars ipsum used in the "Autonomous System" page example is far superior to generic lorem ipsum 😂

I hadn't considered splitting out the field reuse info to another sub-page. This is an interesting idea 👍

For now I have a concern that clicking on the main nav link users see in the sidebar leads to an almost empty page. Sidebar link:

Right now this page only contains the field set description. Moreover, unless the user is looking at one of the very first field sets alphabetically, they won't see how the sidebar section has expanded (as your Base example shows).

Here's two thoughts on how we could address this. Now, and down the line.

For now: that top page should add a mini-table of content that lists all sub-pages to that field set. I think we could add this TOC below the field set definition.

Down the line, I think this top page could contain (in order):

• field set definition paragraph
• TOC of sub-pages
• compact table with [ field name | type | short desc ], where field names are a clickable link that takes the user to the field details sub-page, directly on the correct field section, via a # link

The initial approach would however have a few drawbacks, until we can add the compact table:

• an additional click is required to get to any meaningful information
• most field set's top page would look awkward, especially if their TOC is only one link to their "details" sub-page

[webmat]

Took me a while to realize also that the agent and base static asciidoc was under docs/usage. I had initially envisioned them being alongside the yaml files under schema/. This would have been better for discoverability, but would have required us to copy them around.

So I ultimately agree with your approach of putting them under docs/usage. But we'll have to make sure to document this.

[webmat]

Haha I just realized you were actually providing 3 options 🤦

So feedback above was for the way "Base" is laid out.

Having the usage doc under the list of fields would hinder discoverability too much, I think. The AS field set is really short and doesn't really show this, but if one thinks of adding usage instructions at the bottom of the "event" field set, then we get a good sense of how far down they would be :-)

So I think option 2 (Agent) would strike a great balance for now. This way users clicking the sidebar link would still land exactly on the page they always land on. Then if there's a usage page, they have the sub-page they can visit as well. The only adjustment I'd make is to add the link to the usage page between the description and the field listing, when there's a usage page.

Along with option 2, I think for now I would stick to field reuse information being where it is now, at the bottom of the field list. This section is not ideal, but I think extracting it out to a distinct page may make things worse. Bonus point, by not creating a page out of it, we only have to deal with one link to the usage page when appropriate, but not with awkward 1 or 2 line TOCs :-)

[ebeahan]

So I ultimately agree with your approach of putting them under docs/usage. But we'll have to make sure to document this.

I should have called attention to the usage files under docs/usage. I placed them here to simplify getting the mock-up branch published.

Initially, I had also imagined the usage docs under schema/ and leveraging make to generate the final asciidoc format like we do today with the fieldsets and field details pages. However, this might be unnecessary effort since we already expect the usage docs to be asciidoc files.

Agree we will need clear documentation explaining the file structure.

So I think option 2 (Agent) would strike a great balance for now. This way users clicking the sidebar link would still land exactly on the page they always land on. Then if there's a usage page, they have the sub-page they can visit as well. The only adjustment I'd make is to add the link to the usage page between the description and the field listing, when there's a usage page.

Excellent - this was my preferred choice as well. 😺

Along with option 2, I think for now I would stick to field reuse information being where it is now, at the bottom of the field list. This section is not ideal, but I think extracting it out to a distinct page may make things worse. Bonus point, by not creating a page out of it, we only have to deal with one link to the usage page when appropriate, but not with awkward 1 or 2 line TOCs :-)

OK great.

Side note but related to field reuse - I still plan to revisit some of the field reuse improvements discussed previously in #569.

[ebeahan]

Closing with the merging of #988