Make Features spec machine parseable and move to this repository

[QuintinWillison]

h3. Overview
This issue has come from the a call that @tbedford, AndyNicks and myself had today (16 June 2021).

The source currently lives in our docs repository [here|https://github.com/ably/docs/blob/main/content/client-lib-development-guide/features.textile] and ends up being rendered [at docs.ably.com|https://docs.ably.com/client-lib-development-guide/features/] and, ultimately, also [at ably.com/documentation|https://ably.com/documentation/client-lib-development-guide/features/].

The plan is to move the spec over to this repository, changing its format at the same time (probably from textile to markdown).

h3. Hosting
As part of the scope of this issue we also need to decide where it should live, in terms of a canonical home. It's probably unlikely that we continue to host this in the same location within docs so this is likely to be a new home, with some kind of redirect for old perma-links.

It's highly likely that it will be staged using our {{sdk.ably.com}} bucket, as it needs to be surfaced in CI as part of the pull request workflow, meaning that there will be a home at:

{code}
https://sdk.ably.com/builds/ably/ably-common/main/features-spec/
{code}
There is a related desire to move the entire client library development guide to a new home, and perhaps this spec would continue to live within that scope, in which case there may be an issue to be linked to this soon.

h3. New Format and Validation
The spec is an interesting mix of plain English and canonical anchors (spec points) so markdown feels like the most logical format to present it in going forwards.

Doing that textile to markdown conversation offers us an opportunity to add validation to the spec. We can run a check script in CI which inspects an AST generated from the markdown to validate it against itself. There's also potential for a check to validate that the diff represent by a pull request is legal (especially when it comes to addition, mutation and deletion of spec points).

h3. Motivation
I added this section following [this internal Slack conversation|https://ably-real-time.slack.com/archives/C030C5YLY/p1623844680128800] with niksilver.

We are, as an organisation, rapidly scaling (customers, revenue, team members, teams, etc..). Some of the systems and processes we have in place, such as the mechanisms for maintaining and tracking client library features spec compliance, worked well when Ably was a smaller company but they now present barriers to "bias for action" when it comes to individuals and teams trying to get specific piece of work done in locations where a large amount of wide Ably context is required to be productive.

This move is a relatively small, early step towards separation of concerns. The Docs team are just as keen as the SDK team to break down unnecessary monoliths in order to allow individual, focussed streams of work concern to proceed at pace and with the appropriate level of granularity for those who need to be involved. We need more autonomy in individual teams and that can be achieved by making the interfaces (boundary lines) clear and understood for all - encapsulation is perhaps the right word.

Why is a new home for this document necessary? Because this does not need to be a part of the documentation set used by app developers for day to day development against Ably's client libraries. Spec issues and PRs in the docs repository effectively present noise for the Docs team.

Why move away from textile, which is also technically machine pareseable? Because it's not anywhere near as usable or friendly for the casual (or even regular) contributor. Markdown has won the war - on merit. There are also more tools out there to process markdown (not only to render it to static presentation formats like HTML but also to parse it for validation purposes).

[QuintinWillison]

It's worth also mentioning that there are a number of issues in the docs repository that would need to move over to this repository when we work on this.

[SimonWoolf]

One possibility we might want to consider is having it live in a database? With columns item key (as primary key) & item description, parent, date added, date last significantly modified, minimum (and maximum?) spec versions that it applies to, whether it's mandatory, whether it's deprecated, etc. Maybe even columns for implementation status for each client lib, with the aim of deprecating the spec spreadsheet in favour of this.

Then could have scripts to serialize it (into markdown/textile/whatever) which takes options to output at a particular spec version, or just items that have changed between version x and y, etc).

Just a thought, obv up to sdk team to decide what you want to do 🙂

[QuintinWillison]

Thanks, @SimonWoolf. I think the database idea is interesting, though I would need to understand better how we might audit change history in that context. Git and GitHub present well worn and understood patterns for change management. Perhaps the format is more your concern? (i.e. might we consider a structured data format like JSON? - but then how would we also encapsulate the human readable and formatted elements?)

In terms of using the database to track implementation status, I find that a little bit harder to get on board with as that would still need manual intervention by client library developers to update. My ideal is in fact to be able to extract compliance from annotations in the client library source code bases, so the "ticks" live right next to the code that implements and the code that validates.