Support variable-length registers in `device.yml` register specifications

[PathogenDavid]

• Proposed
• Prototype: Not Started
• Implementation: Not Started
• Specification: Not Started

Summary

Right now the length of a register as defined in device.yml is always a fixed length, this proposal seeks to remove this restriction. The protocol has no such restriction.

Motivation

This has mainly come up due to string registers like the device name register and the proposed semantic version register.

Future uses might include sending back error messages from the device or uploading firmware/configuration blobs.

I think it could be interesting for a device to be able to regurgitate its own device.yml. (Similar to how some APIs expose a well-defined machine-readable endpoint on how to interact with them, EG: YouTube.)

Detailed Design

(Exact implementation open to discussion)

If it doesn't run up against the generator implementations, cleanest would be allowing a special value for length, IE:

DocumentationUrl:
  address: 42
  type: U8
  length: variable
  interfaceType: string
  access: Read
  description: The documentation URL associated with this device

Drawbacks

In an embedded context it can be somewhat problematic to deal with variable-length things, particularly on lower-end MCUs. (As a convention we might not allow writable variable-length registers in the core register set.)

Alternatives

• Continue using fixed-length strings padded with \0 (as R_DEVICE_NAME is today.)
• Could omit length and declare that the variable length comes from the interfaceType
  • However we've talked about wanting interfaceType to be mostly optional so that is somewhat problematic.
  • We could say generators treat all unknown interface types as variable-length arrays of the payload type. Would be silly in some cases but still usable.

Unresolved Questions

• Exact implementation TBD
  • Could use length as 0 or null as a special value to designate variable, but since device.yml files are authored by hand(?) this might be confusing/unclear.
  • Might be weird in case someone wants a write-only register (proposed) where a 0-length register is maybe valid. (IE: For an RPC function that takes no parameters.)
• Should we optionally allow specifying an upper bound for the sake of writable registers?
  • Alternatively, do we want a way to query what a device is capable of having written to it?
  • Both could be hard to define in some circumstances and arbitrary in others.
  • For queries, could allow a response of "unspecified" or just UInt16.MaxValue.
  • Could easily come later when a need arises.
• Will this cause any issues with patterns around logging/indexing data or other tools which work with Harp data in bulk?
  • Bruno had some initial concerns but it sounds like they faded when he remembered they bucket messages by register and/or they could log the data differently(?)
  • Basically the issue might be that if you have a bunch of Harp messages all in one big blob, variable-length messages prevent you from arbitrarily indexing into that blob as if it were an array since you'd need to parse the message headers sequentially to find message boundaries.
  • My personal vibe is that if you're working with data in bulk to the point where you need very fast random access then should shove it into a SQLite database rather than using it as a big blob anyway.
  • Alternatively store the message payloads separately from the headers and replace the payloads with a pointer to the actual data in the separate payload blob. (I think this is Filipe was getting at during SRM.)

Design Meetings

• Proposal stemmed from discussion during 2025-02-20 SRM

[bruno-f-cruz]

• @glopesdev suggests using null instead of 0 to specify variable-length messages in the device.yml

Regarding the standard logging format:

• registers should probably be demux during acquisition time
• harp-python would have to be updated to make a "joint" table