828 MCAP specification for osi tracefiles

doc/architecture/trace_file_naming.adoc → doc/architecture/trace_file_binary_and_human_readable_formats.adoc

@@ -2,11 +2,16 @@ ifndef::include-only-once[]
 :root-path: ../
 include::{root-path}_config.adoc[]
 endif::[]
-= OSI trace file naming conventions
+= Native binary and Human-readable Formats
 
-**Name format**
+== Binary .osi Format
+Messages are separated by a four-byte, little-endian, unsigned integer specifying the length of each message.
 
-The names of OSI trace files should have the following format:
+== Human-readable .txth Format
+Messages are stored as plain text, separated by newlines.
+
+== Naming Convention
+Binary .osi and human-readable .txth files should follow this naming convention:
 
 ----
 <timestamp>_<type>_<osi-version>_<protobuf-version>_<number-of-frames>_<custom-trace-name>.osi
@@ -76,3 +81,5 @@ The recommended file name is:
 ----
 20210818T150542Z_sv_312_300_1523_highway.osi
 ----
+
+NOTE: This naming convention does not apply to .mcap files, they must follow the naming convention described in their section.

doc/architecture/trace_file_mcap_format.adoc

@@ -0,0 +1,121 @@
+ifndef::include-only-once[]
+:root-path: ../
+include::{root-path}_config.adoc[]
+endif::[]
+= MCAP Format
+
+== General Requirements
+- Must comply with the https://mcap.dev/spec[MCAP format specification] version `0x30`
+- Must allow other non-OSI data to be present in the MCAP file
+- Message records must be written into `chunk records` for indexed files
+- Only OSI top-level messages containing a timestamp field are permitted to be directly stored in MCAP channels
+- Must contain only a single scenario with a unique global time
+- An MCAP file is considered a single dataset
+
+== Schema
+- `name` field: Full message type name, including package (e.g., `osi3.SensorData`)
+- `encoding` field: Must be `protobuf`
+- `data` field: String-encoded `google::protobuf::FileDescriptorSet` for the OSI top-level message
+
+== Channel
+- `message_encoding` field: Must be `protobuf``
+- `metadata` field:
+** Must include an `osi_version` key, specifying the OSI SemVer version of the OSI top-level message contained within the channel
+** Must include a `protobuf_version` key, specifying the protobuf SemVer version used to create the OSI top-level message contained within the channel
+** Should include a `description` key, explaining the data's origin and purpose in natural language.
+
+
+== Message
+- `publish_time` field: 
+** Must reflect the timestamp of the stored OSI top-level message
+** Must be in nanoseconds
+- `log_time` field: Must reflect the time when the message was enqueued for MCAP file addition
+** Must reflect the timestamp of the stored OSI top-level message
+** Must be in nanoseconds
+
+
+== File-wide Metadata
+- Must include metadata with the name `versions` containing at least the following key-value pair:
+** `osi`: SemVer version of the minimum required OSI version
+- Must include metadata with the name `asam_osi` containing at least the following key-value pairs:
+** `zero_time`: ISO 8601 YYYYMMDDThhmmss.f formatted point in time representing the zero time of the scenario
+** `timestamp`: ISO 8601 YYYYMMDDThhmmss.f formatted creation time of the file 
+- It is strongly recommended to include metadata with the name `asam_osi` containing the following key-value pairs:
+** `description`: Short human-readable scenario description
+** `creator`: csv of person or company (not tool) creating the file
+** `license`` csv of spdx identifiers
+** `data_sources` csv of model, scenario player, etc.
+- Additional custom metadata may be added, but it is recommended to add a category with the name `context` where the key represents a prefix and the value pointing to the specification of the metadata. This allows to add other (channel-wise) metadata with the stated prefix. Thus, it becomes clear what a metadata is about and where it is specified. The following examples are given:
+** GAIA-X4PLC-AAD SHACL Shape
+*** Assume you want to embed the hdmap of a scenario in the MCAP file.
+*** The `context` category contains the key `GAIA-X4PLC-AAD-hdmap` with the value `https://github.com/GAIA-X4PLC-AAD/ontology-management-base/blob/main/hdmap/hdmap_shacl.ttl`
+*** A channel metadata contains the key `GAIA-X4PLC-AAD-hdmap` with the value of the hdmap data in the given SHACL shape.
+** openDrive Reference
+*** Assume you want to express that oncoming traffic passes on the right side of the road.
+*** The `context` category contains the key `openDrive` with the value `https://publications.pages.asam.net/standards/ASAM_OpenDRIVE/ASAM_OpenDRIVE_Specification/1.8.1/specification/index.html`
+*** A file metadata in a new metadata category with the arbitrary name `specification` contains the key `openDrive-road-rule` with the value `RHT`
+** Cycle time variation of a sensor
+*** Assume you want to express the interface cycle time variation of a sensor.
+*** The `context` category contains the key `iso_23150` with the value` ISO 23150:2011`
+*** A channel containing `OSI3::SensorData` messages has metadata with the key `iso_23150-cycle-time-variation:` and the value `80`
+
+== Compression
+- OSI-compliant tooling must support compression types: `none`, `lz4`, and `zstd`
+
+
+== Naming Convention
+.mcap files must follow this naming convention:
+
+
+----
+<opt. prefix>_<opt. timestamp>_<type>_<opt. suffix>.mcap
+----
+
+When not using an optional field, the corresponding underscore delimiter must be omitted so that no double underscore is present.
+
+[#tab-MCAP-file-naming-convention]
+.MCAP file naming convention
+[cols="1,1"]
+|===
+|Field |Explanation 
+
+|opt. prefix
+|An optional prefix which may be used to specify the type of scenario (e.g. `cut-in`) or uniqueness of the setup (e.g. `target-5m`). May not contain any `_` characters.
+
+|opt. timestamp
+|Defines the absolute start time for a scenario or recording. If following the recommended zero time for the timestamps of the top-level messages, this time must represent the zero time. The format must adhere to ISO 8601 [cite:iso8601].
+
+
+|type
+| Specifies the type of the contained the top-level message(s) and must be one of the following values:
+
+`sv` file contains only `SensorView` messages. +
+`gt` file contains only `GroundTruth` messages. +
+`hvd` file contains only `HostVehicleData` messages. +
+`sd` file contains only SensorData` messages. +
+`tc` file contains only `TrafficCommand` messages. +
+`tcu` file contains only `TrafficCommandUpdate` messages. +
+`tu` file contains only `TrafficUpdate` messages. +
+`mr` file contains only `MotionRequest` messages. +
+`su` file contains only `StreamingUpdate` messages. +
+`multi` file contains multiple, different types of of top-level messages (not including different channels of the same type).
+
+|opt. suffix
+|An optional suffix which may be used the same way as the optional prefix or be used to specify further details like the minimum required OSI version. May not contain any `_` characters.
+
+
+|===
+
+
+**Example**
+
+The following list shows examples of valid OSI MCAP file names:
+
+- `20210818T150542Z_highway_sv.mcap`
+- `20210818T150542Z_highway_sv_run-1.mcap`
+- `20210818T150542Z_highway_gt_OSI-3-7.mcap`
+- `Highway_sd_version-1.mcap`
+- `Highway-cut-in-no-collision_sd.mcap`
+- `Target-5m_sd_resimulated-measurement.mcap`
+
+NOTE: This naming convention does not apply to .osi and .txt files, they should follow the naming convention described in their section.

doc/architecture/trace_file_formats.adoc → doc/architecture/trace_file_overview_file_formats.adoc

@@ -3,22 +3,25 @@ ifndef::include-only-once[]
 include::{root-path}_config.adoc[]
 endif::[]
 [#top-osi_trace_file_formats]
-= OSI trace file formats
+= Overview file formats
 
-There are two formats for storing multiple serialized OSI messages in one trace file.
+There are three formats for storing OSI messages in trace files:
 
 *.osi::
-Binary trace file.
-Messages are separated by a length specification before each message.
-The length is represented by a four-byte, little-endian, unsigned integer.
-The length does not include the integer itself.
+Native binary trace file.
 
 *.txth::
 Human-readable plain-text trace file.
-Messages are separated by newlines.
+
+*.mcap::
+Binary trace file supporting more advanced features like indexed data, additional metadata and more.
 
 NOTE: Previous releases of OSI also supported a so-called plain-text trace file format, with file extension `.txt`.
 This legacy format did not contain plain-text, but rather binary protobuf messages separated by a special separator.
 For obvious reasons the format was deprecated and fully replaced with the `.osi` binary file format.
 This release no longer contains any support for the legacy `.txt` file format.
 These files may be used for manual checks.
+
+
+TIP: For efficient handling of trace files, you can utilize the specification-compliant tools and libraries provided in the companion https://github.com/OpenSimulationInterface/osi-utilities[osi-utilities] repository. For example, convenient writer and reader classes are provided handling OSI messages for the different file formats.
+

doc/open-simulation-interface_user_guide.adoc

@@ -57,19 +57,11 @@ include::./architecture/packaging_layer.adoc[leveloffset=+3]
 
 === OSI trace files
 
-include::./architecture/trace_file_formats.adoc[leveloffset=+3]
+include::./architecture/trace_file_overview_file_formats.adoc[leveloffset=+3]
 
-include::./architecture/trace_file_naming.adoc[leveloffset=+3]
+include::./architecture/trace_file_binary_and_human_readable_formats.adoc[leveloffset=+3]
 
-// === Files and scripts
-
-// include::./architecture/proto-files.adoc[leveloffset=+3]
-
-// include::./architecture/test_scripts.adoc[leveloffset=+3]
-
-include::./architecture/trace_file_example.adoc[leveloffset=+3]
-
-include::./architecture/formatting_script.adoc[leveloffset=+3]
+include::./architecture/trace_file_mcap_format.adoc[leveloffset=+3]
 
 
 // Setting up OSI