Merge pull request #67 from donmendelson/issue65

Alignment of var data and repeating groups #65

Author: donmendelson

v2-0-RC1/doc/01Introduction.md

@@ -185,4 +185,4 @@ Representation of dates and times
 Securities and related financial instruments - Codes for exchanges and
 market identification (MIC)
 
-XML 1.1 schema standards are located here [W3C XML Schema](http://www.w3.org/XML/Schema.html#dev)
+[W3C XML Schema version 1.1](https://www.w3.org/TR/2006/REC-xml11-20060816/)

v2-0-RC1/doc/03MessageStructure.md

@@ -4,7 +4,7 @@ Message Structure
 Message Framing
 ------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-SBE messages have no defined message delimiter. Version 2.0 of SBE makes it possible to walk the elements of a message to determine its limit, even when the message has been extended. Nevertheless, since internal framing depends on a correct starting point and not encountering malformed messages, it may be desirable to use an external framing protocol when used with transports that do not preserve message boundaries, such as when they are transmitted on a streaming
+SBE messages have no defined message delimiter. SBE makes it possible to walk the elements of a message to determine its limit, even when the message has been extended. Nevertheless, since internal framing depends on a correct starting point and not encountering malformed messages, it may be desirable to use an external framing protocol when used with transports that do not preserve message boundaries, such as when they are transmitted on a streaming
 session protocol or when persisting messages in storage. 
 
 ### Simple Open Framing Header
@@ -78,7 +78,7 @@ The fields of the SBE message header are:
 Block length is specified in a message schema, but it is also serialized
 on the wire. By default, block length is set to the sum of the sizes of
 body fields in the message. However, it may be increased to force
-padding at the end of block. See section 3.3.3.3 below.
+padding at the end of block. See [Padding](#padding-at-end-of-a-message-or-group).
 
 ### Message header schema
 
@@ -87,7 +87,7 @@ position as shown below. Each of these fields must be encoded as an
 unsigned integer type. The encoding must carry the name "messageHeader".
 
 The message header is encoded in the same byte order as the message
-body, as specified in a message schema. See section 4.3.1.
+body, as specified in a message schema. See section 4.
 
 Recommended message header encoding
 
@@ -120,9 +120,9 @@ for blockLength.
 
 The total space reserved for the root level of the message not counting
 any repeating groups or variable-length fields. (Repeating groups have
-their own block length; see section 3.4 below. Length of a
+their own block length; see [Group block length](#group-block-length) below. Length of a
 variable-length Data field is given by its corresponding Length field;
-see section 2.7.3 above.) Block length only represents message body
+see section 2.) Block length only represents message body
 fields; it does not include the length of the message header itself,
 which is a fixed size.
 
@@ -134,18 +134,18 @@ schema.
 
 ### Template ID
 
-The identifier of a message type in a message schema. See section 4.5.2
-below for schema attributes of a message.
+The identifier of a message type in a message schema. See section 4
+for schema attributes of a message.
 
 ### Schema ID
 
-The identifier of a message schema. See section 4.3.1 below for schema
+The identifier of a message schema. See section 4 for schema
 attributes.
 
 ### Schema version
 
 The version number of the message schema that was used to encode a
-message. See section 4.3.1 below for schema attributes.
+message. See section 4 for schema attributes.
 
 ### Number of repeating groups
 
@@ -227,6 +227,8 @@ would cause fields to overlap.
 Extra octets specified for padding should never be interpreted as
 business data. They should be filled with binary zeros.
 
+The offset attribute is mutually exclusive with alignment; see below.
+
 Example of fields with specified offsets
 
 ```xml
@@ -247,6 +249,29 @@ Example of fields with specified offsets
 | OrderQty | 4    | 1                       | 16     |
 | Symbol   | 8    | 0                       | 20     |
 
+#### Field byte alignment
+As an alternative to specifying a field's offset from the start of a block, its byte alignment may be specified. If specified, the start of a field should reside at an offset from the start of the buffer that is the next multiple of the alignment attribute. If the offset without alignment would not be a multiple of the alignment attribute, then the buffer is padded with octets to that multiple. The padding should be set to binary zeroes.
+
+The alignment attribute is mutually exclusive with offset.
+
+If neither offset nor alignment attributes are set, then a field is positioned at the next available octet in a buffer without padding.
+
+Computation of padding and effective offset is as follows, where effective-offset is the resulting position in the buffer:
+
+```
+padding = (alignment - (unaligned-offset modulo alignment)) modulo alignment
+effective-offset = unaligned-offset + padding
+```
+
+Example of fields with alignment that is equivalent to the offset example above:
+
+```xml
+<field name="ClOrdID" id="11" type="string14" semanticType="String"/>
+<field name="Side" id="54" type="char" semanticType="char"/>
+<field name="OrderQty" id="38" type="intQty32" alignment="4" semanticType="Qty"/>
+<field name="Symbol" id="55" type="string8" alignment="4" semanticType="String"/>
+```
+
 #### Padding at end of a message or group
 
 In order to force messages or groups to align on byte boundaries or map
@@ -308,6 +333,12 @@ fields. (Length of a variable-length Data field is given by its corresponding
 Length field.) Block length only represents message body fields; it does not
 include the length of the group dimension itself, which is a fixed size.
 
+### Byte alignment of a group entry
+
+If byte alignment of a repeating group is specified, the start of the first field of each entry should reside at an offset from the start of the buffer that is the next multiple of the alignment attribute. If the offset without alignment would not be a multiple of the alignment attribute, then the buffer is padded with octets to that multiple. The padding should be set to binary zeroes.
+
+If byte alignment is not specified for a group, then no padding is introduced to align entries.
+
 ### Padding at end of a group entry
 
 By default, the space reserved for an entry is the sum of a group’s
@@ -322,7 +353,7 @@ each entry and should be set to zeroes by encoders. The blockLength
 value does not include the group dimensions itself.
 
 Note that padding will only result in deterministic alignment if the
-repeating group contains no variable-length fields.
+repeating group contains no variable-length fields. Therefore, the alignment attribute is a more direct solution to that need.
 
 ### Entry counter 
 
@@ -507,7 +538,7 @@ finally, variable-length data fields.
 Message structure validation
 --------------------------------------------------------------------------------------------------------------------------
 
-Aside from message schema validations (see section 4.8 below), these
+Aside from message schema validations (see section 4), these
 validations apply to message structure.
 
 If a message structure violation is detected on a received message, the

v2-0-RC1/doc/04MessageSchema.md

@@ -5,6 +5,8 @@ XML schema for SBE message schemas
 ---------
 See [sbe.xsd](../resources/sbe.xsd) for the normative XML Schema Definition (XSD) for SBE.
 
+The SBE schema is defined with W3C XML Schema Definition Language (XSD) version 1.1.
+
 
 XML namespace
 -----------------------------------------------------------------------------------------------------------
@@ -379,6 +381,10 @@ The `name` and `id` attributes are required. The first is a display name for
 a message, while the latter is a unique numeric identifier, commonly
 called template ID.
 
+### Byte alignment
+
+By default, a message starts at the first available byte of buffer. However, if desired, it may be aligned to a boundary by specifying the `alignment` attribute to improve performance on certain platforms. If specified, the start of the message should reside in the buffer at the next multiple of the alignment attribute. For example, if `alignment=4` then the message should start at the next address that is a multiple of 4.
+
 ### Reserved space
 
 By default, message size is the sum of its field lengths. However, a
@@ -415,6 +421,7 @@ that they are encoded on the wire.
 | semanticType          | Documents value of FIX MsgType for a message                                                                                               | token              | optional    | Listed in FIX specifications                                             |
 | sinceVersion          | Documents the version of a schema in which a message was added                                                                             | nonNegativeInteger | default = 0 |                                                                          |
 | deprecated            | Documents the version of a schema in which a message was deprecated. It should no longer be sent but is documented for back-compatibility. | nonnegativeInteger | optional    | Must be less than or equal to the version of the message schema.         |
+| alignment            | Controls byte alignment of the start of a message in its buffer| positiveInteger | optional    |          |
 
 Note that there need not be a one-to-one relationship between message
 template (identified by `id` attribute) and `semanticType` attribute. You
@@ -442,7 +449,8 @@ These are the common attributes of all field types.
 | id               | Unique field identifier (FIX tag)                                                                                                                                                         | unsignedShort       | required                           |                                                                                                       |
 | description      | Documentation                                                                                                                                                                             | string              | optional                           |                                                                                                       |
 | type             | Encoding type name, one of simple type, composite type or enumeration.                                                                                                                    | string              | required                           | Must match the name attribute of a simple `<type>`, `<composite>` encoding type, `<enum>` or `<set>`. |
-| offset           | Offset to the start of the field within a message or repeating group entry. By default, the offset is the sum of preceding field sizes, but it may be increased to effect byte alignment. | unsignedInt         | optional                           | Must be greater than or equal to the sum of preceding field sizes.                                    |
+| offset           | Offset to the start of the field within a message or repeating group entry. By default, the offset is the sum of preceding field sizes, but it may be increased to effect byte alignment. | unsignedInt         | optional                           | Must be greater than or equal to the sum of preceding field sizes. Mutually exclusive with alignment                                    |
+| alignment            | Controls byte alignment of the start of a field in its buffer| positiveInteger | optional    | Mutually exclusive with offset        |
 | presence         | Field presence                                                                                                                                                                            | enumeration         | Default = required                 | required = field value is required; not tested for null.                                                                                                                                                                                                                                                       optional = field value may be null.  constant = constant value not sent on wire.                                                                                                                         |
 | valueRef         | Constant value of a field as a valid value of an enumeration                                                                                                                              | qualifiedName\_t    | optional  Valid only if presence= ”constant”  | If provided, the qualified name must match the name attribute of a `<validValue>` within an `<enum>`  |
 | sinceVersion     | The version of a message schema in which this field was added.                                                                                                                            | InonnegativeInteger | default=0                          | Must not be greater than version attribute of `<messageSchema>` element.                              |
@@ -487,7 +495,7 @@ The number of members of each type is unbound.
 | id                  | Unique group identifier           | unsignedShort   | required                    |                                                                          |
 | description         | Documentation                     | string          | optional                    |                                                                          |
 | dimensionType       | Dimensions of the repeating group | symbolicName\_t | default = groupSizeEncoding | If specified, must be greater than or equal to the sum of field lengths. |
-
+| alignment            | Controls byte alignment of the start of each group instance in its buffer| positiveInteger | optional    |          |
 `<group>` element inherits attributes of blockType. See `<message>`
 above.
 

v2-0-RC1/doc/05SchemaExtensionMechanism.md

@@ -24,6 +24,8 @@ Compatibility is only ensured under these conditions:
 
 -   Existing fields cannot change data type or move within a message.
 
+-   Message and repeating group byte alignment may not change.
+
 -   A repeating group may be added after existing groups at the root level or nested within another repeating group.
 
 -   A variable-length data field may be added after existing variable-length data at the root level or within a repeating group.