Scenario Selective Inheritance support

[mkudukin]

This PR updates repository schema based on the discussion, and adds example file. Resolves #211.

[kleihan]

@mkudukin why did you remove the sort attribute from the codeType? This attribute is not specific to scenarios and used for display purposes of code sets.

[mkudukin]

Hello @kleihan! We suggest removing the sort attribute and instead displaying items in the order they appear in the XML file. The position reference feature allows for inserting new codes in the middle of existing code sets. This means the solution depends on the scenario proposal. Please let me know if we should open a separate issue/PR for this anyway.

[kleihan]

Codes do not need any ordering in the wire format. They are mutually exclusive values and the optional use of the sort attribute may be used to order them. FIXimate uses this attribute and hence depends on it. A user can decide to not to the attribute and always rely on the physical order in the XML file.

[kleihan]

The following is a short summary of reasons for not approving the proposed schema changes. Scenarios are one of the key features of Orchestra. I fully agree with the need to clearly define relationships between scenarios. Version 1.0 does not provide this. However, I do not think that this proposal is the right one and will enter a separate proposal (#228).

• The chosen syntax deviates from the current Orchestra "language" that is declarative and not procedural. The positioning of elements is done through instructions (before/after) instead of references like we have today.
• We already have an approach for external instructions to modify an XML file with the open-source tool XML Diff/Merge that used the IETF RFC 5261 standard for this.
• Instructions should not be embedded in the XML file and require post-processing by the receiver of an Orchestra XML file.
• The proposed change requires a large number of new types and elements that are not needed to cover the requirements
• Compatibility with the Tablature does not seem possible with the new approach
• Excluding an element reference from a definition is already covered by the V1.0 attribute "supported=forbidden/ignored" and does not require new "unref"-elements
• The code attribute "sort" should not be removed as it is in use
• The ability to reference a scenario not only by ID but also by name is not supported
• Human readability of the XML file is severly impacted and was a design goal for Orchestra
• The complexity added by the proposal should be accompanied by a guideline to explain its usage

[donmendelson]

I agree with @kleihan. From the perspective of a user that has received rules of engagement from a service provider, expected usage should be clear without having to apply scenario transformation rules. The proposal is also overly complex; we should respect the KISS principle.

I concede that such transformations may useful to a publisher of rules of engagement as an audit trail, but that should be kept internally. A client is only interested in the message definitions after scenario transformations have been applied.

[JohnGreenan]

Quote from Hanno:

Instructions should not be embedded in the XML file and require post-processing by the receiver of an Orchestra XML file.

The Orchestra file is the artefact. It's not source code to run into a compiler. It's the be-all and end-all. Else how do you handle bugs where compiler A and compiler B have different outputs?

You then need an arbitration mechanism. Then FIX Trading Community is going to get pulled into arbitration between different implementations.

This is NOT the right path.

On this reason alone I fundamentally disagree with this mechanism.

I reserve the right to find more bones of contention, but this is sufficient to reject.

[jimnup]

I agree with @kleihan. From the perspective of a user that has received rules of engagement from a service provider, expected usage should be clear without having to apply scenario transformation rules. The proposal is also overly complex; we should respect the KISS principle.

I concede that such transformations may useful to a publisher of rules of engagement as an audit trail, but that should be kept internally. A client is only interested in the message definitions after scenario transformations have been applied.

I share the basic view of Don Mendelson.

A few years ago - we discussed this at length. The view is that this should be in the realm of tooling not built into Orchestra. For instance - a tool could use a base repository and then have actions to push new fields out to scenarios. I don't see any reason to try and implement these full semantics into XML. XML is already too complex to be used as a graph database- which effectively these semantic extensions would require.

[jimnup]

Following up from the meeting:

I think that there needs to be a serialization format that does not require applying transformation - the published format that would be consumed by trading systems for automation needs to by simplified. Can you imagine if to read a REST inerface there had to be a processor unit.
There can be two standards. There can be a persistence / inheritance issue for maintaining Orchestra files that is separate from the Orchestra artefact.
I have always maintained that the inheritance and the relationship between a repository and orchetra files that are rules of engagement is the area of tooling.
There could be a set of extensions for repository maintenance - but I would separate that from the orchestra rules of engagement and the machine to machine exchange.

If a group of vendors want to band together to create a tooling specification for managing Orchestra repositories - which would include the inheritance features - I don't have a problem with that - but that would be a management / tooling standard - it should not be part of the orchestra serialization / runtime interface of FIX Orchestra for machine-machine interface.

[JoanPuig]

I like @jimnup idea. We should explore in the next meeting.

[martinswanson]

I agree with @donmendelson and others that Orchestra specifications should be fully declarative, and any necessary pre-processing or transformations to generate correct XML should be handled internally by the publishers.

Below is an overview of our current build pipeline:

Key Notes:

• All pre-processing and transformations are performed at compile time by the publisher, remaining completely transparent to subscribers.
• Subscribers receive fully declarative Orchestra XML files and other generated artifacts ready for use at runtime.
• We leverage Tablature to manage derived specifications, significantly reducing maintenance effort. Tablature automatically copies base elements from a reference specification.
• We utilise various transformations that could be implemented through different build tools (whether proprietary or community-driven). We're particularly interested in the XML Diff/Merge tool and the IETF RFC 5261 standard as a general-purpose solution for applying transformations to Orchestra.
• Integrating additional transformation tools, such as a scenario builder, should be straightforward.
• Validation, while crucial, is distinct from transformation. We believe in having multiple validators to ensure:
  • Compatibility of an Orchestra spec with specific encodings (e.g., the ordering of elements in a repeating group for FIX tag-value).
  • Scenario consistency, either within a single spec or in relation to a reference specification.
  • The presence and use of DC Terms and other metadata for specific documentation formats.
  • ... etc

Regarding @kleihan's other points:

• Tablature compatibility is essential for our workflow.
• We also rely on the sort attribute in some of our tools for display purposes.