[Feature]: Schema Type RAW

[wilwade]

Feature Description

Details

Sometimes if the schema already has a serialization system or is just a single point of data, the overhead of Avro is not desired.

Suggestion

• Create a RAW schema type
• Have the schema type to have a data size limit
• Have the schema be described in some way

Alternatives

• Have FIXED_WIDTH or something that can describe fixed width fields
• Have SINGLE which would be a single point of data

Searched for Related Issues

• I have done a search for related issues and either found none, or noted them

[wilwade]

Notes from the 2025-08-21 Community Call

• Alternative: Make Models Governance Controlled. Currently we have an ENUM for the model type (Avro, Parquet). What if instead we had a way that Governance could add new model types. That would allow for other metadata to be added to the model type, and could be reused.
  • Make sure if we have a dynamic model, we don't have to check even more (increasing the cost of submission)
• What does RAW mean for people consuming the data? How does one know to consume the data?
  • With RAW, Given the Schema Id, you should know what to do with the data?
• We currently don't validate the content (semi-exception being IPFS CIDs)
  • We could with fixed width?
• How do we structure the model?
  • Parquet has a custom structure
  • Avro uses Avro Schemas
• Reminder that new schemas are validated by governance on creation, but validation of data is done at read-time, and not write-time
  • Validation of structure: Anyone
  • Validation of data and meaning: Only those who understand the schema's human meaning
• The simplicity of RAW is nice
  • Still requires some structure that describes the data or where to go to understand the structure
  • Simple as a url, or could do some basic fixed width, regular expression, etc...
  • What about using mimetypes directly? { url, content-type: <mime type, etc...>, description: }, or { other serialization structure data }
  • Perhaps this is "EXTERNAL" or some other named thing?