Repository navigation
Implement a standards-based, machine-readable Extended Data Dictionary #2373
Description
Already, we create an extended data dictionary with describegpt.
However, it's not machine-readable.
Create a machine-readable version that we can use with DP+ to populate the dcat-us:describedBy property.
https://doi-do.github.io/dcat-us/#dataset-describedBy
see #1705
Activity
jqnatividad commentedon Jul 14, 2025
As there is no existing standard format for a data dictionary, we will leverage JSON Schema validation instead.
qsv already supports the creation of a schema by inferring it from its summary stats and frequency table with the
schemacommand and validate similar CSV files with thevalidatecommand.So the Data Dictionary is not just a documentation artifact, it can be used for actual validation work.
And we can add DCAT3 properties without worrying if it invalidates the JSON Schema as the spec has
additionalPropertieswhere we can add the extended data dictionary "stuff" - e.g. summary stats, frequency table, etc.https://tour.json-schema.org/content/03-Objects/02-Additional-Properties
cc @rzmk @samibaig @wardi
jqnatividad commentedon Aug 12, 2025
Based on the learnings while creating the
--jsonoption for thefrequencycommand.Let's go ahead with extending JSON Schema so that is also doubles as a "validating" data dictionary as well.
Not only should it have summary stats, it will also have the frequency table for each field.
https://github.com/dathere/qsv/blob/master/scripts/NYC_311_SR_2010-2020-sample-1M-frequency.json
Take into account Croissant guidance on using properties from external vocabularies (mlcommons/croissant#885), so we can create one common format for both Croissant and DCAT-US v3's describedBy property.
jqnatividad commentedon Aug 18, 2025
The revamped
describegptfor the upcoming qsv 7.0 release is a step towards this.For the initial implementation, we'll be optimizing for the current CKAN data dictionary.
jqnatividad commentedon Nov 7, 2025
Closing this now that
describegpthas JSON output.We can map it to the standard CKAN dictionary and just use the JSONSchema created by the
schemacommand as a CKAN dataset resource that the DRUF can use to validate updates.