Convert all 1.0 example datasets to 1.1

[ccl-core]

We have a rich set of datasets conforming to Croissant 1.0.
Once we have all necessary modifications for 1.1 in place, we should convert them to the 1.1 specs, and store them in the 1.1 folder to be used as examples for users.

[SujataSaurabh]

@ccl-core I have few questions regarding this task I hope you can help with.

Question # 1:
Since the 1.1 spec seems to be backward compatible with 1.0, I suppose we could copy the existing test cases with no change into 1.1 directory, and it should work. Now, do we have list of updates in v1.1 related to fields (or constructs) that are now deprecated and should be removed or replaced with newer v1.1. I could see few differences in the draft spec:

• "repeated" is to be replaced by "is_array"
• Nested "RecordSet" could be replaced by "Annotations"
• etc.

I was wondering if there is a more complete checklist.

Question # 2:
Should we take advantage of the migrations script that I see has been used in the past for updating tests from v0.8 to v1.0, or should these changes expected to be mostly manual?

[ccl-core]

Hi @SujataSaurabh , thank you for this message!

Exactly, once we have included and implemented all modifications we want to have in 1.1, we can run the migration script in python/mlcroissant/mlcroissant/scripts/migrations/migrate.py to migrate the existing datasets in the 1.0 folder. The process should not be manual, because it would be error prone.
Thank you!